Hi all,
Sam and I are working on issuing a PR about the volume rendering
refactor.  This is the last major code change to go in place for 2.4,
pending the quad_proj changes being accepted or rejected.  I'm still
thinking about this refine_by issue, and that might also make it in. 
After the volume_refactor, it's just testing and documentation left:
https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&milest...
After 2.4 is released, I am going to shift my new development
exclusively to the yt-3.0 branch, which is currently hosted here:
http://bitbucket.org/yt_analysis/yt-3.0
This will include geometry refactoring and many remaining
de-Enzoifications of the code base.  3.0 is nearing a usable state,
but will not be ready for production for a while.  However, I think
that allowing the codebases to diverge too rapidly will lead to
problems: a fracturing.  So once the 3.0 codebase can muster feature
parity with 2.x, I am going to ask that all development on the 2.x
branch cease, and all pull requests for features and new functionality
be applied only to the 3.0 branch.  In advance of this, I would like
to solicit help: we've identified a number of relatively
straightforward changes to this code base that can and should be made. 
For instance, the changing of field names.  I'd also like to consider
moving to a more spread-out source code layout, like one class per
file.  Changes like these make it difficult to do cross-branch merges
and transplants, which is why I would like to save them for after
feature parity has been reached and development on 2.x has halted. 
Previous threads:
http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0...http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0...http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0...http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/0019...http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/0019...
If anyone has any strong objections to this strategy, please reply. 
Estimated Timeline:
* June 26, 2.4 release
* July 30, 3.0 reaches rough feature parity
* Fall sometime: 3.0 merged into main 'yt' branch
* Some time after that: 3.0 released
Criteria for Feature Parity:
* All patch-based AMR codes support data region selection
* Parallelism is functional for all parallel-capable routines
* All patch-based AMR codes support projections, slices, cutting planes
* All patch-based AMR codes support volume rendering
* All patch-based AMR codes support halo finding
* Octree-based AMR codes (RAMSES and ART) support data region selection
* Octree-based AMR codes (RAMSES and ART) support slices, projections
and cutting planes
If any of you are interested in helping with this reengineering
effort, I would be extremely interested in collaborating.  My goal for
this effort is to focus on making yt a future-compatible code, and
changing it from an organically grown system into one that we design
based on the years of experience we all have now had with it. 
-Matt

Hi all,
There's been a flurry of activity on the docs this week. 
* All the cookbook recipes have been moved into the docs and the
cookbook shut down.  We now have 45 cookbook recipes, and they're
integrated much better into the docs. 
* Most of the docs have been rewritten to remove PlotCollection
* The data index is now part of the website repo.  It's up, but the
files are still copying. 
Here's my current draft of the changelog for 2.4.  Did I forget anything? 
https://bitbucket.org/MatthewTurk/yt-doc/src/b57d3621ff70/source/changelo...
We're just about good to go.  These things still need to be done:
* Rewrite the remaining docs that use PlotCollection.  Specifically,
orientation/making_plots.rst needs to be completely redone.  Can
someone volunteer to rewrite this document? 
* Volume rendering docs need to be rewritten en masse, as they are
quite out of date and don't reflect best practices or new
functionality (ticket #382). 
* Do we want to put answer testing docs in for this release, or
should we push that off to a point release at a slightly later date? 
* Write docs about the Hub, which I will do. 
* Check to make sure that everything that SHOULD be documented in the
API docs IS documented. 
In the past we've also talked about slides demonstrating new
functionality, as well as the possibility of doing screencasts.  I'm
going to throw together some HTML slides, but if anybody wants to do a
screencast that would be outstanding.  We actually have gotten *many*
views in the past of screencasts. 
I think that's about it.  There are two additional technical tickets
that should be addressed:
https://bitbucket.org/yt_analysis/yt/issue/401/entropy-field-does-not-wor...https://bitbucket.org/yt_analysis/yt/issue/390/transfer-function-bounds
I can do the second, but I don't know what to do about the first. 
Thanks all,
-Matt

Hey everybody,
I hope you're all enjoying your weekend and don't mind taking some time
out of it to think about yt. 
Right now there are only three open tickets for the 2.4 release:
1.  The entropy field in non-enzo data
2.  Transfer function bounds
3.  Volume rendering docs
I think these tickets are things that are fixable on a short timescale. 
Sam, please let us know if you need more time or help with the volume
rendering docs. 
We've put a lot of time into updating the cookbook and making sure the
scripts produce good-looking images based on a variety of datasets.  The
latest version of the docs is available here:
http://yt-project.org/docs/2.4/
I'm curious how everyone feels about setting a time aside sometime this
week to do the official release.  I think the release will attract more
eyeballs if we can point to screencasts about some of the new
functionality, including (but not limited to, see the changelog
<http://yt-project.org/docs/2.4/changelog.html#version-2-4> for the full
[really extensive and impressive] list of changes):
1.  Threaded volume renderer
2.  Plot window
3.  Improved time series analysis
4.  Improved extjs4 reason
5.  The new yt hub
I'm planning on doing a plot window screencast this afternoon.  Does
anyone else want to claim one of the features? 
Lastly, I contacted Kelle Cruz at astrobetter about going a guest post
about the yt 2.4 release.  She is interested - all she needs from us is
a google doc with a draft of the post, including text embedded videos,
and urls.  I'm going to begin the write up sometime this week and then
share the google doc on the yt-dev list so that others can contribute. 
The end is in sight!  This will be by far the best release of yt yet :)
Cheers,
Nathan

Hi all,
For the workshop we used a download.py script written by Stephen to
get all of the data, which was tarred up and put into subdirectories. 
John ZuHone and I have been trying to add more sample data, so that
people can (if they like) get started right away with using yt.  All
of the cookbook recipes have been or are being rewritten to use larger
datasets than the old RD0005 item, and many are being switched to use
FLASH data instead of Enzo data.  All in all, we're trying to push
pretty hard to make the cookbook more useful. 
One question that's come up is: do we want to continue using
download.py?  The burden on uploaders is moderately higher, in that
the files all have to be tarred up in a particular way, and they have
to be added to download.py, but it does provide a measure of
robustness.  If we do this, can we move download.py into the main
distribution, under scripts/ ?  Stephen, John, others who have used
the script, what do you think about this? 
-Matt

Britton has just submitted a pull request to bring LightCone and
LightRay up to date with the current parallelism strategy.  He asks a
few questions that others on this list may have thoughts on; I will be
leaving mine in a comment on the PR. 
Thanks, Britton! 
---------- Forwarded message ----------
From: Britton Smith <pullrequests-noreply(a)bitbucket.org>
Date: Mon, Jul 23, 2012 at 11:01 PM
Subject: [yt_analysis/yt] Modernized LightCone and LightRay (pull request #214)
To: matthewturk(a)gmail.com
A new pull request has been opened by Britton Smith. 
brittonsmith/yt has changes to be pulled into yt_analysis/yt. 
https://bitbucket.org/yt_analysis/yt/pull-request/214/modernized-lightcon...
Title: Modernized LightCone and LightRay
Major changes to the infrastructure supporting the LightCone and LightRay:
1) The LightCone and LightRay generators have now been switched over
to the new SimulationTimeSeries so that they can now, in principle be
used with other simulation codes.  This will require other frontends
to add code similar to what's
yt/frontends/enzo/simulation_handling.py. 
2) Both the LightCone and the LightRay are now parallelized using the
parallel_objects function, so they can be used much more flexibly with
multi-level parallelism.  This also removes the direct mpi4py imports
from LightRay. 
3) LightCone and LightRay, along with there supporting superclass
CosmologySplice, have been moved to
yt/analysis_modules/cosmological_observation. 
This code has been heavily tested and retains all pre-existing
functionality, plus a few more things that will be mentioned in the
documentation and cookbook recipes that will be updated shortly. 
Before accepting this PR:
1.  Are people happy with the new location:
yt/analysis_modules/cosmological_observation?  I believe that all of
this code belongs in a subdirectory together but that was the best
name I could come up with. 
2.  The old EnzoSimulation (yt/analysis_modules/simulation_handling)
class still exists, but it is no longer a dependency for anything, and
is totally replaced with the new EnzoSimulation in yt/frontends/enzo,
which is totally superior.  Can I remove this now or should it be
deprecated and scheduled for removal at a later date? 
Changes to be pulled:
cd09f872c174 by Britton Smith: "Merged." 
f7bc036677fd by Britton Smith: "Merged." 
a303d6f14a33 by Britton Smith: "Changing cosmology subdirectory to
cosmological_observation
and adding imports t…"
b3c9210bf072 by Britton Smith: "Added LightRay to api file." 
b764002ef788 by Britton Smith: "Modernized the light ray generator and
moved it to a new home. 
The direct mpi4py…"
41f4117f4613 by Britton Smith: "Moving light ray to the cosmology subheading." 
9604d5d6187a by Britton Smith: "Adding setup for cosmology module." 
6475d956eec7 by Britton Smith: "Removing light cone generator from setup." 
304187324667 by Britton Smith: "Removing old light cone generator." 
c9a318592c97 by Britton Smith: "Cleaned up unique light cone generator." 
29aa8b84169c by Britton Smith: "Adding keyword to halo mask to specify
the virial overdensity
to be used to calc…"
bdf0cc5fa462 by Britton Smith: "Changing some more variable names." 
f7b9c8466b13 by Britton Smith: "Parallelized light cone halo mask with
parallel objects and
simplified code cons…"
042624ea193e by Britton Smith: "Continued cleanup of light cone halo
mask and changed light cone
variable to low…"
68472c735864 by Britton Smith: "Fixed some style conventions in light
cone and began
fixing light cone halo mask…"
92a989d572f3 by Britton Smith: "Switched lightcone generator from
being a subclass of TimeSeriesData
to using pa…"
64e5330fd707 by Britton Smith: "Merged." 
c7c2fcaa10b1 by Britton Smith: "Changed EnzoSimulation.get_time_series
to allow further selection with find_outp…"
1125ca16272c by Britton Smith: "Adding files for new light cone into
analysis_modules/cosmology. 
There is curren…"
5656cf35de29 by Britton Smith: "Removed enzoisms from cosmology splice." 
db5ec44e0121 by Britton Smith: "Changed absorption spectrum generator
to increase the wavelength
window until th…"
1baa32f77c80 by Britton Smith: "Fixed bug in voigt profile generator
that was causing damping wings to
only be a…"
be28c6705909 by Britton Smith: "Adding files for CosmologyTimeSeries." 
7acc4cace811 by Britton Smith: "Changed EnzoSimulation.get_time_series
to allow further selection with find_outp…"
--
This is an issue notification from bitbucket.org. 
You are receiving this either because you are the participating
in a pull request, or you are following it. 

Hi all,
This PR fixes a number of issues with the plot window.  Drop by IRC and
let us know if you catch any more issues.  Hopefully we have most of the
corner cases covered by now :)
There are also some docs on the plot window in the yt-doc repo under
visualization/plots.rst. 
Cheers,
Nathan
On 7/23/12 12:00 PM, Matthew Turk wrote:
> New comment on pull request:
>
> https://bitbucket.org/yt_analysis/yt-doc/pull-request/37/fixing-sphinx-er...
>
> Matthew Turk (MatthewTurk) said:
>
> Looks good with latest update.  Thanks! 
>
> --
> This is a pull request comment notification from bitbucket.org. 
> You are receiving this either because you are participating
> in a pull request, or you are following it. 

Hi all,
I've managed to fix most of the sphinx errors our docs build produces. 
I still get the a few warnings, but I think we can safely ignore most of
them. 
There are still a few errors that touch parts of the docs I didn't
write, specifically:
yt-doc/source/advanced/creating_frontend.rst:: WARNING: document isn't
included in any toctree
yt-doc/source/advanced/reason_architecture.rst:: WARNING: document isn't
included in any toctree
yt-doc/source/visualizing/simulated_observations.rst:: WARNING: document
isn't included in any toctree
yt-doc/source/workshop.rst:: WARNING: document isn't included in any toctree
The errors imply that these four files aren't accessible by navigating
through the docs.  Does anyone know how should these sections be
integrated into the rest of the docs? 
I also get several errors complaining about missing images from the
cookbook.  Are the images not included in the repository for a reason? 
Is there a helper script somewhere to generate the images?  If so, I'll
look into integrating it into the build process. 
-Nathan

Hi all,
I've been playing around with some TimeSeries examples for the cookbook and I have noticed a couple of things that seem frankly annoying and I just wanted to ping the list and see if there was a reason for them. 
1) If I run more than one task in a script, it has to load all of the pfs again.  Is this by design? 
2) Secondly, I have noticed that what gets returned in a parameter lookup or a quantities computation has one dimension more than desired.  For example, getting the simulation time of each pf in the TimeSeries yields:
times = ts.params.current_time
where times is a list of lists, each sublist with one member, with
na.array(times).shape == (number of pfs, 1)
The same sort of thing happens with returning, say, the angular momentum vector, which will have a shape of (number of pfs, 1, 3). 
It seems to me that what should be returned is a NumPy array with the dimensionality scaled down by one, e.g. the lists "in the middle" should be eliminated. 
Best,
John Z

Hi all,
It seems that the recent changes to the docs have broken the internal
links to api references.  Since the docstrings are inside the api
references, this means the docstrings don't show up in new builds of the
docs.  I'm not sure if this is a sphinx issue or an issue with the tip
of yt-doc. 
I think there was some discussion about consolidating the api reference
pages, which I am in favor of, but we should still make sure the
docstrings are available inside the html docs. 
Nathan