GalSim-developers/GalSim — pull request #487 by rmjarvis (2013-11-12 07:05:10+00:00)
Title: Fix some bugs in config when the number of objects per image is not constant.

In building one of the Great3 branches, Rachel discovered that the config stuff doesn't work correctly when the number of objects per file is not constant. There were a number of places where it implicitly assumed that the number of objects per image is constant.

I originally thought this would be a pretty small change that I could safely just merge directly into master, but there turned out to be quite a few rather subtle issues that took me a while to track down. Anyway, it's not on a normal issue branch; it's on `nobj_bug`. So if you want to check it out and run the tests, that's the branch to use.

The changes are:
- The calculations that determine the number of objects were happening after loading the _first_ set of input objects (catalogs, dict, etc.), but not after loading the input objects for the file in question. So if the number of objects depends on a dict item, then it gets the wrong number of objects. (There was already a procedure to handle the case where the number of objects depends on a catalog.) Because of some other "features" in the code, the number of objects for the second and later files came out to the value for the previous file. This was the behavior Rachel was seeing. Now input items are marked to indicate whether they might be needed at file scope, and if so, they are processed in the initial processing of each file, before the number of objects is computed. The only two that are relevant currently are Dict and FitsHeader.
- In general, the cached items (called `current` in the config code) related to input fields were not getting cleared when an input object was rebuilt (e.g. a new dict file is read in). I had thought this was the main cause of Rachel's problem, but for various reasons it was not actually affecting things in her case. But now I explicitly invalidate the cached values related to an input item whenever a new input object is constructed, just to be safe.
- We construct this index scheme for indexing into a catalog automatically if the user omits the index field, which is convenient, since the default is usually what you would want. However, the way the index is used is basically `obj_num % nitems` where `obj_num` is a running count of all objects being processed, and `nitems` is the number of items in the current file. The index was wrapping on the number of items in the catalog, which works if nitems is the same for all catalogs, but not if nitems is changing. I added a workaround for now, but I think it would be better to switch the sequencing index to be the object number _within_ a file, rather than the overall object number. This would make the automatic sequencing scheme work, and it is probably what users would expect if they set up their own indexing scheme. But it would be a non-backwards compatible change for runs that don't have variable nobjects, so I'll do that on branch #482, rather than here.
- I changed demo9 to have a variable number of objects per file, ranging from 15 to 30 randomly. (It had been 20 for all files.) This effectively serves as a unit test for at least some of the effects of having a variable number of objects per file.
- When I made the change to demo9, I discovered that file-scope items don't have an rng available to be used. So I added this.

I _think_ I tracked down all the places that were implicitly assuming the number of objects is constant, but it is certainly possible that I could have missed some. The change to demo9 helps. And I ran the script Rachel sent me, and that seems to be working now.

This branch will need to get merged into the release/1.0 branch as well, and we'll need to make a 1.0.1 with it. However, I propose that we hold off on that until Rachel is finished making the rest of the Great3 images in case we discover another bug.