GalSim-developers/GalSim — pull request #475 by rmjarvis (2013-10-14 06:12:59+00:00)
Title: #474 Speed ups to RealGalaxy calculations

This branch includes a number of speed improvements that were important for the Great3 real galaxy branches.

- I learned about multiprocessing Managers, which provide an easy (-ish) way to avoid passing the full RealGalaxyCatalog to each process. Instead, we now keep all the input objects (most notably the RGC) in the root process, and put proxies to them in the config file that we pass to each sub-process. The Manager handles all the communication between the processes, so the proxies can be used (mostly) just like a regular object.
- There were a lot of niggly details to get this working, which you'll see if you look through the diff. e.g. Proxies can only use methods from the original object, not attributes. So I had to make getters for some things that we had been using attributes for. e.g. `rgc.nobjects` is now `rgc.getNObjects()`. (The attributes are still available as well -- I just added the getters where necessary.)
- There were a few issues w.r.t. the non-picklability of some of our objects. So, for example, I now build the noise CF object in `RealGalaxy`, rather than in `RealGalaxyCatalog`, since the CF that was returned by `rgc.getNoise()` is not picklable.
- These changes allowed me to have the `preload=True` option to RGC happen in the constructor as it sounds like it would. Before, this didn't work with multiprocessing, so we actually did the "preload" the first time `getGal` or `getPSF` was called.
- I also changed the `preload=False` option to keep the file handles of any files opened and use them again if we use the file again. I suspect this change will mean that `preload=False` (which is still the default) will be the preferred usage in almost all cases, since it only opens the files that are actually used, and it only opens each one once.
- I think pyfits has a bug in its memory mapping code (`pyfits.open(..., memmap=True)`) which is the default as of version 3.1. It took me a while to figure it out, but with nproc > 1, it would hang after running a long time when trying to access an hdu of an already opened file. With nproc = 1, it would eventually die with `mmap.error: [Errno 24] Too many open files` (long after all the files were already open). So I think somewhere in the pyfits mmap code it is leaving a file handle open. Anyway, using `memmap=False` fixed the problem and doesn't seem to otherwise lead to any performance hits.
- Another big speedup came from making the size of the nosie padding be a "goodFFTSize". We saw problems with bad FFT sizes in OpticalPSF. The same improvement happened here by rounding `noise_pad_size` up to `2**n` or `3*2**n`.
- Rachel had requested an `output.noclobber` option in config to avoid overwriting files that have already finished successfully. This is now implemented. See demo9.yaml for an example.
- I also added an `output.skip` option to let people skip files for other reasons than that they already exist. (Easy enough to do while I was adding the noclobber option.)
- I added the ability to access `file_num`, `image_num`, and `obj_num` in Eval strings. It seems like this is the most likely way anyone would actually use the `output.skip` option. Plus it's just a good idea to let people use these values.
- I had to make one minor change to the test suite. The uncorrelated noise variance wasn't coming out right for one of the tests after I made noise_pad_size round up to a better FFT size. Making the "big image" larger made it work, so I think this was just a case of a lucky success for the particular values used. Maybe it still is now with the larger image too. Hard to say.

Anyway, Rachel, you should try out your timing tests with and without preload using this branch now. I suspect preload=False will be slightly better, but I only tested with nproc=2, not 24, so I'm not sure how that might affect the timing. Actually, it wouldn't surprise me if the difference was negligible now.

The times are still around 0.3 - 0.4 sec per stamp (when the file is already open, so the pyfits I/O time doesn't contribute), which is larger than I'd like, but much better than before, and probably good enough to finish the great3 real galaxy run in a finite time.

I think the way forward for further improvements is to do 2x2 (or higher) binning of the real galaxy images. The pixel scale for the ground images is much larger than the pixel scale of the RealGalaxy images. So we are wasting a fair amount of time with the higher resolution InterpolatedImages. But making sure that works correctly (especially getting the noise attribute adjusted correctly) is a bigger job than I think we want to tackle right now...