GalSim-developers/GalSim — pull request #408 by barnabytprowe (2013-04-23 15:21:35+00:00)
Title: #401 - Simple utility script for comparing two GSObjects with adaptive moments at user-specified accuracy

To aid validation on #325 , #343 etc., I have developed a little testing utility that compares `draw` and `drawShoot` images for a GSObject specified on input, repeating the photon shooting side of the test until a desired statistical accuracy is achieved.

The new utilities are the functions `galsim.utilities.compare_dft_vs_photon_object()` and `galsim.utilities.compare_dft_vs_photon_config()`, and they produce an output data structure storing the test information (a `galsim.utilities.ComparisonShapeData` instance).

* The `galsim.utilities.compare_dft_vs_photon_object()` takes raw GSObjects as inputs, with an optional PSF. Because these aren't picklable, this code has to run in single core mode.

* `galsim.utilities.compare_dft_vs_photon_config()` takes a config dict as input, and so can run in parallel by setting the `nproc` keyword.

These are currently the lowest level, simplest tests I could devise, since it made sense to automate the comparison and estimation of errors for the simplest possible case. From this I believe that it's possible to construct more complex tests (e.g. a test of consistency in the application of a shear, or magnification) using these units as building blocks, or at the very least the pattern they contain.

I have developed this on a separate branch from where it was started on #325 as it seemed to make sense to have it merged onto master so that @tomaszkacprzak could also make use of the error analysis it contains.

There is a test script `devel/modules/test_comparison_basic.py` that compares results, and we do get broadly statistically consistent (but not exactly the same - this is not expected due to the way the RNG seeds are set in config) answers for the object vs config tests. Here are subset of results (those with `n_photons_per_trial = 1e7`) showing differences in adaptive moments for a Sersic galaxy convolved with a Moffat:
```
Config results with N_PHOTONS = 10000000
g1obs_draw = -0.0898199845034
delta_g1obs = 8.98768245653e-05 +/- 2.17830355557e-05

g2obs_draw = -0.0725085640446
delta_g2obs = 1.30623499664e-06 +/- 2.4744899477e-05

sigma_draw = 2.55980110168
delta_sigma = -0.000833353648583 +/- 9.48064918375e-05

image size = 48
pixel scale = 0.27
wmult = 4.0

total time taken = 188.750197887 s
total number of iterations = 6
number of trials per iteration = 32
number of photons per trial = 10000000

Object results with N_PHOTONS = 10000000
g1obs_draw = -0.0898199845034
delta_g1obs = 0.00010969640416 +/- 2.47694126377e-05

g2obs_draw = -0.0725085640446
delta_g2obs = 6.59167235808e-05 +/- 2.31891643556e-05

sigma_draw = 2.55980110168
delta_sigma = -0.000861751536528 +/- 9.44758402361e-05

image size = 48
pixel scale = 0.27
wmult = 4.0

total time taken = 740.399615049 s
total number of iterations = 6
number of trials per iteration = 32
number of photons per trial = 10000000
```

Apart from this consistency check, I don't yet have a good idea for unit tests. Suggestions warmly welcomed!