GalSim-developers/GalSim — pull request #503 by rmjarvis (2014-01-25 14:20:22+00:00)
Title: #344 Speed up gzip and bzip2 I/O.

This is not a huge issue, but it's been sitting here for awhile, so I finally got around to looking into it. I have updated the code that handles gzip and bzip2 to try a few different options in order until it finds one that works, including an option of using a subprocess to a system gzip or bzip2 call. I also made a script tests/time_zip.py, which runs through the different options I have for 3 different image sizes in each case for both reading and writing. Here are the results for 6 different systems I have access to:
```
System	OSX 10.6.8	OSX 10.6.8	Linux 2.6	Linux 2.6	Linux 2.6	Linux 2.6
Computer name (for ref)	Flute	Flute(fink)	Susi	Folio	Otto	Astro0034
Python version	2.6.1	2.7.3	2.6.6	2.7.3	2.6.6	2.7.5
Pyfits/Astropy version	3.2	0.3	2.3	0.2.4	2.4	0.2.5

Write big .gz file
time for gzip_in_mem	8.18	10.36	26.85	7.18	6.89	7.50
time for gzip_tmp	9.62	12.19	23.92	7.94	6.78	8.31
time for gzip_call	7.11	8.80	fail	6.62	fail	6.78
time for gzip_call2	8.87	10.57	24.09	7.79	6.47	7.77
Write medium .gz file (20X)
time for gzip_in_mem	6.35	9.33	11.21	5.74	5.56	6.36
time for gzip_tmp	7.57	10.45	19.54	6.58	5.45	6.92
time for gzip_call	5.93	9.09	fail	5.80	fail	5.66
time for gzip_call2	7.02	10.32	19.91	7.49	5.35	6.71
Write small .gz file (400X)
time for gzip_in_mem	5.57	6.80	11.34	7.41	4.45	8.40
time for gzip_tmp	6.62	8.03	20.98	11.64	4.54	11.77
time for gzip_call	7.60	9.07	fail	8.82	fail	8.69
time for gzip_call2	7.95	9.83	24.11	9.03	6.48	12.58

Write big .bz2 file
time for bz2_in_mem	19.57	23.32	22.47	18.99	19.82	18.69
time for bz2_tmp	21.07	24.83	32.61	19.78	19.64	19.44
time for bzip2_call	18.98	22.81	fail	19.17	fail	17.93
time for bzip2_call2	21.41	24.89	33.10	19.80	19.90	19.12
Write medium .bz2 file (20X)
time for bz2_in_mem	15.99	19.15	17.95	15.24	15.37	15.15
time for bz2_tmp	17.45	19.99	26.70	15.92	15.26	15.76
time for bzip2_call	16.21	19.92	fail	16.18	fail	14.64
time for bzip2_call2	17.15	20.50	27.07	17.46	15.26	15.51
Write small .bz2 file (400X)
time for bz2_in_mem	12.27	14.07	16.29	13.82	11.55	13.00
time for bz2_tmp	13.27	15.00	26.06	15.26	11.63	14.83
time for bzip2_call	15.24	16.91	fail	16.61	fail	16.65
time for bzip2_call2	15.45	18.30	30.08	18.40	14.68	19.70

Read big .gz file
time for gzip_in_mem	3.00	4.47	1.75	2.65	2.94	3.27
time for gzip_tmp	3.40	4.27	10.85	1.93	1.44	2.22
time for gunzip_call	1.36	1.61	0.92	1.13	1.51	1.28
time for pyfits_open	2.98	4.48	0.75	2.66	1.43	3.62
Read medium .gz file (20X)
time for gzip_in_mem	2.14	3.34	1.33	1.95	2.02	2.32
time for gzip_tmp	2.60	5.09	8.97	1.46	0.94	1.65
time for gunzip_call	1.23	1.58	1.10	1.07	1.16	1.23
time for pyfits_open	2.11	3.23	0.56	1.99	0.89	2.31
Read small .gz file (400X)
time for gzip_in_mem	1.97	3.00	1.49	2.38	1.90	2.69
time for gzip_tmp	2.47	60.51	9.61	2.85	1.05	5.21
time for gunzip_call	4.71	6.41	5.71	4.35	4.85	6.94
time for pyfits_open	1.98	3.09	1.85	2.44	1.00	2.84

Read big .bz2 file
time for bz2_in_mem	28.16	33.53	16.27	25.54	22.71	21.98
time for bz2_tmp	11.71	13.62	16.52	9.52	8.28	8.48
time for bunzip2_call	10.12	11.61	5.95	8.78	7.95	7.20
Read medium .bz2 file (20X)
time for bz2_in_mem	22.12	25.52	13.05	20.39	18.14	17.45
time for bz2_tmp	10.00	13.20	12.88	7.99	6.43	6.89
time for bunzip2_call	8.20	10.23	4.94	7.60	6.44	6.03
Read small .bz2 file (400X)
time for bz2_in_mem	17.04	19.34	10.91	16.85	14.35	14.38
time for bz2_tmp	7.68	55.43	14.07	7.65	5.14	8.39
time for bunzip2_call	8.96	10.74	9.69	11.07	7.93	9.96
```

It's kind of an interesting table, since the performance of the different methods are rather system-specific. Some takeaway points:
* For big files, the best option is usually the call to the system program (`*_call` in the table).
* For small files, the subprocess overhead matters a lot and that generally becomes the slowest option.
* The current default algorithm (`*_in_mem`) is really bad for reading bzip2 files.
* The current backup algorithm (`*_tmp`) is not as bad as I thought, and is sometimes quite good. But occasionally it is abysmal. Also, as the `in_mem` algorithm never fails for any of my systems, it seems to never come into play currently. I'm guessing I must have made that for a system with an old version of pyfits that has since been upgraded or that I no longer have access to.
* Rachel's original comment that prompted this issue was for extremely slow writes to gzip files. She found the GalSim algorithm (probably the `in_mem` option, although it's possible I changed the code since then) to be much much slower than running gzip on the file after the fact (essentially equivalent to `gzip_call2`). None of my systems seem to reproduce this difference. So I'm curious if Rachel still sees a big difference between these on her system.

The upshot is that I think the code should use the `_call` method as its first choice in all cases. It is generally the fastest option for big files, which is when the gz or bzip2 option is most likely to be used, and when the difference in the timings is most important. For small files, people probably won't bother with gz or bz2 compression, so it shouldn't matter that we are using a slow option in that case.

For the second choice if the `_call` method fails, I guess it makes sense to stick with the `_in_mem` option, even though it is not always the fastest. It's probably not worth trying to be too careful about when it is faster than `_tmp`, since it's sort of an edge case, so we just want to make sure we have a backup that works. And we can keep the `_tmp` as a third choice in case the old pyfits version for which `_in_mem` failed turns up somewhere.

But I'm also curious if anyone else has wildly different results than these. Feel free to check out this branch, #344, and run the program time_zip.py in the tests directory, and post your results. Especially post them if they seem to differ substantially from any of my systems.