praxes/hexrd — pull request #60 by mwiebe (2014-11-11 05:29:02+00:00)
Title: "hexrd cache-frames" subcommand to save the compressed coo_matrixes

Was thinking and experimenting a bit with the multiprocessing on Windows, and concluded that the best way to mitigate the lack of 'fork' multiprocessing is by caching the frame list of coo_matrix values. I've implemented this as both 1) a new subcommand and 2) automatically cache when fit-grains is called. The reason this helps is that while on Linux/OSX, we can load the frames once, then share the same memory via fork multiprocessing, on Windows, this has to go through pickle or similar. Explicitly caching seems like a good strategy, and since the data is so much smaller and quicker to load, is probably useful to have anyway.

I've confirmed identical before/after results on my machine. The .npz file is 1.3MB compared to multiple GB for the source images. On my laptop, loading the original frames took 209s, saving the cache took 12s, and loading the cache took 4s.

```
[py27] D:\Downloads\ruby_example>hexrd fit-grains multiruby.yml
14-11-10 21:08:00 - working_dir not specified, defaulting to D:\Downloads\ruby_example
14-11-10 21:08:00 - === begin fit-grains ===
14-11-10 21:08:00 - *** begin analysis "multiruby" ***
14-11-10 21:08:00 - logging to D:\Downloads\ruby_example\multiruby\fit-grains.log
14-11-10 21:08:01 - setting active material to 'ruby'
14-11-10 21:08:01 - no frame cache file D:\Downloads\ruby_example\frame_cache.npz found, generating cache
14-11-10 21:08:01 - reading 1440 frames of data
14-11-10 21:11:31 - read 1440 frames in 209.55 seconds
14-11-10 21:11:43 - wrote 1440 frames to cache in 12.156 seconds
14-11-10 21:11:43 - find_orientations:eta:mask not specified, defaulting to 5
14-11-10 21:11:43 - find_orientations:omega:period not specified, defaulting to [0.0, 360.0]
14-11-10 21:11:43 - fit_grains:npdiv not specified, defaulting to 2
14-11-10 21:11:43 - fit_grains:estimate not specified, defaulting to None
14-11-10 21:11:43 - fitting grains using default initial estimate
14-11-10 21:11:43 - fitting grains for 4 of 4 orientations
14-11-10 21:11:43 - 1 of 8 available processors requested
14-11-10 21:11:43 - running pullspots with 1 processors
14-11-10 21:11:43 - multiprocessing disabled
|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>| Time: 00:00:15 |<<<<<<<<<<<<<<<<<<<<<<<<<<<<<|
14-11-10 21:11:59 - processed 4 grains in 0.263283 minutes
14-11-10 21:11:59 - *** end analysis "multiruby" ***
14-11-10 21:11:59 - === end fit-grains ===

[py27] D:\Downloads\ruby_example>hexrd fit-grains multiruby.yml
14-11-10 21:13:16 - working_dir not specified, defaulting to D:\Downloads\ruby_example
14-11-10 21:13:16 - === begin fit-grains ===
14-11-10 21:13:16 - *** begin analysis "multiruby" ***
14-11-10 21:13:16 - logging to D:\Downloads\ruby_example\multiruby\fit-grains.log
14-11-10 21:13:17 - setting active material to 'ruby'
14-11-10 21:13:17 - reading 1440 frames from cache
14-11-10 21:13:21 - read 1440 cached frames in 3.915 seconds
14-11-10 21:13:21 - find_orientations:eta:mask not specified, defaulting to 5
14-11-10 21:13:21 - find_orientations:omega:period not specified, defaulting to [0.0, 360.0]
14-11-10 21:13:21 - fit_grains:npdiv not specified, defaulting to 2
14-11-10 21:13:21 - fit_grains:estimate not specified, defaulting to None
14-11-10 21:13:21 - fitting grains using default initial estimate
14-11-10 21:13:21 - fitting grains for 4 of 4 orientations
14-11-10 21:13:21 - 1 of 8 available processors requested
14-11-10 21:13:21 - running pullspots with 1 processors
14-11-10 21:13:21 - multiprocessing disabled
|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>| Time: 00:00:14 |<<<<<<<<<<<<<<<<<<<<<<<<<<<<<|
14-11-10 21:13:36 - processed 4 grains in 0.24575 minutes
14-11-10 21:13:36 - *** end analysis "multiruby" ***
14-11-10 21:13:36 - === end fit-grains ===

[py27] D:\Downloads\ruby_example>dir *.npz
Directory of D:\Downloads\ruby_example

10/11/2014 09:11 PM 1,325,514 frame_cache.npz
```