GreenGroup/RMG-Py — pull request #111 by rwest (2013-04-05 20:47:48+00:00)
Title: Using SCOOP for distributed parallel processing

SCOOP (Scalable Concurrent Operations in Python) is a distributed task module allowing concurrent parallel programming on various environments, from heterogeneous grids to supercomputers.
http://code.google.com/p/scoop/

This branch implements it for bits of RMG-Py.

Do a `pip install scoop`, checkout this branch, then try it with `python -m scoop -vv rmg.py path/to/input.py` or see the queue submission file in `examples/rmg/scoop/` for running on a multi-computer cluster.

I don't think this is ready to be merged yet, but am making a pull request as somewhere to store discussions.

So far:

- [x] Database cache saving (in the main thread) and loading (in each worker)
- [x] Functioning on a multi-core machine
- [x] Functioning across several nodes on a cluster (via LSF queuing system)
- [x] Implement for Thermo estimation.
- [ ] Implement for Reactor simulations.
- [ ] Implement for Pressure dependence calculations.
- [ ] Get logging information to end up where you expect it.

Until we implement QMThermo, parallelizing thermo estimation alone doesn't save an awful lot of time.
Implementing the reactor simulations and PDep calculations may be more difficult, or at least involve more pickling and passing. E.g. can we unpickle an updated network after a remotely executed PDep calc, or will all the reactions and species point to the wrong things?