Yelp/Tron — pull request #258 by Codeacious (2013-07-22 22:23:36+00:00)
Title: Implemented #221, called tronstore

This is what is hopefully really close to a final version of tronstore, a new process to be run as a child alongside of trond that will parallelize state serialization. Trond will now spawn this process as part of the `PersistenceManagerFactory` by default, rather than independently creating each store type class.

Here's a basic overview of what's happened structurally:

- There's now a new folder in `tron/serialize/runstate`, called `tronstore`. This, obviously, holds all the new files related to tronstore.

- PersistenceManagerFactory now spawns only `ParallelStore` objects, a new, top level class that implements the same front facing methods as the old *Store objects (ShelveStore, YamlStore, etc), as well as one new front facing method, `update_from_config`.

- The PersistentStateManager now has an `update_from_config` method that calls ParallelStore's update_from_config method, and `StateChangeWatcher.update_from_config` has been updated to use this.

- The ParallelStore object creates a `StoreProcessProtocol` object (located in `tronstore/process.py`) that handles all low level process management and communication. It does couple with ProcessProtocol in assuming that everything sent to tronstore is going to be either a StoreRequest or a StoreResponse (more on that later), and relying on the ParallelStore object to update its `StoreRequestFactory` whenever the configuration is updated.

- The StoreProcessProtocol object works by using the Python multiprocessing module. It spawns `tronstore.main()` from `tronstore/tronstore.py` as a daemonized process, passing in a Pipe object for message communication. Pipes, by default, use pickle (not cPickle) for serialization, so instead, there are factories for request and response objects, which are serialized based on the Tron configuration given at runtime. The raw, serialized text is what is actually sent over the Pipe object. Pipes work somewhat like a two-way Queue, so there's no need to deal with chunking (despite there being some chunking related code left over in `tronstore/chunking.py`).

- The two factory objects, `StoreRequestFactory` and `StoreResponseFactory` live in `tronstore/messages.py`. They deal with constructing and reconstructing `StoreRequest`s and `StoreResponse`s to and from tronstore, respectively. There's not too much here- just an id for matching the requests and responses together (which is actually done by the StoreProcessProtocol), and a serialization class mapping (the actual methods for serialization/deserialization live in `tronstore/transport.py`; the mapping references these classes).

- There's four types of requests, which have enumerators in `tronstore/msg_enums.py`. They are SAVE, RESTORE, CONFIG, and SHUTDOWN. Which type is sent is controlled by ParallelStore, which constructs requests according to whatever method was called. SAVE is a non-blocking requests, while RESTORE, CONFIG and SHUTDOWN are. Blocking requests time out, after which a log entry is written and a failure is returned.

- tronstore itself works by polling for requests on the Pipe. When a request is received, it usually delegates a thread to handle and serve the request. It puts the thread in a queue, which is then actually started by another thread that acts as a thread pool (it simply makes sure more than a certain number of threads aren't already running). If the request was a SHUTDOWN or CONFIG request, tronstore will wait until all other requests have finished, and then take appropriate action. Responses from all requests are sent back over the Pipe to trond.

- There are signal handlers in tronstore for `SIGINT`, `SIGTERM`, and `SIGHUP`, which are the three signals also registered to custom handlers in trond. The handlers do nothing, so that trond can take the correct, complete action when one of the signals is recieved. This is necessary, as signals are sent to all processes in a process group, and tronstore is spawned in the same group as trond. To allow for multiple processes with signal handling, a flag, `installSignalHandlers`, had to be switched off on the Twisted reactor.

- There's a couple new configuration options, both living in the `state_persistence` section of the config, called `transport_method` and `db_store_method`. Both must be set to one one of `json`, `msgpack`, `pickle`, or `yaml`, but db_store_method isn't actually enforced unless `store_type` is set to `sql` or `mongo`. store_type is still the same as before, but is now instead used by tronstore to set up its own internal store object (new store objects are located in `tronstore/store.py`).

- There is a script in `tools/` that will migrate the stored state of Tron from 0.6.1 to 0.6.2. It's got a really long docstring describing how to use it. The old migrate_state.py script has been updated to migrate state for tronstore state objects.