crits/crits — pull request #268 by thelok (2014-10-07 21:22:15+00:00)
Title: Services thread/process pools

Closes issue #244

The purpose of the thread/process pools is to throttle the number of simultaneous services tasks in CRITs.

So the default SERVICE_MODEL is still thread/process. If a user wants to change it to thread_pool or process_pool then they need to explicitly set it in the control panel under System->Services. The pool size can be controlled in the control panel as well.

Keep in mind that process_pool spawns additional PIDs and has a copy of the memory footprint of the main CRITs process so it could be memory intensive. @dsnellgrove mentioned he runs CRITs in daemon mode within their mod_wsgi in issue #231 but I haven't tried to experiment with that.

There were certain services (metacap and chopshop) that could not run in pool mode because the underlying libraries spawned their own processes/threads and it could not be in 'daemon' mode. See #231, #244

Processes can be run in "compatibility mode" (yeah the name sucks) which means that if the CRITs instance is set to run in either process_pool or thread_pool then the service will instead run in process or thread mode, respectively. Each service can set their compatibility mode in their __init__.py script by setting the "compatability_mode" variable. By default all services have this value set to False if not defined.

This PR goes along with https://github.com/crits/crits_services/pull/57