haypo/pysandbox — pull request #21 by philpep (2013-05-02 15:31:02+00:00)
Title: WIP: Persistent child using multiprocessing and queues

Hi haypo, thanks for pysandbox!

I plan to use it in production in a application which process lot of data and we want to provide user scripts for post-processing. But as these scripts are executed very often, we can't spawn a subprocess for each. So there is work in progress to use the same process to execute unsafe code.

Let me known what you thinks about the code, and if you're interested I'll be happy to improve it given what you say.

Encore merci ;)

COMMIT MESSAGE:

Prevent creation of a subprocess for each call on call() or execute() by
using a single process per Sandbox instance. This improve the
performance when using many call()/execute() on the same Sandbox
instance.

The process will be killed an re spawned if it takes more than
config.timeout to execute a single task.

The process is started on first call on call() or execute() and
terminated when the sandbox instance deleted or when the parent died.

USAGE:

>>> from sandbox import Sandbox, SandboxConfig
>>> sandbox = Sandbox(config=SandboxConfig(persistent_child=True))
>>> ctx = {"a": 1, "b": 2}
>>> sandbox.execute("c = a + b", None, {"a": 1, "b": 2})
>>> ctx
{"a": 1, "b": 2, "c": 3}
>>> sandbox.call(lambda x: x, 1)
1

TODO:

* call() only work for simple functions without external dependencies.
It's seems hard to fix. But in the use case of a constant function
with pickables arguments, we may instantiate the child and give him a
reference to the function.
* multiprocessing.Queue() use threading, thus we can't limit thread
creation in the child process. Maybe we should use a lower level
queue class
* stdout/stderr not tested
* Only tested with python 2.7 on Debian.