StackStorm/st2 — pull request #789 by Kami (2014-11-10 12:08:18+00:00)
Title: [WIP] [RFC] Sensor sandboxing and isolation

This pull request introduces sensor sandboxing and isolation. Sensors are sandboxed by running them in a separate process. The sensor process uses Python binary from the virtual environment which is specific to the pack to which the sensor belongs.

Related pull request which includes "packs" command changes is #778.

(Note: Currently this is just a prototype and not everything is hooked up and working yet).

## Benefits

In addition to the "common" sandboxing / isolation benefits, this approach also has some other benefits:

1. It's easier to scale and run sensors on multiple servers.

Sensor wrapper is mostly self-sustaining and doesn't depend on the whole reactor process running.

This means only the pack content and sensor wrapper needs to be located on the remote host where we want to run the sensors.

2. Looser coupling.

Sensors are less coupled to the container process and other st2 components.

They only communicate with st2 using the API endpoint which is used to dispatch the triggers. Previously, they were coupled to the container process.

3. Easier to debug and test the sensors.

User can run and test the sensors directly using the sensor wrapper script.

## Downsides

One of the main downsides of this approach is increased resource consumption (memory and CPU).

I personally view this more as a trade off than a downside (we trade security and isolation for some additional resource consumption).

This should also be a less of an issue when we better support "scale-out" deployments (running components on multiple servers).

## Open questions / To discuss

Here are some things which still need to be discussed. Some of them are just things I have observed while working on this change and don't need to be addressed immediately.

1. Sensor communication with the container process (aka dispatching triggers)

Lakshmi and I have discussed this on Slack. We both agreed that having sensors dispatch trigger events by sending a request to the API is a good approach.

Using this approach means we don't need a bi-directional communication between the container and sensor and the sensor wrapper process is more or less self-sustaining.

For this to work, we need to add special API endpoint which takes a trigger and dispatches a payload. We can't use the existing webhook sensor anymore. The existing webhook sensor assumes it has direct access to the container service which is not true anymore.

2. API authentication

This is something we probably don't need and want to tackle right now since all the sensor processes will run on the same server as the other st2 components.

Eventually though (when we scale out and support running components on multiple servers), we should figure out how sensors should authenticate to the API when they dispatch a trigger. One approach which I think is reasonable is generating a temporary and limited access API key for each sensor process.

This key would have limited access (only allow to post triggers to a special endpoint) and be revoked when the sensor process exits.

3. Running multiple instances of the same sensor

Currently, you can only run one instance of a particular sensor class.

This works fine for now, but I do see a use case where you would want to run multiple instances of the same sensor with different configuration options (e.g. you want to monitor multiple JIRA instances, etc.).

This is a bigger task and would require quite a bit of refactoring so it's not something we probably want to do right now, but it's just something to keep in mind and think about.

To support this use case, we would need to (among other things) allow user to use different config files and the sensor class would need to expose a method for returning it's ID (e.g. for JIRA sensor this would probably just be .).

## Other

If we are all on the same page and agree that is the right approach, I fill finish this PR and work on a new PR which introduces the same changes for the Python runner actions.