Improve in-memory testing API

This commit extends the Options API to more directly support in-memory
testing removing the need to do hacky things with the custom settings
Fix a bug in parallel soak testing behaviour