Repository: chaostoolkit/chaostoolkit-lib

Issue:
Pass configuration and settings to global controls
As per #99, controls loaded from the settings are not provided the experiment configuration and secrets. However, as per https://github.com/chaostoolkit/chaostoolkit-lib/issues/99#issuecomment-487628313, this could be handy and expected.

Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=h1) Report
> Merging [#117](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=desc) into [master](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/commit/e49afa4e463e441316904850fc6986d35ee63dc8?src=pr&el=desc) will **increase** coverage by `0.32%`.
> The diff coverage is `88.15%`.

[![Impacted file tree graph](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/graphs/tree.svg?width=650&token=XAcvtqktVg&height=150&src=pr)](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #117      +/-   ##
==========================================
+ Coverage   82.85%   83.18%   +0.32%     
==========================================
  Files          23       23              
  Lines        1540     1558      +18     
==========================================
+ Hits         1276     1296      +20     
+ Misses        264      262       -2
```


| [Impacted Files](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [chaoslib/caching.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvY2FjaGluZy5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [chaoslib/control/python.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvY29udHJvbC9weXRob24ucHk=) | `86.58% <100%> (+1.58%)` | :arrow_up: |
| [chaoslib/control/\_\_init\_\_.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvY29udHJvbC9fX2luaXRfXy5weQ==) | `87.82% <100%> (+0.4%)` | :arrow_up: |
| [chaoslib/experiment.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvZXhwZXJpbWVudC5weQ==) | `90.11% <82.35%> (+0.35%)` | :arrow_up: |
| [chaoslib/provider/python.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvcHJvdmlkZXIvcHl0aG9uLnB5) | `86.3% <0%> (+1.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=footer). Last update [e49afa4...9c05e5c](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

codecov-io: # [Codecov](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=h1) Report
> Merging [#117](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=desc) into [master](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/commit/e49afa4e463e441316904850fc6986d35ee63dc8?src=pr&el=desc) will **increase** coverage by `0.32%`.
> The diff coverage is `88.15%`.

[![Impacted file tree graph](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/graphs/tree.svg?width=650&token=XAcvtqktVg&height=150&src=pr)](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #117      +/-   ##
==========================================
+ Coverage   82.85%   83.18%   +0.32%     
==========================================
  Files          23       23              
  Lines        1540     1558      +18     
==========================================
+ Hits         1276     1296      +20     
+ Misses        264      262       -2
```


| [Impacted Files](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [chaoslib/caching.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvY2FjaGluZy5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [chaoslib/control/python.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvY29udHJvbC9weXRob24ucHk=) | `86.58% <100%> (+1.58%)` | :arrow_up: |
| [chaoslib/control/\_\_init\_\_.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvY29udHJvbC9fX2luaXRfXy5weQ==) | `87.82% <100%> (+0.4%)` | :arrow_up: |
| [chaoslib/experiment.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvZXhwZXJpbWVudC5weQ==) | `90.11% <82.35%> (+0.35%)` | :arrow_up: |
| [chaoslib/provider/python.py](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117/diff?src=pr&el=tree#diff-Y2hhb3NsaWIvcHJvdmlkZXIvcHl0aG9uLnB5) | `86.3% <0%> (+1.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=footer). Last update [e49afa4...9c05e5c](https://codecov.io/gh/chaostoolkit/chaostoolkit-lib/pull/117?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1f326ad..58d7ee9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,18 @@
 
 - Fix to ensure a control's `configuration` parameter is populated when it the 
   control is being `configured` [#114][114]
+- Load and apply global controls, those declared in the settings, from the
+  `run_experiment` function rather than out of band [#116][116]
+
+  This means that global controls will not be applied before the experiment
+  is validated but only when it's executed.
+
+  It also means the global controls will be provided with the experiment, its
+  configuration and secrets in case global controls must access these.
+
+[114]: https://github.com/chaostoolkit/chaostoolkit-lib/issues/114
+[116]: https://github.com/chaostoolkit/chaostoolkit-lib/issues/116
+
 
 ## [1.2.0][] - 2019-04-17
 
diff --git a/chaoslib/caching.py b/chaoslib/caching.py
index d31ebc0..066ca31 100644
--- a/chaoslib/caching.py
+++ b/chaoslib/caching.py
@@ -2,11 +2,12 @@
 # Builds an in-memory cache of all declared activities so they can be
 # referenced from other places in the experiment
 from functools import wraps
+import inspect
 from typing import List, Union
 
 from logzero import logger
 
-from chaoslib.types import Activity, Experiment
+from chaoslib.types import Activity, Experiment, Settings
 
 
 __all__ = ["cache_activities", "clear_cache", "lookup_activity", "with_cache"]
@@ -48,11 +49,18 @@ def with_cache(f):
     function.
     """
     @wraps(f)
-    def wrapped(experiment: Experiment):
+    def wrapped(experiment: Experiment, settings: Settings = None):
         try:
             if experiment:
                 cache_activities(experiment)
-            return f(experiment)
+
+            sig = inspect.signature(f)
+            arguments = {
+                "experiment": experiment
+            }
+            if "settings" in sig.parameters:
+                arguments["settings"] = settings
+            return f(**arguments)
         finally:
             clear_cache()
     return wrapped
diff --git a/chaoslib/control/__init__.py b/chaoslib/control/__init__.py
index a8057eb..31fc704 100644
--- a/chaoslib/control/__init__.py
+++ b/chaoslib/control/__init__.py
@@ -3,7 +3,6 @@ from contextlib import contextmanager
 from copy import deepcopy
 from typing import List, Union
 
-import contextvars
 from logzero import logger
 
 from chaoslib.control.python import apply_python_control, cleanup_control, \
@@ -19,7 +18,10 @@ __all__ = ["controls", "initialize_controls", "cleanup_controls",
            "validate_controls", "Control", "initialize_global_controls",
            "cleanup_global_controls"]
 
-global_controls = contextvars.ContextVar('global_controls', default=[])
+# Should this be protected in some fashion? chaoslib isn't meant to be used
+# concurrently so there is little promise we can support several instances
+# at once. When the day comes...
+global_controls = []
 
 
 def initialize_controls(experiment: Experiment,
@@ -47,7 +49,7 @@ def initialize_controls(experiment: Experiment,
 
         provider = control.get("provider")
         if provider and provider["type"] == "python":
-            initialize_control(control, configuration, secrets)
+            initialize_control(control, experiment, configuration, secrets)
 
 
 def cleanup_controls(experiment: Experiment):
@@ -112,9 +114,14 @@ def validate_controls(experiment: Experiment):
             validate_python_control(c)
 
 
-def initialize_global_controls(settings: Settings):
+def initialize_global_controls(experiment: Experiment,
+                               configuration: Configuration, secrets: Secrets,
+                               settings: Settings):
     """
-    Load and initialize controls declared in the settings
+    Load and initialize controls declared in the settings.
+
+    Notice, if a control fails during its initialization, it is not registered
+    at all and will not be applied throughout the experiment.
     """
     controls = []
     for name, control in settings.get("controls", {}).items():
@@ -123,19 +130,30 @@ def initialize_global_controls(settings: Settings):
 
         provider = control.get("provider")
         if provider and provider["type"] == "python":
-            initialize_control(
-                control, configuration=None, secrets=None,
-                settings=settings)
+            try:
+                initialize_control(
+                    control, experiment=experiment,
+                    configuration=configuration, secrets=secrets,
+                    settings=settings)
+            except Exception:
+                logger.debug(
+                    "Control initialization '{}' failed. "
+                    "It will not be registered.".format(
+                        control['name']),
+                    exc_info=True)
+                # we don't use a control that failed its initialization
+                continue
+
         controls.append(control)
-    global_controls.set(controls)
+    set_global_controls(controls)
 
 
 def cleanup_global_controls():
     """
     Unload and cleanup global controls
     """
-    controls = global_controls.get()
-    global_controls.set([])
+    controls = get_global_controls()
+    reset_global_controls()
 
     for control in controls:
         name = control['name']
@@ -143,14 +161,19 @@ def cleanup_global_controls():
 
         provider = control.get("provider")
         if provider and provider["type"] == "python":
-            cleanup_control(control)
+            try:
+                cleanup_control(control)
+            except Exception:
+                logger.debug(
+                    "Control cleanup '{}' failed".format(control['name']),
+                    exc_info=True)
 
 
 def get_global_controls() -> List[ControlType]:
     """
     All the controls loaded from the settings.
     """
-    return global_controls.get()
+    return global_controls[:]
 
 
 class Control:
@@ -216,6 +239,21 @@ def get_controls(experiment: Experiment) -> List[Control]:
     return controls
 
 
+def set_global_controls(controls: List[ControlType]):
+    """
+    Set the controls loaded from the settings.
+    """
+    global_controls.clear()
+    global_controls.extend(controls)
+
+
+def reset_global_controls():
+    """
+    Invalidate all loaded global controls.
+    """
+    global_controls.clear()
+
+
 def get_context_controls(level: str, experiment: Experiment,
                          context: Union[Activity, Experiment]) \
                          -> List[Control]:
@@ -226,7 +264,7 @@ def get_context_controls(level: str, experiment: Experiment,
     If a control is declared at the current level, do override it with an
     top-level ine.
     """
-    glbl_controls = global_controls.get()[:]
+    glbl_controls = get_global_controls()
     top_level_controls = experiment.get("controls", [])
     controls = context.get("controls", [])
     controls.extend(glbl_controls)
@@ -282,6 +320,7 @@ def apply_controls(level: str, experiment: Experiment,
     settings = get_loaded_settings() or None
     controls = get_context_controls(level, experiment, context)
     if not controls:
+        logger.debug("No controls to apply on '{}'".format(level))
         return
 
     for control in controls:
diff --git a/chaoslib/control/python.py b/chaoslib/control/python.py
index c67b0fc..a746625 100644
--- a/chaoslib/control/python.py
+++ b/chaoslib/control/python.py
@@ -28,7 +28,8 @@ _level_mapping = {
 }
 
 
-def initialize_control(control: Control, configuration: Configuration,
+def initialize_control(control: Control, experiment: Experiment,
+                       configuration: Configuration,
                        secrets: Secrets, settings: Settings = None):
     """
     Initialize a control by calling its `configure_control` function.
@@ -40,6 +41,9 @@ def initialize_control(control: Control, configuration: Configuration,
     arguments = {}
     sig = inspect.signature(func)
 
+    if "experiment" in sig.parameters:
+        arguments["experiment"] = experiment
+
     if "secrets" in sig.parameters:
         arguments["secrets"] = secrets
 
diff --git a/chaoslib/experiment.py b/chaoslib/experiment.py
index 00f1429..9f1d7ab 100644
--- a/chaoslib/experiment.py
+++ b/chaoslib/experiment.py
@@ -11,7 +11,8 @@ from chaoslib import __version__
 from chaoslib.activity import ensure_activity_is_valid, run_activities
 from chaoslib.caching import with_cache, lookup_activity
 from chaoslib.control import initialize_controls, controls, cleanup_controls, \
-    validate_controls, Control
+    validate_controls, Control, initialize_global_controls, \
+    cleanup_global_controls
 from chaoslib.deprecation import warn_about_deprecated_features
 from chaoslib.exceptions import ActivityFailed, ChaosException, \
     InterruptExecution, InvalidActivity, InvalidExperiment
@@ -22,9 +23,11 @@ from chaoslib.hypothesis import ensure_hypothesis_is_valid, \
 from chaoslib.loader import load_experiment
 from chaoslib.rollback import run_rollbacks
 from chaoslib.secret import load_secrets
-from chaoslib.types import Configuration, Experiment, Journal, Run, Secrets
-
+from chaoslib.settings import get_loaded_settings
+from chaoslib.types import Configuration, Experiment, Journal, Run, Secrets, \
+    Settings
 
+initialize_global_controls
 __all__ = ["ensure_experiment_is_valid", "run_experiment", "load_experiment"]
 
 
@@ -151,7 +154,8 @@ def get_background_pools(experiment: Experiment) -> ThreadPoolExecutor:
 
 
 @with_cache
-def run_experiment(experiment: Experiment) -> Journal:
+def run_experiment(experiment: Experiment,
+                   settings: Settings = None) -> Journal:
     """
     Run the given `experiment` method step by step, in the following sequence:
     steady probe, action, close probe.
@@ -179,8 +183,10 @@ def run_experiment(experiment: Experiment) -> Journal:
         logger.warning("Dry mode enabled")
 
     started_at = time.time()
+    settings = settings if settings is not None else get_loaded_settings()
     config = load_configuration(experiment.get("configuration", {}))
     secrets = load_secrets(experiment.get("secrets", {}), config)
+    initialize_global_controls(experiment, config, secrets, settings)
     initialize_controls(experiment, config, secrets)
     activity_pool, rollback_pool = get_background_pools(experiment)
 
@@ -188,81 +194,85 @@ def run_experiment(experiment: Experiment) -> Journal:
     journal = initialize_run_journal(experiment)
 
     try:
-        control.begin("experiment", experiment, experiment, config, secrets)
-        # this may fail the entire experiment right there if any of the
-        # probes fail or fall out of their tolerance zone
         try:
-            state = run_steady_state_hypothesis(
-                experiment, config, secrets, dry=dry)
-            journal["steady_states"]["before"] = state
-            if state is not None and not state["steady_state_met"]:
-                p = state["probes"][-1]
-                raise ActivityFailed(
-                    "Steady state probe '{p}' is not in the given "
-                    "tolerance so failing this experiment".format(
-                        p=p["activity"]["name"]))
-        except ActivityFailed as a:
-            journal["steady_states"]["before"] = state
-            journal["status"] = "failed"
-            logger.fatal(str(a))
-        else:
+            control.begin(
+                "experiment", experiment, experiment, config, secrets)
+            # this may fail the entire experiment right there if any of the
+            # probes fail or fall out of their tolerance zone
             try:
-                journal["run"] = apply_activities(
-                    experiment, config, secrets, activity_pool, dry)
-            except Exception:
-                journal["status"] = "aborted"
-                logger.fatal(
-                    "Experiment ran into an un expected fatal error, "
-                    "aborting now.", exc_info=True)
+                state = run_steady_state_hypothesis(
+                    experiment, config, secrets, dry=dry)
+                journal["steady_states"]["before"] = state
+                if state is not None and not state["steady_state_met"]:
+                    p = state["probes"][-1]
+                    raise ActivityFailed(
+                        "Steady state probe '{p}' is not in the given "
+                        "tolerance so failing this experiment".format(
+                            p=p["activity"]["name"]))
+            except ActivityFailed as a:
+                journal["steady_states"]["before"] = state
+                journal["status"] = "failed"
+                logger.fatal(str(a))
             else:
                 try:
-                    state = run_steady_state_hypothesis(
-                        experiment, config, secrets, dry=dry)
-                    journal["steady_states"]["after"] = state
-                    if state is not None and not state["steady_state_met"]:
-                        journal["deviated"] = True
-                        p = state["probes"][-1]
-                        raise ActivityFailed(
-                            "Steady state probe '{p}' is not in the given "
-                            "tolerance so failing this experiment".format(
-                                p=p["activity"]["name"]))
-                except ActivityFailed as a:
-                    journal["status"] = "failed"
-                    logger.fatal(str(a))
-    except InterruptExecution as i:
-        journal["status"] = "interrupted"
-        logger.fatal(str(i))
-    except (KeyboardInterrupt, SystemExit):
-        journal["status"] = "interrupted"
-        logger.warn("Received an exit signal, "
-                    "leaving without applying rollbacks.")
-    else:
-        journal["status"] = journal["status"] or "completed"
-        journal["rollbacks"] = apply_rollbacks(
-            experiment, config, secrets, rollback_pool, dry)
-
-    journal["end"] = datetime.utcnow().isoformat()
-    journal["duration"] = time.time() - started_at
-
-    has_deviated = journal["deviated"]
-    status = "deviated" if has_deviated else journal["status"]
-
-    logger.info(
-        "Experiment ended with status: {s}".format(s=status))
-
-    if has_deviated:
+                    journal["run"] = apply_activities(
+                        experiment, config, secrets, activity_pool, dry)
+                except Exception:
+                    journal["status"] = "aborted"
+                    logger.fatal(
+                        "Experiment ran into an un expected fatal error, "
+                        "aborting now.", exc_info=True)
+                else:
+                    try:
+                        state = run_steady_state_hypothesis(
+                            experiment, config, secrets, dry=dry)
+                        journal["steady_states"]["after"] = state
+                        if state is not None and not state["steady_state_met"]:
+                            journal["deviated"] = True
+                            p = state["probes"][-1]
+                            raise ActivityFailed(
+                                "Steady state probe '{p}' is not in the given "
+                                "tolerance so failing this experiment".format(
+                                    p=p["activity"]["name"]))
+                    except ActivityFailed as a:
+                        journal["status"] = "failed"
+                        logger.fatal(str(a))
+        except InterruptExecution as i:
+            journal["status"] = "interrupted"
+            logger.fatal(str(i))
+        except (KeyboardInterrupt, SystemExit):
+            journal["status"] = "interrupted"
+            logger.warn("Received an exit signal, "
+                        "leaving without applying rollbacks.")
+        else:
+            journal["status"] = journal["status"] or "completed"
+            journal["rollbacks"] = apply_rollbacks(
+                experiment, config, secrets, rollback_pool, dry)
+
+        journal["end"] = datetime.utcnow().isoformat()
+        journal["duration"] = time.time() - started_at
+
+        has_deviated = journal["deviated"]
+        status = "deviated" if has_deviated else journal["status"]
+
         logger.info(
-            "The steady-state has deviated, a weakness may have been "
-            "discovered")
+            "Experiment ended with status: {s}".format(s=status))
 
-    control.with_state(journal)
+        if has_deviated:
+            logger.info(
+                "The steady-state has deviated, a weakness may have been "
+                "discovered")
 
-    try:
-        control.end("experiment", experiment, experiment, config, secrets)
-    except ChaosException:
-        pass
+        control.with_state(journal)
+
+        try:
+            control.end("experiment", experiment, experiment, config, secrets)
+        except ChaosException:
+            logger.debug("Failed to close controls", exc_info=True)
 
-    cleanup_controls(experiment)
+    finally:
+        cleanup_controls(experiment)
+        cleanup_global_controls()
 
     return journal
 


Test patch:
diff --git a/tests/fixtures/controls/dummy.py b/tests/fixtures/controls/dummy.py
index 6ad71a3..84c56ee 100644
--- a/tests/fixtures/controls/dummy.py
+++ b/tests/fixtures/controls/dummy.py
@@ -1,24 +1,19 @@
-import os
-from typing import Any, Callable, Dict, List, Union
+from typing import Any, Dict, List
 
 from chaoslib.types import Activity, Configuration, \
     Experiment, Hypothesis, Journal, Run, Secrets, Settings
 
-value_from_config = None
 
-
-def configure_control(configuration: Configuration, secrets: Secrets,
-                      settings: Settings):
-    global value_from_config
+def configure_control(experiment: Experiment, configuration: Configuration,
+                      secrets: Secrets, settings: Settings):
     if configuration:
-        value_from_config = configuration.get("dummy-key", "default")
+        experiment["control-value"] = configuration.get("dummy-key", "default")
     elif settings:
-        value_from_config = settings.get("dummy-key", "default")
+        experiment["control-value"] = settings.get("dummy-key", "default")
 
 
 def cleanup_control():
-    global value_from_config
-    value_from_config = None
+    pass
 
 
 def before_experiment_control(context: Experiment, **kwargs):
diff --git a/tests/fixtures/controls/dummy_with_failing_cleanup.py b/tests/fixtures/controls/dummy_with_failing_cleanup.py
new file mode 100644
index 0000000..b39ce9a
--- /dev/null
+++ b/tests/fixtures/controls/dummy_with_failing_cleanup.py
@@ -0,0 +1,2 @@
+def cleanup_control():
+    raise RuntimeError("cleanup control")
diff --git a/tests/fixtures/controls/dummy_with_failing_init.py b/tests/fixtures/controls/dummy_with_failing_init.py
new file mode 100644
index 0000000..1b7a698
--- /dev/null
+++ b/tests/fixtures/controls/dummy_with_failing_init.py
@@ -0,0 +1,10 @@
+from chaoslib.types import Configuration, Experiment, Secrets, Settings
+
+
+def configure_control(experiment: Experiment, configuration: Configuration,
+                      secrets: Secrets, settings: Settings):
+    raise RuntimeError("init control")
+
+
+def before_experiment_control(context: Experiment, **kwargs):
+    context["should_never_been_called"] = True
diff --git a/tests/fixtures/experiments.py b/tests/fixtures/experiments.py
index 4a031ec..5bc7a65 100644
--- a/tests/fixtures/experiments.py
+++ b/tests/fixtures/experiments.py
@@ -213,19 +213,9 @@ ExperimentWithVariousTolerances = {
 }
 
 
-
-ExperimentWithControls = {
+ExperimentNoControls = {
     "title": "do cats live in the Internet?",
     "description": "an experiment of importance",
-    "controls": [
-        {
-            "name": "dummy",
-            "provider": {
-                "type": "python",
-                "module": "fixtures.controls.dummy"
-            }
-        }
-    ],
     "steady-state-hypothesis": {
         "title": "hello",
         "probes": [
@@ -241,6 +231,23 @@ ExperimentWithControls = {
 }
 
 
+ExperimentWithControls = deepcopy(ExperimentNoControls)
+ExperimentWithControls["controls"] = [
+    {
+        "name": "dummy",
+        "provider": {
+            "type": "python",
+            "module": "fixtures.controls.dummy"
+        }
+    }
+]
+
+ExperimentUsingConfigToConfigureControls = deepcopy(ExperimentNoControls)
+ExperimentUsingConfigToConfigureControls["configuration"] = {
+    "dummy-key": "blah blah"
+}
+
+
 ExperimentWithControlsAtVariousLevels = deepcopy(ExperimentWithControls)
 ExperimentWithControlsAtVariousLevels["method"][0]["controls"] = [
         {
diff --git a/tests/test_control.py b/tests/test_control.py
index 62503f7..3a23703 100644
--- a/tests/test_control.py
+++ b/tests/test_control.py
@@ -25,22 +25,23 @@ from fixtures.controls import dummy as DummyControl
 
 
 def test_initialize_controls_will_configure_a_control():
+    exp = deepcopy(experiments.ExperimentWithControls)
     initialize_controls(
-        experiments.ExperimentWithControls, configuration={
+        exp, configuration={
             "dummy-key": "dummy-value"
         })
-    assert DummyControl.value_from_config == "dummy-value"
-    cleanup_controls(experiments.ExperimentWithControls)
+    assert exp["control-value"] == "dummy-value"
+    cleanup_controls(exp)
 
 
 def test_initialize_controls_will_cleanup_a_control():
-    cleanup_controls(experiments.ExperimentWithControls)
-    assert DummyControl.value_from_config == None
+    exp = deepcopy(experiments.ExperimentWithControls)
+    cleanup_controls(exp)
 
 
 def test_controls_are_applied_before_and_after_experiment():
     exp = deepcopy(experiments.ExperimentWithControls)
-    with controls("experiment", exp, context=exp) as c:
+    with controls("experiment", exp, context=exp):
         assert "before_experiment_control" in exp
         assert exp["before_experiment_control"] is True
 
@@ -55,12 +56,12 @@ def test_controls_are_applied_before_and_after_experiment():
 def test_controls_are_applied_before_and_but_not_after_experiment():
     exp = deepcopy(experiments.ExperimentWithControls)
     exp["controls"][0]["scope"] = "before"
-    with controls("experiment", exp, context=exp) as c:
+    with controls("experiment", exp, context=exp):
         assert "before_experiment_control" in exp
         assert exp["before_experiment_control"] is True
 
         exp["dry"] = True
-        journal = run_experiment(exp)
+        run_experiment(exp)
 
     assert "after_experiment_control" not in exp
 
@@ -68,7 +69,7 @@ def test_controls_are_applied_before_and_but_not_after_experiment():
 def test_controls_are_applied_not_before_and_but_after_experiment():
     exp = deepcopy(experiments.ExperimentWithControls)
     exp["controls"][0]["scope"] = "after"
-    with controls("experiment", exp, context=exp) as c:
+    with controls("experiment", exp, context=exp):
         assert "before_experiment_control" not in exp
 
         exp["dry"] = True
@@ -81,7 +82,7 @@ def test_controls_are_applied_not_before_and_but_after_experiment():
 
 def test_controls_may_interrupt_experiment():
     exp = deepcopy(experiments.ExperimentCanBeInterruptedByControl)
-    with controls("experiment", exp, context=exp) as c:
+    with controls("experiment", exp, context=exp):
         exp["dry"] = True
         journal = run_experiment(exp)
         assert journal["status"] == "interrupted"
@@ -90,7 +91,7 @@ def test_controls_may_interrupt_experiment():
 def test_controls_are_applied_before_and_after_hypothesis():
     exp = deepcopy(experiments.ExperimentWithControls)
     hypo = exp["steady-state-hypothesis"]
-    with controls("hypothesis", exp, context=hypo) as c:
+    with controls("hypothesis", exp, context=hypo):
         assert "before_hypothesis_control" in hypo
         assert hypo["before_hypothesis_control"] is True
 
@@ -104,7 +105,7 @@ def test_controls_are_applied_before_and_after_hypothesis():
 
 def test_controls_are_applied_before_and_after_method():
     exp = deepcopy(experiments.ExperimentWithControls)
-    with controls("method", exp, context=exp) as c:
+    with controls("method", exp, context=exp):
         assert "before_method_control" in exp
         assert exp["before_method_control"] is True
 
@@ -118,7 +119,7 @@ def test_controls_are_applied_before_and_after_method():
 
 def test_controls_are_applied_before_and_after_rollbacks():
     exp = deepcopy(experiments.ExperimentWithControls)
-    with controls("rollback", exp, context=exp) as c:
+    with controls("rollback", exp, context=exp):
         assert "before_rollback_control" in exp
         assert exp["before_rollback_control"] is True
 
@@ -136,7 +137,7 @@ def test_controls_are_applied_before_and_after_activities():
 
     activities = get_all_activities(exp)
     for activity in activities:
-        with controls("activity", exp, context=activity) as c:
+        with controls("activity", exp, context=activity):
             assert activity["before_activity_control"] is True
 
             run = execute_activity(exp, activity, None, None, dry=False)
@@ -150,11 +151,11 @@ def test_no_controls_get_applied_when_none_defined():
     exp = deepcopy(experiments.ExperimentWithoutControls)
     exp["dry"] = True
 
-    with controls("experiment", exp, context=exp) as c:
+    with controls("experiment", exp, context=exp):
         assert "before_experiment_control" not in exp
 
         exp["dry"] = True
-        journal = run_experiment(exp)
+        run_experiment(exp)
 
     assert "after_experiment_control" not in exp
 
@@ -277,20 +278,16 @@ def test_controls_are_applied_when_they_are_not_top_level():
 
 
 def test_load_global_controls_from_settings():
-    exp = deepcopy(experiments.ExperimentWithControls)
+    exp = deepcopy(experiments.ExperimentNoControls)
+    activities = get_all_activities(exp)
 
-    try:
-        run_experiment(exp)
-        activities = get_all_activities(exp)
-        for activity in activities:
-            if "controls" in activity:
-                assert "before_activity_control" not in activity
-                assert "after_activity_control" not in activity
-    finally:
-        cleanup_global_controls()
+    for activity in activities:
+        assert "before_activity_control" not in activity
+        assert "after_activity_control" not in activity
 
-    initialize_global_controls({
-        "dummy-key": "blah",
+    assert get_global_controls() == []
+    run_experiment(exp, settings={
+        "dummy-key": "hello there",
         "controls": {
             "dummy": {
                 "provider": {
@@ -300,22 +297,20 @@ def test_load_global_controls_from_settings():
             }
         }
     })
+    assert get_global_controls() == []
+    assert exp["control-value"] == "hello there"
 
-    try:
-        run_experiment(exp)
-        activities = get_all_activities(exp)
-        for activity in activities:
-            if "controls" in activity:
-                assert activity["before_activity_control"] is True
-                assert activity["after_activity_control"] is True
-    finally:
-        cleanup_global_controls()
+    for activity in activities:
+        assert "before_activity_control" in activity
+        assert "after_activity_control" in activity
+        assert activity["before_activity_control"] is True
+        assert activity["after_activity_control"] is True
 
 
 def test_get_globally_loaded_controls_from_settings():
     assert get_global_controls() == []
 
-    initialize_global_controls({
+    initialize_global_controls({}, {}, {}, {
         "controls": {
             "dummy": {
                 "provider": {
@@ -335,3 +330,132 @@ def test_get_globally_loaded_controls_from_settings():
     finally:
         cleanup_global_controls()
         assert get_global_controls() == []
+
+
+def test_load_global_controls_from_settings_configured_via_exp_config():
+    exp = deepcopy(experiments.ExperimentUsingConfigToConfigureControls)
+    activities = get_all_activities(exp)
+
+    for activity in activities:
+        assert "before_activity_control" not in activity
+        assert "after_activity_control" not in activity
+
+    assert get_global_controls() == []
+    run_experiment(exp, settings={
+        "controls": {
+            "dummy": {
+                "provider": {
+                    "type": "python",
+                    "module": "fixtures.controls.dummy"
+                }
+            }
+        }
+    })
+    assert get_global_controls() == []
+    assert exp["control-value"] == "blah blah"
+
+    for activity in activities:
+        assert "before_activity_control" in activity
+        assert "after_activity_control" in activity
+        assert activity["before_activity_control"] is True
+        assert activity["after_activity_control"] is True
+
+
+def test_apply_controls_even_on_background_activity():
+    exp = deepcopy(experiments.ExperimentNoControls)
+    exp["method"][0]["background"] = True
+    exp["method"][0]["pauses"] = {
+        "after": 1
+    }
+    activities = get_all_activities(exp)
+
+    for activity in activities:
+        assert "before_activity_control" not in activity
+        assert "after_activity_control" not in activity
+
+    assert get_global_controls() == []
+    run_experiment(exp, settings={
+        "dummy-key": "hello there",
+        "controls": {
+            "dummy": {
+                "provider": {
+                    "type": "python",
+                    "module": "fixtures.controls.dummy"
+                }
+            }
+        }
+    })
+    assert get_global_controls() == []
+    assert exp["control-value"] == "hello there"
+
+    for activity in activities:
+        assert "before_activity_control" in activity
+        assert "after_activity_control" in activity
+        assert activity["before_activity_control"] is True
+        assert activity["after_activity_control"] is True
+
+
+def test_control_cleanup_cannot_fail_the_experiment():
+    exp = deepcopy(experiments.ExperimentNoControls)
+    try:
+        run_experiment(exp, settings={
+            "dummy-key": "hello there",
+            "controls": {
+                "dummy": {
+                    "provider": {
+                        "type": "python",
+                        "module": "fixtures.controls.dummy_with_failing_cleanup"
+                    }
+                }
+            }
+        })
+    except:
+        pytest.fail("Failed to run experiment with a broken cleanup control")
+
+
+def test_control_initialization_cannot_fail_the_experiment():
+    exp = deepcopy(experiments.ExperimentNoControls)
+    try:
+        run_experiment(exp, settings={
+            "dummy-key": "hello there",
+            "controls": {
+                "dummy": {
+                    "provider": {
+                        "type": "python",
+                        "module": "fixtures.controls.dummy_with_failing_init"
+                    }
+                }
+            }
+        })
+    except:
+        pytest.fail("Failed to run experiment with a broken init control")
+
+
+def test_control_failing_its_initialization_must_not_be_registered():
+    exp = deepcopy(experiments.ExperimentNoControls)
+    run_experiment(exp, settings={
+        "dummy-key": "hello there",
+        "controls": {
+            "dummy-failed": {
+                "provider": {
+                    "type": "python",
+                    "module": "fixtures.controls.dummy_with_failing_init"
+                }
+            },
+            "dummy": {
+                "provider": {
+                    "type": "python",
+                    "module": "fixtures.controls.dummy"
+                }
+            }
+        }
+    })
+
+    assert "should_never_been_called" not in exp
+
+    activities = get_all_activities(exp)
+    for activity in activities:
+        assert "before_activity_control" in activity
+        assert "after_activity_control" in activity
+        assert activity["before_activity_control"] is True
+        assert activity["after_activity_control"] is True