Repository: quaquel/EMAworkbench

Issue:
ValueError: cannot convert float to NaN
In recent weeks, various people have been complaining about getting a ValueError: cannot convert float to NaN. 

The error is triggerd in lines [285-288 in callback.py](https://github.com/quaquel/EMAworkbench/blob/1b5175d96953ed9bf466790dc9249dae6802bcb7/ema_workbench/em_framework/callbacks.py#L285-L288). Note that until https://github.com/quaquel/EMAworkbench/commit/ba4abbbbaa68d56ea604a242bb721fcd65b39afb, this same code was used in both the `__init__` and `_store_outcome`. Commit https://github.com/quaquel/EMAworkbench/commit/ba4abbbbaa68d56ea604a242bb721fcd65b39afb was just a first step towards issolating the problem and investigating it further.

What is strange about this error is that [lines 285-288 in callback.py](https://github.com/quaquel/EMAworkbench/blob/1b5175d96953ed9bf466790dc9249dae6802bcb7/ema_workbench/em_framework/callbacks.py#L285-L288) the workbench where it originates has been in the workbench since 27 March 2019 and has been part of the 2.0 release in April 2019. Until the last month or so, I have never seen this error before nor received questions about it. My hunch is that something in NumPy related to the reworking of dtypes is now triggering this error.

So what is going on? The code itself is actually wrong. If you initialize a NumPy array with any dtype other than float, you will get a ValleError because of np.NaN is a float and cannot be assigned to a non-float NumPy array. 

What is the motivation for this code? The key idea is to set up NumPy arrays in which to store the results and set some value on them that is instantly recognizable as a missing value if a given experiment fails. A simple use case is when using VensimModel. If an experiment triggers a floating-point error in Vensim, this raises a CaseError, but the experiments will continue. Thus there can be failed experiments, and the purpose of using NaNs is to make this clear if you return to stored results after a while. However, since np.NaN is a float; it cannot serve as a general solution.

So what other solutions do we have for this problem?
1. have some custom flag for floats, integer, and object that is used by the workbench to flag missing values. The problem with this is that seperate flags will be needed for any conceivable dtype. Moreover, what to pick as a sensible flag? 
2. Use [numpy.ma](https://numpy.org/doc/stable/reference/maskedarray.generic.html). Numpy.ma provides masked arrays that act as normal numpy arrays but come with a mask for missing or invalid entries. The drawback is that this potentially requires a bigger code overhaul, depending on the desired behavior. Basically, how should the workbench handle failed experiments? 
3. Shift to pandas for storing the outcomes, and use [pd.NA](https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html?highlight=pd%20na). However, this is still an experimental feature and seems to be primarily added to pandas because of NumPy's lack of an integer equivalent for NaN. 

I am inclined toward solution 2, but I am not convinced yet. Also, there might be other possible solutions that I have not thought of yet. 

In my view, the workbench should be able to execute experiments and continue even if individual experiments fail. In particular, when scaling the workbench to use HPC resources, this is desirable behavior unless many experiments fail. 

However, once finished with the experiments, the workbench should make it easy to identify which experiments have failed. Having masked arrays makes this easy. Take the mask from a given outcome, apply it to the experiment's data frame, and you know which experiments failed. 

Ideally, the workbench would even log the failed experiments in some easy-to-read overview and do the same when loading results that contain masked elements in the outcomes. If no experiments fail, it is easy to unmask the outcomes, turning them into regular NumPy arrays. 

When using optimization, failed experiments should trigger a failure in the optimization. 





Discussion:
coveralls: 
[![Coverage Status](https://coveralls.io/builds/52706499/badge)](https://coveralls.io/builds/52706499)

Coverage increased (+0.1%) to 80.167% when pulling **67ab57fa8b2cc1668a3ee6e6a079a529cfa3eaec on masked_arrays** into **abfaafb3966a7464a6553a49eef25e0852b1f7bf on master**.

EwoutH: Nice work, I will take an in depth look after #170 is merged.

I must say, we are making quite some progress in only a few weeks!

Patch:
diff --git a/.coveragerc b/.coveragerc
deleted file mode 100644
index 7390043..0000000
--- a/.coveragerc
+++ /dev/null
@@ -1,20 +0,0 @@
-[run]
-source =
-    ema_workbench/em_framework/*
-    ema_workbench/analysis/*
-    ema_workbench/util/*
-
-[report]
-omit =
-    */python?.?/*
-    */site-packages/nose/*
-    */test/*
-    */examples/*
-
-# Regexes for lines to exclude from consideration
-exclude_lines =
-    # Have to re-enable the standard pragma
-    pragma: no cover
-
-    # Don't complain about missing debug-only code:
-    def __repr__
\ No newline at end of file
diff --git a/.gitignore b/.gitignore
index d0fef4e..ee9a5cc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -176,5 +176,16 @@ pip-log.txt
 #Mr Developer
 .mr.developer.cfg
 
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
 # Mac crap
 .DS_Store
diff --git a/ema_workbench/em_framework/callbacks.py b/ema_workbench/em_framework/callbacks.py
index 0daa397..74d8a94 100644
--- a/ema_workbench/em_framework/callbacks.py
+++ b/ema_workbench/em_framework/callbacks.py
@@ -41,18 +41,21 @@ class AbstractCallback(ProgressTrackingMixIn, metaclass=abc.ABCMeta):
 
     Parameters
     ----------
-    uncs : list
-            a list of the parameters over which the experiments
-            are being run.
+    uncertainties : list
+                    list of uncertain parameters
+    levers : list
+             list of lever parameters
     outcomes : list
                a list of outcomes
     nr_experiments : int
                      the total number of experiments to be executed
     reporting_interval : int, optional
-                         the interval at which to provide progress
-                         information via logging.
+                         the interval between progress logs
     reporting_frequency: int, optional
                          the total number of progress logs
+    log_progress : bool, optional
+                   if true, progress is logged, if false, use
+                   tqdm progress bar.
 
 
     Attributes
@@ -60,6 +63,10 @@ class AbstractCallback(ProgressTrackingMixIn, metaclass=abc.ABCMeta):
     i : int
         a counter that keeps track of how many experiments have been
         saved
+    nr_experiments: int
+    outcomes : list
+    parameters : list
+                 combined list of uncertain parameters and lever parameters
     reporting_interval : int,
                          the interval between progress logs
 
@@ -68,8 +75,8 @@ class AbstractCallback(ProgressTrackingMixIn, metaclass=abc.ABCMeta):
     def __init__(
         self,
         uncertainties,
-        outcomes,
         levers,
+        outcomes,
         nr_experiments,
         reporting_interval=None,
         reporting_frequency=10,
@@ -77,6 +84,11 @@ class AbstractCallback(ProgressTrackingMixIn, metaclass=abc.ABCMeta):
     ):
         super().__init__(nr_experiments, reporting_frequency, _logger, log_progress)
 
+        self.i = 0
+        self.nr_experiments = nr_experiments
+        self.outcomes = outcomes
+        self.parameters = uncertainties + levers
+
         if reporting_interval is None:
             reporting_interval = max(1, int(round(nr_experiments / reporting_frequency)))
 
@@ -111,14 +123,33 @@ class AbstractCallback(ProgressTrackingMixIn, metaclass=abc.ABCMeta):
 
 
 class DefaultCallback(AbstractCallback):
-    """
-    default callback system
-    callback can be used in perform_experiments as a means for
+    """Default callback class
+
+    Parameters
+    ----------
+    uncertainties : list
+                    list of uncertain parameters
+    levers : list
+             list of lever parameters
+    outcomes : list
+               a list of outcomes
+    nr_experiments : int
+                     the total number of experiments to be executed
+    reporting_interval : int, optional
+                         the interval between progress logs
+    reporting_frequency: int, optional
+                         the total number of progress logs
+    log_progress : bool, optional
+                   if true, progress is logged, if false, use
+                   tqdm progress bar.
+
+
+    Callback can be used in perform_experiments as a means for
     specifying the way in which the results should be handled. If no
     callback is specified, this default implementation is used. This
     one can be overwritten or replaced with a callback of your own
     design. For example if you prefer to store the result in a database
-    or write them to a text file
+    or write them to a text file.
     """
 
     shape_error_msg = "can only save up to 2d arrays, this array is {}d"
@@ -126,7 +157,7 @@ class DefaultCallback(AbstractCallback):
 
     def __init__(
         self,
-        uncs,
+        uncertainties,
         levers,
         outcomes,
         nr_experiments,
@@ -138,9 +169,10 @@ class DefaultCallback(AbstractCallback):
 
         Parameters
         ----------
-        uncs : list
-                a list of the parameters over which the experiments
-                are being run.
+        uncertainties : list
+                        list of uncertain parameters
+        levers : list
+                 list of lever parameters
         outcomes : list
                    a list of outcomes
         nr_experiments : int
@@ -155,7 +187,7 @@ class DefaultCallback(AbstractCallback):
 
         """
         super().__init__(
-            uncs,
+            uncertainties,
             levers,
             outcomes,
             nr_experiments,
@@ -163,25 +195,22 @@ class DefaultCallback(AbstractCallback):
             reporting_frequency,
             log_progress,
         )
-        self.i = 0
+
         self.cases = None
         self.results = {}
-        self.outcomes = outcomes
 
         # determine data types of parameters
         columns = []
         dtypes = []
-        self.parameters = []
 
-        for parameter in uncs + levers:
+        for parameter in self.parameters:
             name = parameter.name
-            self.parameters.append(name)
             dtype = "float"
 
-            if isinstance(parameter, CategoricalParameter):
-                dtype = "object"
-            elif isinstance(parameter, BooleanParameter):
+            if isinstance(parameter, BooleanParameter):
                 dtype = "bool"
+            elif isinstance(parameter, CategoricalParameter):
+                dtype = "object"
             elif isinstance(parameter, IntegerParameter):
                 dtype = "int"
             columns.append(name)
@@ -191,7 +220,6 @@ class DefaultCallback(AbstractCallback):
             columns.append(name)
             dtypes.append("object")
 
-        # FIXME:: issue with fragmented data frame warning
         index = np.arange(nr_experiments)
         column_dict = {
             name: pd.Series(dtype=dtype, index=index) for name, dtype in zip(columns, dtypes)
@@ -199,9 +227,8 @@ class DefaultCallback(AbstractCallback):
         df = pd.concat(column_dict, axis=1).copy()
 
         self.cases = df
-        self.nr_experiments = nr_experiments
 
-        for outcome in outcomes:
+        for outcome in self.outcomes:
             shape = outcome.shape
             if shape is not None:
                 shape = (nr_experiments,) + shape
@@ -234,7 +261,9 @@ class DefaultCallback(AbstractCallback):
                 _logger.debug(message)
             else:
                 try:
-                    self.results[outcome][case_id] = outcome_res
+                    self.results[outcome][
+                        case_id,
+                    ] = outcome_res
                 except KeyError:
                     data = np.asarray(outcome_res)
 
@@ -248,7 +277,9 @@ class DefaultCallback(AbstractCallback):
                     shape.insert(0, self.nr_experiments)
 
                     self.results[outcome] = self._setup_outcomes_array(shape, data.dtype)
-                    self.results[outcome][case_id] = outcome_res
+                    self.results[outcome][
+                        case_id,
+                    ] = outcome_res
 
     def __call__(self, experiment, outcomes):
         """
@@ -271,41 +302,55 @@ class DefaultCallback(AbstractCallback):
         self._store_outcomes(experiment.experiment_id, outcomes)
 
     def get_results(self):
-        return self.cases, self.results
+        results = {}
+        for k, v in self.results.items():
+            if not np.ma.is_masked(v):
+                results[k] = v.data
+            else:
+                _logger.warning("some experiments have failed, returning masked result arrays")
+                results[k] = v
+
+        return self.cases, results
 
     def _setup_outcomes_array(self, shape, dtype):
-        array = np.empty(shape, dtype=dtype)
-        array[:] = np.nan
+        array = np.ma.empty(shape, dtype=dtype)
+        array.mask = True
         return array
 
 
 class FileBasedCallback(AbstractCallback):
-    """
-    Callback that stores data in csv files while running
+    """Callback that stores data in csv files while running th model
 
     Parameters
     ----------
-    uncs : collection of Parameter instances
-    levers : collection of Parameter instances
-    outcomes : collection of Outcome instances
+    uncertainties : list
+                    list of uncertain parameters
+    levers : list
+             list of lever parameters
+    outcomes : list
+               a list of outcomes
     nr_experiments : int
+                     the total number of experiments to be executed
     reporting_interval : int, optional
-    reporting_frequency : int, optional
+                         the interval between progress logs
+    reporting_frequency: int, optional
+                         the total number of progress logs
+    log_progress : bool, optional
+                   if true, progress is logged, if false, use
+                   tqdm progress bar.
 
+    Warnings
+    --------
+    This class is still in beta.
     the data is stored in ./temp, relative to the current
     working directory. If this directory already exists, it will be
     overwritten.
 
-    Warnings
-    --------
-    This class is still in beta. API is expected to change over the
-    coming months.
-
     """
 
     def __init__(
         self,
-        uncs,
+        uncertainties,
         levers,
         outcomes,
         nr_experiments,
@@ -313,7 +358,7 @@ class FileBasedCallback(AbstractCallback):
         reporting_frequency=10,
     ):
         super().__init__(
-            uncs,
+            uncertainties,
             levers,
             outcomes,
             nr_experiments,
@@ -321,11 +366,6 @@ class FileBasedCallback(AbstractCallback):
             reporting_frequency=reporting_frequency,
         )
 
-        self.i = 0
-        self.nr_experiments = nr_experiments
-        self.outcomes = [outcome.name for outcome in outcomes]
-        self.parameters = [parameter.name for parameter in uncs + levers]
-
         self.directory = os.path.abspath("./temp")
         if os.path.exists(self.directory):
             shutil.rmtree(self.directory)
@@ -333,13 +373,15 @@ class FileBasedCallback(AbstractCallback):
 
         self.experiments_fh = open(os.path.join(self.directory, "experiments.csv"), "w")
 
-        header = self.parameters + ["scenario_id", "policy", "model"]
+        # write experiments.csv header row
+        header = [p.name for p in self.parameters] + ["scenario_id", "policy", "model"]
         writer = csv.writer(self.experiments_fh)
         writer.writerow(header)
 
         self.outcome_fhs = {}
         for outcome in self.outcomes:
-            self.outcome_fhs[outcome] = open(os.path.join(self.directory, outcome + ".csv"), "w")
+            name = outcome.name
+            self.outcome_fhs[name] = open(os.path.join(self.directory, f"{name}.csv"), "w")
 
     def _store_case(self, experiment):
         scenario = experiment.scenario
@@ -347,11 +389,12 @@ class FileBasedCallback(AbstractCallback):
 
         case = []
         for parameter in self.parameters:
+            name = parameter.name
             try:
-                value = scenario[parameter]
+                value = scenario[name]
             except KeyError:
                 try:
-                    value = policy[parameter]
+                    value = policy[name]
                 except KeyError:
                     value = np.nan
             finally:
@@ -366,14 +409,15 @@ class FileBasedCallback(AbstractCallback):
 
     def _store_outcomes(self, outcomes):
         for outcome in self.outcomes:
-            data = outcomes[outcome]
+            name = outcome.name
+            data = outcomes[name]
 
             try:
                 data = [str(entry) for entry in data]
             except TypeError:
                 data = [str(data)]
 
-            fh = self.outcome_fhs[outcome]
+            fh = self.outcome_fhs[name]
             writer = csv.writer(fh)
             writer.writerow(data)
 
@@ -402,7 +446,7 @@ class FileBasedCallback(AbstractCallback):
         # TODO:: metadata
 
         self.experiments_fh.close()
-        for value in self.outcome_fhs.items():
+        for value in self.outcome_fhs.values():
             value.close()
 
         return self.directory


Test patch:
diff --git a/test/test_em_framework/test_callback.py b/test/test_em_framework/test_callback.py
index a17afdd..aa5ceaa 100644
--- a/test/test_em_framework/test_callback.py
+++ b/test/test_em_framework/test_callback.py
@@ -4,190 +4,267 @@ Created on 22 Jan 2013
 .. codeauthor:: jhkwakkel <j.h.kwakkel (at) tudelft (dot) nl>
 """
 import random
-import unittest
 import pytest
 
 import numpy as np
 
-from ema_workbench.em_framework.callbacks import DefaultCallback
+import ema_workbench.em_framework.callbacks as callbacks
+from ema_workbench.em_framework.callbacks import DefaultCallback, FileBasedCallback
 from ema_workbench.em_framework.parameters import (
     CategoricalParameter,
     RealParameter,
     IntegerParameter,
+    BooleanParameter,
 )
 from ema_workbench.em_framework.points import Policy, Scenario, Experiment
 from ema_workbench.util import EMAError
-from ema_workbench.em_framework.outcomes import ScalarOutcome, ArrayOutcome, TimeSeriesOutcome
+from ema_workbench.em_framework.outcomes import (
+    ScalarOutcome,
+    ArrayOutcome,
+    TimeSeriesOutcome,
+)
 from ema_workbench.em_framework.util import NamedObject
 
 
-class TestDefaultCallback(unittest.TestCase):
-    def test_init(self):
-        # let's add some uncertainties to this
-        uncs = [RealParameter("a", 0, 1), RealParameter("b", 0, 1)]
-        outcomes = [
-            ScalarOutcome("scalar"),
-            ArrayOutcome("array", shape=(10,)),
-            TimeSeriesOutcome("timeseries"),
-        ]
-        callback = DefaultCallback(uncs, [], outcomes, nr_experiments=100)
-
-        assert callback.i == 0
-        assert callback.nr_experiments == 100
-        assert callback.cases.shape[0] == 100
-        #         self.assertEqual(callback.outcomes, outcomes)
-
-        names = callback.cases.columns.values.tolist()
-        names = set(names)
-        assert names == {"a", "b", "policy", "model", "scenario"}
-
-        assert "scalar" not in callback.results
-        assert "timeseries" not in callback.results
-        assert "array" in callback.results
-
-        a = np.all(np.isnan(callback.results["array"]))
-        assert a
+def test_store_results(mocker):
+    nr_experiments = 3
+    uncs = [RealParameter("a", 0, 1), RealParameter("b", 0, 1)]
+    outcomes = [TimeSeriesOutcome("test")]
+    model = NamedObject("test")
 
-        # with levers
-        levers = [RealParameter("c", 0, 10)]
+    experiment = Experiment(0, model, Policy("policy"), Scenario(a=1, b=0), 0)
 
-        callback = DefaultCallback(uncs, levers, outcomes, nr_experiments=100)
+    # case 1 scalar shape = (1)
+    callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
+    model_outcomes = {outcomes[0].name: 1.0}
+    callback(experiment, model_outcomes)
 
-        assert callback.i == 0
-        assert callback.nr_experiments == 100
-        assert callback.cases.shape[0] == 100
-        #         self.assertEqual(callback.outcomes, [o.name for o in outcomes])
+    _, out = callback.get_results()
 
-        names = callback.cases.columns.values.tolist()
-        names = set(names)
-        assert names == {"a", "b", "c", "policy", "model", "scenario"}
+    assert outcomes[0].name in set(out.keys())
+    assert out[outcomes[0].name].shape == (3,)
 
-        assert "scalar" not in callback.results
-        assert "timeseries" not in callback.results
-        assert "array" in callback.results
+    # case 2 time series shape = (1, nr_time_steps)
+    callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
+    model_outcomes = {outcomes[0].name: np.random.rand(10)}
+    callback(experiment, model_outcomes)
 
-        a = np.all(np.isnan(callback.results["array"]))
-        assert a
+    _, out = callback.get_results()
+    assert outcomes[0].name in out.keys()
+    assert out[outcomes[0].name].shape == (3, 10)
 
-    def test_store_results(self):
-        nr_experiments = 3
-        uncs = [RealParameter("a", 0, 1), RealParameter("b", 0, 1)]
-        outcomes = [TimeSeriesOutcome("test")]
-        model = NamedObject("test")
+    # case 3 maps etc. shape = (x,y)
+    callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
+    model_outcomes = {outcomes[0].name: np.random.rand(2, 2)}
+    callback(experiment, model_outcomes)
 
-        experiment = Experiment(0, model, Policy("policy"), Scenario(a=1, b=0), 0)
+    _, out = callback.get_results()
+    assert outcomes[0].name in out.keys()
+    assert out[outcomes[0].name].shape == (3, 2, 2)
 
-        # case 1 scalar shape = (1)
-        callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
-        model_outcomes = {outcomes[0].name: 1.0}
+    # case 4 assert raises EMAError
+    callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
+    model_outcomes = {outcomes[0].name: np.random.rand(2, 2, 2)}
+    with pytest.raises(EMAError):
         callback(experiment, model_outcomes)
 
-        _, out = callback.get_results()
-
-        assert outcomes[0].name in set(out.keys())
-        assert out[outcomes[0].name].shape == (3,)
-
-        # case 2 time series shape = (1, nr_time_steps)
-        callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
-        model_outcomes = {outcomes[0].name: np.random.rand(10)}
-        callback(experiment, model_outcomes)
-
-        _, out = callback.get_results()
-        assert outcomes[0].name in out.keys()
-        assert out[outcomes[0].name].shape == (3, 10)
-
-        # case 3 maps etc. shape = (x,y)
-        callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
-        model_outcomes = {outcomes[0].name: np.random.rand(2, 2)}
-        callback(experiment, model_outcomes)
-
-        _, out = callback.get_results()
-        assert outcomes[0].name in out.keys()
-        assert out[outcomes[0].name].shape == (3, 2, 2)
-
-        # case 4 assert raises EMAError
-        callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
-        model_outcomes = {outcomes[0].name: np.random.rand(2, 2, 2)}
-        with pytest.raises(EMAError):
-            callback(experiment, model_outcomes)
-
-    #         # KeyError
-    #         with mock.patch('ema_workbench.util.ema_logging.debug') as mocked_logging:
-    #             callback = DefaultCallback(uncs, [], outcomes,
-    #                                        nr_experiments=nr_experiments)
-    #             model_outcomes = {'incorrect': np.random.rand(2,)}
-    #             callback(experiment, model_outcomes)
-    #
-    #             for outcome in outcomes:
-    #                 mocked_logging.assert_called_with("%s not specified as outcome in msi" % outcome.name)
-
-    def test_store_cases(self):
-        nr_experiments = 3
-        uncs = [
-            RealParameter("a", 0, 1),
-            RealParameter("b", 0, 1),
-            CategoricalParameter("c", [0, 1, 2]),
-            IntegerParameter("d", 0, 1),
-        ]
-        outcomes = [TimeSeriesOutcome("test")]
-        case = {unc.name: random.random() for unc in uncs}
-        case["c"] = int(round(case["c"] * 2))
-        case["d"] = int(round(case["d"]))
-
-        model = NamedObject("test")
-        policy = Policy("policy")
-        scenario = Scenario(**case)
-        experiment = Experiment(0, model.name, policy, scenario, 0)
-
-        callback = DefaultCallback(
-            uncs, [], outcomes, nr_experiments=nr_experiments, reporting_interval=1
-        )
-        model_outcomes = {outcomes[0].name: 1.0}
-        callback(experiment, model_outcomes)
-
-        experiments, _ = callback.get_results()
-        design = case
-        design["policy"] = policy.name
-        design["model"] = model.name
-        design["scenario"] = scenario.name
-
-        names = experiments.columns.values.tolist()
-        for name in names:
-            entry_a = experiments[name][0]
-            entry_b = design[name]
-
-            assert entry_a == entry_b, "failed for " + name
-
-        # with levers
-        nr_experiments = 3
-        uncs = [RealParameter("a", 0, 1), RealParameter("b", 0, 1)]
-        levers = [RealParameter("c", 0, 1), RealParameter("d", 0, 1)]
-        outcomes = [TimeSeriesOutcome("test")]
-        case = {unc.name: random.random() for unc in uncs}
-
-        model = NamedObject("test")
-        policy = Policy("policy", c=1, d=1)
-        scenario = Scenario(**case)
-        experiment = Experiment(0, model.name, policy, scenario, 0)
-
-        callback = DefaultCallback(
-            uncs, levers, outcomes, nr_experiments=nr_experiments, reporting_interval=1
-        )
-        model_outcomes = {outcomes[0].name: 1.0}
-        callback(experiment, model_outcomes)
-
-        experiments, _ = callback.get_results()
-        design = case
-        design["c"] = 1
-        design["d"] = 1
-        design["policy"] = policy.name
-        design["model"] = model.name
-        design["scenario"] = scenario.name
-
-        names = experiments.columns.values.tolist()
-        for name in names:
-            assert experiments[name][0] == design[name]
-
-
-if __name__ == "__main__":
-    unittest.main()
+    # case 5 assert raises KeyError
+    callback = DefaultCallback(uncs, [], outcomes, nr_experiments=nr_experiments)
+    model_outcomes = {"some_other_name": np.random.rand(2, 2, 2)}
+    mock = mocker.patch("ema_workbench.em_framework.callbacks._logger.debug")
+    callback._store_outcomes(1, model_outcomes)
+    assert mock.call_count == 2
+
+
+def test_init():
+    # let's add some uncertainties to this
+    uncs = [RealParameter("a", 0, 1), RealParameter("b", 0, 1)]
+    outcomes = [
+        ScalarOutcome("scalar"),
+        ArrayOutcome("array", shape=(10,)),
+        TimeSeriesOutcome("timeseries"),
+    ]
+    callback = DefaultCallback(uncs, [], outcomes, nr_experiments=100)
+
+    assert callback.i == 0
+    assert callback.nr_experiments == 100
+    assert callback.cases.shape[0] == 100
+    assert callback.reporting_interval == 100
+    #         self.assertEqual(callback.outcomes, outcomes)
+
+    names = callback.cases.columns.values.tolist()
+    names = set(names)
+    assert names == {"a", "b", "policy", "model", "scenario"}
+
+    assert "scalar" not in callback.results
+    assert "timeseries" not in callback.results
+    assert "array" in callback.results
+    assert np.ma.is_masked(callback.results["array"])
+
+    # with levers
+    levers = [RealParameter("c", 0, 10)]
+
+    callback = DefaultCallback(
+        uncs,
+        levers,
+        outcomes,
+        nr_experiments=1000,
+        reporting_interval=None,
+        reporting_frequency=4,
+    )
+
+    assert callback.i == 0
+    assert callback.nr_experiments == 1000
+    assert callback.cases.shape[0] == 1000
+    assert callback.reporting_interval == 250
+    #         self.assertEqual(callback.outcomes, [o.name for o in outcomes])
+
+    names = callback.cases.columns.values.tolist()
+    names = set(names)
+    assert names == {"a", "b", "c", "policy", "model", "scenario"}
+
+    assert "scalar" not in callback.results
+    assert "timeseries" not in callback.results
+    assert "array" in callback.results
+    assert np.ma.is_masked(callback.results["array"])
+
+
+#         # KeyError
+#         with mock.patch('ema_workbench.util.ema_logging.debug') as mocked_logging:
+#             callback = DefaultCallback(uncs, [], outcomes,
+#                                        nr_experiments=nr_experiments)
+#             model_outcomes = {'incorrect': np.random.rand(2,)}
+#             callback(experiment, model_outcomes)
+#
+#             for outcome in outcomes:
+#                 mocked_logging.assert_called_with("%s not specified as outcome in msi" % outcome.name)
+
+
+def test_store_cases():
+    nr_experiments = 3
+    uncs = [
+        RealParameter("a", 0, 1),
+        RealParameter("b", 0, 1),
+        CategoricalParameter("c", [0, 1, 2]),
+        IntegerParameter("d", 0, 1),
+        BooleanParameter("e"),
+    ]
+    outcomes = [TimeSeriesOutcome("test")]
+    case = {unc.name: random.random() for unc in uncs}
+    case["c"] = int(round(case["c"] * 2))
+    case["d"] = int(round(case["d"]))
+
+    model = NamedObject("test")
+    policy = Policy("policy")
+    scenario = Scenario(**case)
+    experiment = Experiment(0, model.name, policy, scenario, 0)
+
+    callback = DefaultCallback(
+        uncs, [], outcomes, nr_experiments=nr_experiments, reporting_interval=1
+    )
+    model_outcomes = {outcomes[0].name: 1.0}
+    callback(experiment, model_outcomes)
+
+    experiments, _ = callback.get_results()
+    design = case
+    design["policy"] = policy.name
+    design["model"] = model.name
+    design["scenario"] = scenario.name
+
+    names = experiments.columns.values.tolist()
+    for name in names:
+        entry_a = experiments[name][0]
+        entry_b = design[name]
+
+        assert entry_a == entry_b, "failed for " + name
+
+    # with levers
+    nr_experiments = 3
+    uncs = [RealParameter("a", 0, 1), RealParameter("b", 0, 1)]
+    levers = [RealParameter("c", 0, 1), RealParameter("d", 0, 1)]
+    outcomes = [TimeSeriesOutcome("test")]
+    case = {unc.name: random.random() for unc in uncs}
+
+    model = NamedObject("test")
+    policy = Policy("policy", c=1, d=1)
+    scenario = Scenario(**case)
+    experiment = Experiment(0, model.name, policy, scenario, 0)
+
+    callback = DefaultCallback(
+        uncs, levers, outcomes, nr_experiments=nr_experiments, reporting_interval=1
+    )
+    model_outcomes = {outcomes[0].name: 1.0}
+    callback(experiment, model_outcomes)
+
+    experiments, _ = callback.get_results()
+    design = case
+    design["c"] = 1
+    design["d"] = 1
+    design["policy"] = policy.name
+    design["model"] = model.name
+    design["scenario"] = scenario.name
+
+    names = experiments.columns.values.tolist()
+    for name in names:
+        assert experiments[name][0] == design[name]
+
+
+def test_get_results(mocker):
+    nr_experiments = 3
+    uncs = [
+        RealParameter("a", 0, 1),
+    ]
+    outcomes = [ScalarOutcome("other_test")]
+    outcomes[0].shape = (1,)
+
+    callback = DefaultCallback(
+        uncs, [], outcomes, nr_experiments=nr_experiments, reporting_interval=1
+    )
+
+    # test warning
+    mock = mocker.patch("ema_workbench.em_framework.callbacks._logger.warning")
+    callback.get_results()
+    assert mock.call_count == 1
+
+    # test without warning
+    mock = mocker.patch("ema_workbench.em_framework.callbacks._logger.warning")
+    callback.results = {k: v.data for k, v in callback.results.items()}
+    callback.get_results()
+    assert mock.call_count == 0
+
+
+def test_filebasedcallback(mocker):
+    # only most basic assertions are checked
+
+    mock_os = mocker.patch("ema_workbench.em_framework.callbacks.os")
+    mock_shutil = mocker.patch("ema_workbench.em_framework.callbacks.shutil")
+    mock_open = mocker.patch("builtins.open")
+
+    nr_experiments = 3
+    uncs = [
+        RealParameter("a", 0, 1),
+        RealParameter("c", 0, 1),
+    ]
+    levers = [
+        RealParameter("b", 0, 1),
+    ]
+
+    outcomes = [ScalarOutcome("other"), TimeSeriesOutcome("time")]
+
+    model = NamedObject("test")
+    scenario = Scenario(**{"a": random.random()})
+    policy = Policy(name="policy", **{"b": random.random()})
+    experiment = Experiment(0, model.name, policy, scenario, 0)
+
+    callback = FileBasedCallback(
+        uncs, levers, outcomes, nr_experiments=nr_experiments, reporting_interval=1
+    )
+
+    # we should have opened 3 files: experiments and 2 outcome files
+    assert mock_open.call_count == 3
+    assert "other" in callback.outcome_fhs.keys()
+    assert "time" in callback.outcome_fhs.keys()
+
+    callback(experiment, {"other": 1, "time": [1, 2, 3, 4]})
+
+    callback.get_results()