Repository: fitbenchmarking/fitbenchmarking

Issue:
New option: maximum time
Some softwares that have been added lately have a long runtime. We should have an option to limit the amount of time any minimizer can take.

This could be an option that just takes the maximum number of seconds, or it could take a string like "100s", "1m,40s","2h,15m",etc.

Discussion:
Anders-Markvardsen: Here are some first test results using WSL2 ubuntu 18.04. Running `fitbenchmarking -o examples/options_template_anders.ini -p examples/benchmark_problems/NIST/average_difficulty/` with max_runtime=600 and num_runs=1 gives

![image](https://user-images.githubusercontent.com/1135476/127751207-4a30ca83-9ec4-435f-8276-cc3242421f94.png)

with max_runtime=0.1 and num_runs=1 gives

![image](https://user-images.githubusercontent.com/1135476/127751233-ef2eb4e5-5006-449f-b2f2-8e9b1f2f3535.png)

with max_runtime=0.1 and num_runs=5 gives

![image](https://user-images.githubusercontent.com/1135476/127751249-b9e64c00-73b7-44ef-a29a-b355504356d3.png)

Hence looks good so far


Anders-Markvardsen: Testing on Windows. Something is not quite working with the timings on Windows. Running again `fitbenchmarking -o examples/options_template_anders.ini -p examples/benchmark_problems/NIST/average_difficulty/ with num_runs=1. Setting max_runtime=600 or max_runtime=2.5 both have all runs completing and tables displaying as expected, e.g. 

<img src="https://user-images.githubusercontent.com/1135476/127768297-6713ea34-231f-44da-8931-759fd350d142.png" width="250" height="250">

the table above suggest setting max_runtime=0.1 should have some passing fine, however none are with this value - hence this is a bug.

Secondly, picking instead e.g. max_runtime=1.1 now some are passing through

<img src="https://user-images.githubusercontent.com/1135476/127768366-75d5b66e-a603-4a8f-bfbe-4a4f6b2fa467.png" width="250" height="250">

As can be by comparing the numbers in the first row in the above two table, with max_runtime=1.1 all though be passing through but only some are. Maybe `timeit` time thinks by default different on windows?


jess-farmer: Currently when running Mantid or SASView problems, I am finding that every minimizer (other than `mantid`) times out, no matter what `max_runtime` is set to. It seems that the call to `controller.fit` within this line:
```
runtime_list.extend(timeit.Timer(
                                                setup=controller.prepare,
                                                stmt=controller.fit
                                                ).repeat(num_runs, 1))
```
is not running properly and hangs until the max runtime is exceeded.

@Anders-Markvardsen - I think the tables you showed at the meeting yesterday were for SASView problems and run from this branch, did you experience any similar problems to the above, or if you run the same command again following my recent changes, are you still able to produce the same results? I have also removed the serialization of the controller, which we suspected was causing the timing problems on Windows, please could you test if this change results in an improvement in the timings for Windows?
Anders-Markvardsen: @jess-farmer will repeat. About running against mantid not done this yet but in the process. Will aim to join the stand-up today
jess-farmer: @Anders-Markvardsen thanks - I'm unfortunately not able to make today's stand up so if you could also let me know your findings on here/send me a message on slack that would be much appreciated.
Anders-Markvardsen: @jess-farmer no worries. I have repeating the Windows. With max_runtime = 0.1 no run completed and with max_runtime = 1.1 observe

![image](https://user-images.githubusercontent.com/1135476/128323953-6965d402-ecda-47bd-990e-027ee456c44b.png)

hence appears the timing less accurate. Thanks
Anders-Markvardsen: Here are more observation using Windows with commits as of this point in time. max_runtime = 600

![image](https://user-images.githubusercontent.com/1135476/128333507-53b8eff2-f46d-4d09-9b7e-65eeeef1d116.png)

max_runtime = 2 then all passes and

![image](https://user-images.githubusercontent.com/1135476/128334665-c6bd7753-79ce-4858-85c9-6e1a9786aba6.png)

max_runtime = 1.5 then

![image](https://user-images.githubusercontent.com/1135476/128335546-63336892-8b5b-4008-bd66-813c4f77b57a.png)

so timings with this tech solution similar-ish in terms of windows interpret run_maxtime to above 





Anders-Markvardsen: On ubuntu 18.04 Sasview still appear to run fine for me. With max_runtime = 3:

![image](https://user-images.githubusercontent.com/1135476/128336542-7c68a93e-dfd0-4b55-b697-ad7fd48936d5.png)

jess-farmer: @Anders-Markvardsen many thanks for testing these scenarios and sharing your observations
AndrewLister-STFC: This works for me on WSL and the code all looks very neat. I'll hold off on a review until the tests are up and running again though.
robertapplin: Thanks for testing @AndrewLister-STFC , the tests are running ok now
tyronerees: @Anders-Markvardsen  -- would you be able to check this does what you need?  Andrew's been through the code, and is happy with it, so I think we just need a quick functional approval from you.
Anders-Markvardsen: Here are some observation running this on Windows, and with new implementation it should be agnostic to platform. Running `examples\benchmark_problems\NIST\average_difficulty` with a high max_runtime gives

<img src="https://user-images.githubusercontent.com/1135476/134373855-ac22ddbb-a679-4d8f-aa67-f6bf34466485.png" width="400" height="150">

Selecting max_runtime to 2.1 and 1.1 make no difference to above, which means, as we know the value of max_runtime may not match exactly runtime in tables. Picking value of 0.1 gives

<img src="https://user-images.githubusercontent.com/1135476/134374809-e0a7cc5f-b60b-4d71-a0ad-c30d6c1286f2.png" width="400" height="150">

which is a sufficiently acceptable result in that the effect of max_runtime generally stopped those jobs which where running the longest





Patch:
diff --git a/docs/source/users/options/fitting_option.rst b/docs/source/users/options/fitting_option.rst
index 1e56aeed..ecba3125 100644
--- a/docs/source/users/options/fitting_option.rst
+++ b/docs/source/users/options/fitting_option.rst
@@ -162,3 +162,25 @@ Default is ``weighted_nlls``
 
     [FITTING]
     cost_func_type: weighted_nlls
+
+Maximum Runtime (:code:`max_runtime`)
+--------------------------------------
+
+This sets the maximum runtime a minimizer has to solve one benchmark
+problem `num_runs` number of times, where `num_runs` is another option a
+user can set. If the minimizer is still running after the maximum time
+has elapsed, then this result will be skipped and FitBenchmarking will move
+on to the next minimizer / benchmark dataset combination. The main purpose
+of this option is to get to result tables quicker by limit the runtime.
+
+`max_runtime` is set by specifying a number in unit of seconds. Please note
+that depending on platform the time specified with `max_runtime` may not
+match entirely with the absolute run-times specified in tables. Hence you
+may have to experiment a bit with this option to get the cutoff you want.
+
+Default is 600 seconds
+
+.. code-block:: rst
+
+    [FITTING]
+    max_runtime: 600
diff --git a/examples/options_template.ini b/examples/options_template.ini
index 300ae403..76870e8f 100644
--- a/examples/options_template.ini
+++ b/examples/options_template.ini
@@ -6,6 +6,7 @@ software: bumps
 	  minuit
 
 num_runs: 1
+max_runtime: 600
 
 [MINIMIZERS]
 
diff --git a/fitbenchmarking/controllers/base_controller.py b/fitbenchmarking/controllers/base_controller.py
index 8442964c..fc9d57d5 100644
--- a/fitbenchmarking/controllers/base_controller.py
+++ b/fitbenchmarking/controllers/base_controller.py
@@ -19,7 +19,7 @@ class Controller:
 
     __metaclass__ = ABCMeta
 
-    VALID_FLAGS = [0, 1, 2, 3, 4, 5]
+    VALID_FLAGS = [0, 1, 2, 3, 4, 5, 6]
 
     #: Within the controller class, you must
     #: initialize a dictionary, ``algorithm_check``,
@@ -140,6 +140,9 @@ class Controller:
         # in the fitting software have support for bounds
         self.support_for_bounds = False
 
+        # The timer used to check if the 'max_runtime' is exceeded.
+        self.timer = cost_func.problem.timer
+
     @property
     def flag(self):
         """
@@ -149,6 +152,7 @@ class Controller:
         | 3: `Software raised an exception`
         | 4: `Solver doesn't support bounded problems`
         | 5: `Solution doesn't respect parameter bounds`
+        | 6: `Solver has exceeded maximum allowed runtime`
         """
 
         return self._flag
@@ -190,6 +194,15 @@ class Controller:
             raise ControllerAttributeError('Either minimizer or parameter_set '
                                            'is set to None.')
 
+    def execute(self):
+        """
+        Starts and stops the timer used to check if the fit reaches
+        the 'max_runtime'. In the middle, it calls self.fit().
+        """
+        self.timer.start()
+        self.fit()
+        self.timer.stop()
+
     def eval_chisq(self, params, x=None, y=None, e=None):
         """
         Computes the chisq value
diff --git a/fitbenchmarking/core/fitting_benchmarking.py b/fitbenchmarking/core/fitting_benchmarking.py
index e0939eea..14727a84 100644
--- a/fitbenchmarking/core/fitting_benchmarking.py
+++ b/fitbenchmarking/core/fitting_benchmarking.py
@@ -22,6 +22,7 @@ from fitbenchmarking.utils import fitbm_result, misc, output_grabber
 from fitbenchmarking.utils.exceptions import (FitBenchmarkException,
                                               ControllerAttributeError,
                                               IncompatibleMinimizerError,
+                                              MaxRuntimeError,
                                               NoJacobianError,
                                               NoHessianError,
                                               UnknownMinimizerError,
@@ -496,11 +497,12 @@ def loop_over_hessians(controller, options, minimizer_name,
 
         try:
             with grabbed_output:
+                controller.timer.reset()
                 # Calls timeit repeat with repeat = num_runs and
                 # number = 1
                 runtime_list = timeit.Timer(
                     setup=controller.prepare,
-                    stmt=controller.fit
+                    stmt=controller.execute
                 ).repeat(num_runs, 1)
 
                 runtime = sum(runtime_list) / num_runs
@@ -531,26 +533,27 @@ def loop_over_hessians(controller, options, minimizer_name,
                 raise ControllerAttributeError(
                     "Either the computed runtime or chi_sq values "
                     "was a NaN.")
-        # Catching all exceptions as this means runtime cannot be
-        # calculated
-        # pylint: disable=broad-except
-        except Exception as excp:
-            LOGGER.warning(str(excp))
+        except MaxRuntimeError as ex:
+            LOGGER.warning(str(ex))
+            controller.flag = 6
+        except Exception as ex:  # pylint: disable=broad-except
+            LOGGER.warning(str(ex))
+            controller.flag = 3
 
+        if controller.flag in [3, 6]:
+            # If there was an exception, set the runtime and
+            # cost function value to be infinite
             runtime = np.inf
-            controller.flag = 3
             controller.final_params = \
                 None if not problem.multifit \
                 else [None] * len(controller.data_x)
 
             chi_sq = np.inf if not problem.multifit \
                 else [np.inf] * len(controller.data_x)
-
-        # If bounds have been set, check that they have
-        # been respected by the minimizer and set error
-        # flag if not
-        if controller.problem.value_ranges is not None \
-                and controller.flag != 3:
+        elif controller.problem.value_ranges is not None:
+            # If bounds have been set, check that they have
+            # been respected by the minimizer and set error
+            # flag if not
             controller.check_bounds_respected()
 
         # record algorithm type for specified minimizer
diff --git a/fitbenchmarking/parsing/fitting_problem.py b/fitbenchmarking/parsing/fitting_problem.py
index 72bc0561..7670b36c 100644
--- a/fitbenchmarking/parsing/fitting_problem.py
+++ b/fitbenchmarking/parsing/fitting_problem.py
@@ -14,6 +14,7 @@ import numpy as np
 
 from fitbenchmarking.utils.exceptions import FittingProblemError, \
     IncorrectBoundsError
+from fitbenchmarking.utils.timer import TimerWithMaxTime
 
 
 # Using property getters and setters means that the setter does not always use
@@ -104,6 +105,9 @@ class FittingProblem:
         #: Callable function for the Hessian
         self.hessian = None
 
+        # The timer used to check if the 'max_runtime' is exceeded.
+        self.timer = TimerWithMaxTime(self.options.max_runtime)
+
     def eval_model(self, params, **kwargs):
         """
         Function evaluation method
@@ -117,6 +121,9 @@ class FittingProblem:
         if self.function is None:
             raise FittingProblemError('Cannot call function before setting '
                                       'function.')
+
+        self.timer.check_elapsed_time()
+
         x = kwargs.get("x", self.data_x)
         return self.function(x, *params)
 
diff --git a/fitbenchmarking/results_processing/runtime_table.py b/fitbenchmarking/results_processing/runtime_table.py
index 20565038..1086c3e6 100644
--- a/fitbenchmarking/results_processing/runtime_table.py
+++ b/fitbenchmarking/results_processing/runtime_table.py
@@ -9,9 +9,9 @@ from fitbenchmarking.results_processing.base_table import Table
 class RuntimeTable(Table):
     """
 
-    The timing results are calculated from an average using the
+    The timing results are calculated from an average (over num_runs) using the
     `timeit <https://docs.python.org/2/library/timeit.html>`_  module
-    in python. The number of runtimes can be set in :ref:`options`.
+    in python. num_runs is set in :ref:`options`.
 
     """
 
diff --git a/fitbenchmarking/results_processing/tables.py b/fitbenchmarking/results_processing/tables.py
index 911c3da1..18eadd8e 100644
--- a/fitbenchmarking/results_processing/tables.py
+++ b/fitbenchmarking/results_processing/tables.py
@@ -21,7 +21,8 @@ ERROR_OPTIONS = {0: "Successfully converged",
                  2: "Software run but didn't converge to solution",
                  3: "Software raised an exception",
                  4: "Solver doesn't support bounded problems",
-                 5: "Solution doesn't respect parameter bounds"}
+                 5: "Solution doesn't respect parameter bounds",
+                 6: "Solver has exceeded maximum allowed runtime"}
 
 SORTED_TABLE_NAMES = ["compare", "acc", "runtime", "local_min"]
 
diff --git a/fitbenchmarking/utils/exceptions.py b/fitbenchmarking/utils/exceptions.py
index 0074c58e..fa9c5a96 100644
--- a/fitbenchmarking/utils/exceptions.py
+++ b/fitbenchmarking/utils/exceptions.py
@@ -205,3 +205,11 @@ class NoHessianError(FitBenchmarkException):
     """
     class_message = 'Could not find Hessian class'
     error_code = 22
+
+
+class MaxRuntimeError(FitBenchmarkException):
+    """
+    Indicates a minimizer has taken too long to run
+    """
+    class_message = 'Minimizer runtime exceeded maximum runtime'
+    error_code = 23
diff --git a/fitbenchmarking/utils/options.py b/fitbenchmarking/utils/options.py
index 852a3547..76c22653 100644
--- a/fitbenchmarking/utils/options.py
+++ b/fitbenchmarking/utils/options.py
@@ -128,7 +128,8 @@ class Options:
          'software': ['scipy', 'scipy_ls'],
          'jac_method': ['scipy'],
          'hes_method': ['default'],
-         'cost_func_type': 'weighted_nlls'}
+         'cost_func_type': 'weighted_nlls',
+         'max_runtime': 600}
     DEFAULT_JACOBIAN = \
         {'analytic': ['cutest'],
          'scipy': ['2-point'],
@@ -212,6 +213,7 @@ class Options:
         self.jac_method = self.read_value(fitting.getlist, 'jac_method')
         self.hes_method = self.read_value(fitting.getlist, 'hes_method')
         self.cost_func_type = self.read_value(fitting.getstr, 'cost_func_type')
+        self.max_runtime = self.read_value(fitting.getfloat, 'max_runtime')
 
         jacobian = config['JACOBIAN']
         self.num_method = {}
@@ -323,7 +325,8 @@ class Options:
                              'algorithm_type': list_to_string(
                                  self.algorithm_type),
                              'software': list_to_string(self.software),
-                             'jac_method': list_to_string(self.jac_method)}
+                             'jac_method': list_to_string(self.jac_method),
+                             'max_runtime': self.max_runtime}
         config['JACOBIAN'] = {k: list_to_string(m)
                               for k, m in self.num_method.items()}
 
diff --git a/fitbenchmarking/utils/timer.py b/fitbenchmarking/utils/timer.py
new file mode 100644
index 00000000..e867066c
--- /dev/null
+++ b/fitbenchmarking/utils/timer.py
@@ -0,0 +1,61 @@
+"""
+Implements the TimerWithMaxTime class used for checking the
+'max_runtime' is not exceeded.
+"""
+from time import time
+from fitbenchmarking.utils.exceptions import MaxRuntimeError
+
+
+class TimerWithMaxTime:
+    """
+    A timer class used for checking if the 'max_runtime' is exceeded when
+    executing the fits.
+    """
+
+    def __init__(self, max_runtime: float):
+        """
+        Initialise the timer used for checking whether or not the
+        'max_runtime' option is exceeded while processing the fits.
+
+        :param max_runtime: The maximum allowed runtime in seconds.
+        :type max_runtime: float`
+        """
+        self.max_runtime: float = max_runtime
+        self.total_elapsed_time: float = 0.0
+        self.start_time: float = None
+
+    def start(self) -> None:
+        """
+        Starts the timer by recording the current time.
+        """
+        self.start_time = time()
+
+    def stop(self) -> None:
+        """
+        Stops the timer if it is timing something. The elapsed time
+        since starting the timer is added onto the total elapsed time.
+        """
+        if self.start_time is not None:
+            self.total_elapsed_time += time() - self.start_time
+            self.start_time = None
+
+    def reset(self) -> None:
+        """
+        Resets the timer so it can be used for timing a different fit
+        combination.
+        """
+        self.total_elapsed_time = 0.0
+        self.start_time = None
+
+    def check_elapsed_time(self) -> None:
+        """
+        Checks whether the max runtime has been exceeded. Raises a
+        MaxRuntimeError exception if it has been exceeded. Otherwise,
+        it carries on.
+        """
+        is_timing = self.start_time is not None
+        active_elapsed_time = time() - self.start_time if is_timing else 0.0
+
+        if self.total_elapsed_time + active_elapsed_time > self.max_runtime:
+            self.reset()
+            raise MaxRuntimeError


Test patch:
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_hessians.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_hessians.py
index a4543311..8c1fd32f 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_hessians.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_hessians.py
@@ -13,6 +13,7 @@ from fitbenchmarking.cost_func.nlls_cost_func import NLLSCostFunc
 from fitbenchmarking.parsing.parser_factory import parse_problem_file
 from fitbenchmarking.utils import output_grabber
 from fitbenchmarking.utils.options import Options
+from fitbenchmarking.utils.timer import TimerWithMaxTime
 
 # Due to construction of the controllers two folder functions
 # pylint: disable=unnecessary-pass
@@ -63,13 +64,16 @@ class DummyController(Controller):
         self.count = self.count % len(self.flag_expected)
 
 
-def make_cost_function(file_name='cubic.dat', minimizers=None):
+def make_cost_function(file_name='cubic.dat', minimizers=None,
+                       max_runtime=None):
     """
     Helper function that returns a simple fitting problem
     """
     options = Options()
     if minimizers:
         options.minimizers = minimizers
+    if max_runtime:
+        options.max_runtime = max_runtime
 
     bench_prob_dir = os.path.dirname(inspect.getfile(mock_problems))
     fname = os.path.join(bench_prob_dir, file_name)
@@ -172,6 +176,26 @@ class LoopOverHessiansTests(unittest.TestCase):
                                self.grabbed_output)
         check_bounds_respected.assert_not_called()
 
+    @patch.object(TimerWithMaxTime, 'reset', lambda *args: None)
+    def test_max_runtime_exceeded(self):
+        """
+        Test that the correct flag is set when the max_runtime is exceeded.
+        """
+        self.cost_func = make_cost_function(minimizers=self.minimizers,
+                                            max_runtime=0.1)
+        self.cost_func.problem.timer.total_elapsed_time = 5
+        self.controller = DummyController(cost_func=self.cost_func)
+        self.options = self.cost_func.problem.options
+        self.grabbed_output = output_grabber.OutputGrabber(self.options)
+        self.controller.parameter_set = 0
+
+        jacobian = False
+        self.controller.minimizer = "deriv_free_algorithm"
+        results, _, _ = loop_over_hessians(self.controller, self.options,
+                                           self.controller.minimizer,
+                                           jacobian, self.grabbed_output)
+        self.assertEqual(results[0]["error_flag"], 6)
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_jacobians.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_jacobians.py
index f2b7525d..7712eb68 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_jacobians.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_jacobians.py
@@ -99,7 +99,8 @@ class LoopOverJacobiansTests(unittest.TestCase):
         self.options = self.problem.options
         self.grabbed_output = output_grabber.OutputGrabber(self.options)
 
-    def mock_func_call(self, *args):
+    @staticmethod
+    def mock_func_call(*args):
         """
         Mock function to be used instead of loop_over_hessians
         """
diff --git a/fitbenchmarking/hessian/tests/test_hessian.py b/fitbenchmarking/hessian/tests/test_hessian.py
index b6394676..e2548e07 100644
--- a/fitbenchmarking/hessian/tests/test_hessian.py
+++ b/fitbenchmarking/hessian/tests/test_hessian.py
@@ -119,7 +119,6 @@ class TestHessianClass(TestCase):
         Setting up tests
         """
         options = Options()
-        self.fitting_problem = FittingProblem(options)
         options.cost_func_type = "nlls"
         self.fitting_problem = FittingProblem(options)
         self.fitting_problem.function = f_ls
diff --git a/fitbenchmarking/utils/tests/test_options_fitting.py b/fitbenchmarking/utils/tests/test_options_fitting.py
index 84e90a71..73da4f1a 100644
--- a/fitbenchmarking/utils/tests/test_options_fitting.py
+++ b/fitbenchmarking/utils/tests/test_options_fitting.py
@@ -69,6 +69,14 @@ class FittingOptionTests(unittest.TestCase):
         actual = self.options.cost_func_type
         self.assertEqual(expected, actual)
 
+    def test_max_runtime_default(self):
+        """
+        Checks max_rutime default
+        """
+        expected = 600
+        actual = self.options.max_runtime
+        self.assertEqual(expected, actual)
+
 
 class UserFittingOptionTests(unittest.TestCase):
     """
@@ -229,3 +237,20 @@ class UserFittingOptionTests(unittest.TestCase):
         config_str = \
             "[FITTING]\ncost_func_type: normal_dist"
         self.shared_invalid('cost_func_type', config_str)
+
+    def test_minimizer_max_runtime_type_valid(self):
+        """
+        Checks user set max_runtime is valid
+        """
+        set_option = 10
+        config_str = \
+            "[FITTING]\nmax_runtime: 10"
+        self.shared_valid('max_runtime', set_option, config_str)
+
+    def test_minimizer_max_runtime_invalid(self):
+        """
+        Checks user set max_runtime is invalid
+        """
+        config_str = \
+            "[FITTING]\nmax_runtime: 10 seconds"
+        self.shared_invalid('max_runtime', config_str)
diff --git a/fitbenchmarking/utils/tests/test_timer.py b/fitbenchmarking/utils/tests/test_timer.py
new file mode 100644
index 00000000..55d4b127
--- /dev/null
+++ b/fitbenchmarking/utils/tests/test_timer.py
@@ -0,0 +1,67 @@
+"""
+Tests for fitbenchmarking.utils.timer.TimerWithMaxTime
+"""
+from unittest import TestCase
+from time import sleep
+
+from fitbenchmarking.utils.exceptions import MaxRuntimeError
+from fitbenchmarking.utils.timer import TimerWithMaxTime
+
+
+class TimerWithMaxTimeTests(TestCase):
+    """
+    Tests for base software controller class methods.
+    """
+
+    def setUp(self):
+        self.max_runtime = 60.0
+        self.timer = TimerWithMaxTime(self.max_runtime)
+
+    def test_that_start_and_stop_will_record_the_expected_time_elapsed(self):
+        """
+        Tests that the elapsed time is greater than the sleep time.
+        """
+        sleep_time = 1.0
+
+        self.timer.start()
+        sleep(sleep_time)
+        self.timer.stop()
+
+        self.assertGreaterEqual(self.timer.total_elapsed_time, sleep_time)
+
+    def test_that_reset_will_reset_the_total_elapsed_time(self):
+        """
+        Test the timer data is reset as expected.
+        """
+        self.timer.total_elapsed_time = 5.5
+        self.timer.start_time = 1.2
+
+        self.timer.reset()
+
+        self.assertEqual(self.timer.total_elapsed_time, 0.0)
+        self.assertEqual(self.timer.start_time, None)
+
+    def test_that_stop_will_not_raise_even_if_the_timer_is_not_running(self):
+        """
+        Test that calling stop when the timer is not running will not cause
+        any problems.
+        """
+        self.timer.stop()
+
+    def test_check_elapsed_time_raises_if_the_max_runtime_is_exceeded(self):
+        """
+        Check the expected MaxRuntimeError exception is raised when
+        the max_runtime is exceeded.
+        """
+        sleep_time = 2.0
+        self.max_runtime = 1.0
+        self.timer = TimerWithMaxTime(self.max_runtime)
+
+        self.timer.start()
+        sleep(sleep_time)
+
+        with self.assertRaises(MaxRuntimeError):
+            self.timer.check_elapsed_time()
+
+        self.assertEqual(self.timer.total_elapsed_time, 0.0)
+        self.assertEqual(self.timer.start_time, None)