Repository: fitbenchmarking/fitbenchmarking

Issue:
Azure test incorrect report
**Description of the error**
The Windows tests in Azure say they pass when they actually fail.
![failed_tests](https://user-images.githubusercontent.com/15614849/88036165-15f35900-cb3b-11ea-9010-0298591a0cf8.png)

**Describe the expected result**
Azure correctly reports passes and fails



Discussion:
Anders-Markvardsen: Windows build fails on python 3.6 with error on `import mock` which makes sense since mock refer to pre python 3.3 and from python 3.3 onward unittest.mock available, and external mock not included requirements.txt. @wathen what am I missing, how come line line 9 of `core/tests/test_fitting_benchmarking_benchmark.py` is `import mock` works on linux builds. Separately how come mock used and not unittest.mock?
wathen: > Windows build fails on python 3.6 with error on `import mock` which makes sense since mock refer to pre python 3.3 and from python 3.3 onward unittest.mock available, and external mock not included requirements.txt. @wathen what am I missing, how come line line 9 of `core/tests/test_fitting_benchmarking_benchmark.py` is `import mock` works on linux builds. Separately how come mock used and not unittest.mock?

Hmm, I'm not sure why but if `unittest.mock` works the same as `mock` then we might as well just use that instead?
Anders-Markvardsen: @wathen thanks and looks good with the changes. Separate to this PR I wondering if we should create a follow up issue to investigate the small differences in output, e.g. these are for all_parsers.txt:

```
                                bumps                    dfo                 minuit                     scipy             scipy_ls
                               amoeba                  dfogn                 minuit               Nelder-Mead      lm-scipy-no-jac
cubic, Start 1  1.119e-14 (2.899e+07)  5.244e-14 (1.358e+08)  3.586e-11 (9.288e+10)  6.267e-05 (1.623e+17)[1]        3.861e-22 (1)
cubic, Start 2  1.146e-14 (2.969e+07)  2.424e-17 (6.278e+04)  7.579e-06 (1.963e+16)     7.176e-11 (1.859e+11)        3.861e-22 (1)
Problem Def 1   6.784e+05 (5.225e+15)          1.299e-10 (1)       4.214e-07 (3245)      3.837e-09 (29.55)[1]  0.03966 (3.055e+08)

                                bumps                    dfo                 minuit                     scipy             scipy_ls
                               amoeba                  dfogn                 minuit               Nelder-Mead      lm-scipy-no-jac
cubic, Start 1  1.119e-14 (2.899e+07)  7.196e-14 (1.864e+08)  2.352e-07 (6.091e+14)  6.267e-05 (1.623e+17)[1]        3.861e-22 (1)
cubic, Start 2  1.146e-14 (2.969e+07)  1.312e-14 (3.397e+07)  7.579e-06 (1.963e+16)     7.176e-11 (1.859e+11)        3.861e-22 (1)
Problem Def 1   6.784e+05 (5.225e+15)          1.299e-10 (1)       4.214e-07 (3245)      3.837e-09 (29.55)[1]  0.03966 (3.055e+08)

```

Maybe we can briefly discuss in out weekly fitbenchmarking catch up tomorrow

Patch:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 229fd6e2..32d6dd95 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -11,8 +11,6 @@ strategy:
   matrix:
     Python36:
       python.version: '3.6'
-    Python37:
-      python.version: '3.7'
 
 steps:
 - task: UsePythonVersion@0
@@ -21,15 +19,20 @@ steps:
   displayName: 'Use Python $(python.version)'
 
 - script: |
-    pip install .
-  displayName: 'Install FitBenchmarking'
+    pip install -r requirements.txt
+    pip install .[bumps,DFO,minuit,SAS]
+  displayName: 'Install FitBenchmarking and requirements'
 
 - script: |
-    pip install pytest pytest-azurepipelines
-    pytest fitbenchmarking\utils
-    pytest fitbenchmarking\core
-    pytest fitbenchmarking\results_processing
+    pytest fitbenchmarking\results_processing --test-type default
+    pytest fitbenchmarking\utils --test-type default
+    pytest fitbenchmarking\core --test-type default
+    pytest fitbenchmarking\results_processing --test-type default
     pytest fitbenchmarking\parsing --test-type default
     pytest fitbenchmarking\controllers --test-type default
-    pytest fitbenchmarking\cli\tests
-  displayName: 'pytest utils tests'
+    pytest fitbenchmarking\cli --test-type default
+  displayName: 'unit testing'
+
+- script: |
+    pytest fitbenchmarking\systests --test-type default
+  displayName: 'system testing'
diff --git a/fitbenchmarking/utils/options.py b/fitbenchmarking/utils/options.py
index 58d83b4b..d650efa9 100644
--- a/fitbenchmarking/utils/options.py
+++ b/fitbenchmarking/utils/options.py
@@ -243,12 +243,12 @@ class Options(object):
     def results_dir(self, value):
         self._results_dir = os.path.abspath(value)
 
-    def write(self, file_name):
+    def _create_config(self):
         """
-        Write the contents of the options object to a new options file.
+        Return the contents of the options object as a ConfigParser object,
+        which e.g. then can be written to a file object or other stream
 
-        :param file_name: The path to the new options file
-        :type file_name: str
+        :return: ConfigParser
         """
         config = configparser.ConfigParser(converters={'list': read_list,
                                                        'str': str})
@@ -278,9 +278,28 @@ class Options(object):
                              'append': self.log_append,
                              'external_output': self.external_output}
 
+        return config
+
+    def write(self, file_name):
+        """
+        Write the contents of the options object to a new options file.
+
+        :param file_name: The path to the new options file
+        :type file_name: str
+        """
+        config = self._create_config()
+
         with open(file_name, 'w') as f:
             config.write(f)
 
+    def write_to_stream(self, file_object):
+        """
+        Write the contents of the options object to a file object.
+
+        :type file_object: file object
+        """
+        self._create_config().write(file_object)
+
 
 def read_list(s):
     """


Test patch:
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark.py
index 43743a80..7b9ce63e 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark.py
@@ -6,7 +6,7 @@ import inspect
 import copy
 import os
 import unittest
-import mock
+from unittest import mock
 
 from fitbenchmarking import mock_problems
 from fitbenchmarking.utils import fitbm_result
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark_problems.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark_problems.py
index e56d758b..33836cf5 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark_problems.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_benchmark_problems.py
@@ -6,7 +6,6 @@ from __future__ import (absolute_import, division, print_function)
 import inspect
 import os
 import unittest
-import mock
 
 from fitbenchmarking import mock_problems
 from fitbenchmarking.utils import fitbm_result, exceptions
@@ -110,7 +109,7 @@ class LoopOverBenchmarkProblemsTests(unittest.TestCase):
         dict_test(unselected_minimzers, {"scipy": []})
         dict_test(minimizer_dict, {"scipy": ['TNC', 'lm-scipy']})
 
-    @mock.patch('{}.loop_over_starting_values'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_starting_values'.format(FITTING_DIR))
     def test_run_multiple_benchmark_problems(self, loop_over_starting_values):
         """
         Checks that all benchmark problems run with no failures
@@ -126,7 +125,7 @@ class LoopOverBenchmarkProblemsTests(unittest.TestCase):
         expected_list_length = len(self.list_results) * 2
         self.shared_tests(expected_list_length, expected_problem_fails)
 
-    @mock.patch('{}.loop_over_starting_values'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_starting_values'.format(FITTING_DIR))
     def test_run_multiple_failed_problems(self, loop_over_starting_values):
         """
         Checks that multiple failed problems are reported correctly
@@ -141,7 +140,7 @@ class LoopOverBenchmarkProblemsTests(unittest.TestCase):
         expected_list_length = len(self.list_results)
         self.shared_tests(expected_list_length, expected_problem_fails)
 
-    @mock.patch('{}.loop_over_starting_values'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_starting_values'.format(FITTING_DIR))
     def test_check_no_results_produced(self, loop_over_starting_values):
         """
         Checks that multiple failed problems are reported correctly
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_minimizers.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_minimizers.py
index 8d918be3..b0972524 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_minimizers.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_minimizers.py
@@ -5,8 +5,6 @@ from __future__ import (absolute_import, division, print_function)
 import inspect
 import os
 import unittest
-import mock
-
 
 from fitbenchmarking import mock_problems
 from fitbenchmarking.utils import fitbm_result, output_grabber
@@ -129,7 +127,7 @@ class LoopOverMinimizersTests(unittest.TestCase):
         assert minimizer_failed == self.minimizers
         assert new_minimizer_list == []
 
-    @mock.patch('{}.loop_over_jacobians'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_jacobians'.format(FITTING_DIR))
     def test_run_minimzers_selected(self, loop_over_jacobians):
         """
         Tests that some minimizers are selected
@@ -148,7 +146,7 @@ class LoopOverMinimizersTests(unittest.TestCase):
         assert minimizer_failed == ["deriv_free_algorithm"]
         assert new_minimizer_list == ["general"]
 
-    @mock.patch('{}.loop_over_jacobians'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_jacobians'.format(FITTING_DIR))
     def test_run_minimzers_all(self, loop_over_jacobians):
         """
         Tests that all minimizers are selected
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py
index b5a1956c..18902f46 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py
@@ -5,7 +5,6 @@ from __future__ import (absolute_import, division, print_function)
 import inspect
 import os
 import unittest
-import mock
 import numpy as np
 
 from fitbenchmarking import mock_problems
@@ -112,7 +111,7 @@ class LoopOverSoftwareTests(unittest.TestCase):
         dict_test(unselected_minimzers, expected_minimizer_failed)
         dict_test(minimizer_dict, expected_minimizer_dict)
 
-    @mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
     def test_run_software(self, loop_over_minimizers):
         """
         Checks that results are produced for all minimizers within the
@@ -135,7 +134,7 @@ class LoopOverSoftwareTests(unittest.TestCase):
         self.shared_test(expected_list_len, expected_problem_fails,
                          expected_minimizer_failed, expected_minimizer_success)
 
-    @mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
     def test_run_software_failed_minimizers(self, loop_over_minimizers):
         """
         Checks that the failed minimizers are reported
@@ -161,7 +160,7 @@ class LoopOverSoftwareTests(unittest.TestCase):
         self.shared_test(expected_list_len, expected_problem_fails,
                          expected_minimizer_failed, expected_minimizer_success)
 
-    @mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
     def test_run_software_all_failed_minimizers(self, loop_over_minimizers):
         """
         Tests that when all minimizers raise and exception for a problem
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_starting_values.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_starting_values.py
index ff4b8d31..71945971 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_starting_values.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_starting_values.py
@@ -5,7 +5,6 @@ from __future__ import (absolute_import, division, print_function)
 import inspect
 import os
 import unittest
-import mock
 
 from fitbenchmarking import mock_problems
 from fitbenchmarking.utils import fitbm_result, output_grabber
@@ -109,7 +108,7 @@ class LoopOverStartingValuesTests(unittest.TestCase):
         dict_test(unselected_minimzers, expected_unselected_minimzers)
         dict_test(minimizer_dict, {"scipy": ['TNC', 'lm-scipy']})
 
-    @mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
     def test_run_multiple_starting_values(self, loop_over_fitting_software):
         """
         Checks that all selected minimizers run with multiple starting
@@ -127,7 +126,7 @@ class LoopOverStartingValuesTests(unittest.TestCase):
         self.shared_tests(expected_list_length, expected_problem_fails,
                           expected_unselected_minimzers)
 
-    @mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
     def test_run_one_starting_values(self, loop_over_fitting_software):
         """
         Checks that all selected minimizers run with one starting
@@ -146,7 +145,7 @@ class LoopOverStartingValuesTests(unittest.TestCase):
         self.shared_tests(expected_list_length, expected_problem_fails,
                           expected_unselected_minimzers)
 
-    @mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
     def test_run_reports_failed_problems(self, loop_over_fitting_software):
         """
         Checks that the failed problems are reported correctly
@@ -163,7 +162,7 @@ class LoopOverStartingValuesTests(unittest.TestCase):
         self.shared_tests(expected_list_length, expected_problem_fails,
                           expected_unselected_minimzers)
 
-    @mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
+    @unittest.mock.patch('{}.loop_over_fitting_software'.format(FITTING_DIR))
     def test_run_reports_unselected_minimizers(self,
                                                loop_over_fitting_software):
         """
diff --git a/fitbenchmarking/core/tests/test_results_output.py b/fitbenchmarking/core/tests/test_results_output.py
index fabe639f..303da671 100644
--- a/fitbenchmarking/core/tests/test_results_output.py
+++ b/fitbenchmarking/core/tests/test_results_output.py
@@ -7,10 +7,6 @@ import unittest
 import os
 import shutil
 import numpy as np
-# mock module are used as decorators so raise a pylint error
-# pylint: disable=import-error
-import mock
-# pylint: enable=import-error
 
 from fitbenchmarking.jacobian.scipy_jacobian import Scipy
 from fitbenchmarking.parsing.fitting_problem import FittingProblem
@@ -239,7 +235,7 @@ class CreatePlotsTests(unittest.TestCase):
             generate_mock_results()
         self.best_results = preproccess_data(self.results)
 
-    @mock.patch('fitbenchmarking.results_processing.plots.Plot')
+    @unittest.mock.patch('fitbenchmarking.results_processing.plots.Plot')
     def test_create_plots_with_params(self, plot_mock):
         """
         Tests for create_plots where the results object params are not None
@@ -247,7 +243,7 @@ class CreatePlotsTests(unittest.TestCase):
         expected_plot_initial_guess = "initial_guess"
         expected_plot_best = "plot_best"
         expected_plot_fit = "plot_fit"
-        plot_instance = mock.MagicMock()
+        plot_instance = unittest.mock.MagicMock()
         plot_instance.plot_initial_guess.return_value = \
             expected_plot_initial_guess
         plot_instance.plot_best.return_value = expected_plot_best
@@ -268,7 +264,7 @@ class CreatePlotsTests(unittest.TestCase):
             assert all(r.figure_link == expected_plot_fit
                        if not r.is_best_fit else True for r in result)
 
-    @mock.patch('fitbenchmarking.results_processing.plots.Plot')
+    @unittest.mock.patch('fitbenchmarking.results_processing.plots.Plot')
     def test_create_plots_without_params(self, plot_mock):
         """
         Tests for create_plots where the results object params are None
@@ -278,7 +274,7 @@ class CreatePlotsTests(unittest.TestCase):
             for r in result:
                 r.params = None
         expected_plot_initial_guess = "initial_guess"
-        plot_instance = mock.MagicMock()
+        plot_instance = unittest.mock.MagicMock()
         plot_instance.plot_initial_guess.return_value = \
             expected_plot_initial_guess
 
diff --git a/fitbenchmarking/systests/expected_results/all_parsers.txt b/fitbenchmarking/systests/linux_expected_results/all_parsers.txt
similarity index 100%
rename from fitbenchmarking/systests/expected_results/all_parsers.txt
rename to fitbenchmarking/systests/linux_expected_results/all_parsers.txt
diff --git a/fitbenchmarking/systests/expected_results/default_parsers.txt b/fitbenchmarking/systests/linux_expected_results/default_parsers.txt
similarity index 100%
rename from fitbenchmarking/systests/expected_results/default_parsers.txt
rename to fitbenchmarking/systests/linux_expected_results/default_parsers.txt
diff --git a/fitbenchmarking/systests/expected_results/multifit.txt b/fitbenchmarking/systests/linux_expected_results/multifit.txt
similarity index 100%
rename from fitbenchmarking/systests/expected_results/multifit.txt
rename to fitbenchmarking/systests/linux_expected_results/multifit.txt
diff --git a/fitbenchmarking/systests/test_regression.py b/fitbenchmarking/systests/test_regression.py
index 6e19a247..08d60466 100644
--- a/fitbenchmarking/systests/test_regression.py
+++ b/fitbenchmarking/systests/test_regression.py
@@ -7,6 +7,7 @@ try:
 except ImportError:
     from itertools import izip_longest as zip_longest
 import os
+from sys import platform
 import tempfile
 from unittest import TestCase
 import pytest
@@ -29,23 +30,29 @@ class TestRegressionAll(TestCase):
         Create an options file, run it, and get the results.
         """
         opts = setup_options()
-        opt_file = tempfile.NamedTemporaryFile(suffix='.ini')
-        opts.write(opt_file.name)
-
+        opt_file = tempfile.NamedTemporaryFile(suffix='.ini',
+                                               mode='w',
+                                               delete=False)
+        opts.write_to_stream(opt_file)
+        opt_file.close()
         problem = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                                os.pardir,
                                                'mock_problems',
                                                'all_parsers_set'))
         run([problem], options_file=opt_file.name, debug=True)
-
+        os.remove(opt_file.name)
         opts = setup_options(multifit=True)
-        opt_file = tempfile.NamedTemporaryFile(suffix='.ini')
-        opts.write(opt_file.name)
+        opt_file = tempfile.NamedTemporaryFile(suffix='.ini',
+                                               mode='w',
+                                               delete=False)
+        opts.write_to_stream(opt_file)
+        opt_file.close()
         problem = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                                os.pardir,
                                                'mock_problems',
                                                'multifit_set'))
         run([problem], options_file=opt_file.name, debug=True)
+        os.remove(opt_file.name)
 
     def test_results_consistent_all(self):
         """
@@ -55,7 +62,7 @@ class TestRegressionAll(TestCase):
         """
 
         expected_file = os.path.join(os.path.dirname(__file__),
-                                     'expected_results',
+                                     '{}_expected_results'.format(platform),
                                      'all_parsers.txt')
 
         actual_file = os.path.join(os.path.dirname(__file__),
@@ -79,7 +86,7 @@ class TestRegressionAll(TestCase):
         """
 
         expected_file = os.path.join(os.path.dirname(__file__),
-                                     'expected_results',
+                                     '{}_expected_results'.format(platform),
                                      'multifit.txt')
 
         actual_file = os.path.join(os.path.dirname(__file__),
@@ -113,15 +120,20 @@ class TestRegressionDefault(TestCase):
 
         # Get defaults which should have minimizers for every software
         opts = setup_options()
-        opt_file = tempfile.NamedTemporaryFile(suffix='.ini')
-        opts.write(opt_file.name)
-
+        opt_file = tempfile.NamedTemporaryFile(suffix='.ini',
+                                               mode='w',
+                                               delete=False)
+        # while opt_file is open it cannot be reoponed on Windows NT or later
+        # and hence option available is to write to the stream directly
+        opts.write_to_stream(opt_file)
+        opt_file.close()
         problem = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                                os.pardir,
                                                'mock_problems',
                                                'default_parsers'))
 
         run([problem], options_file=opt_file.name, debug=True)
+        os.remove(opt_file.name)
 
     def test_results_consistent(self):
         """
@@ -131,7 +143,7 @@ class TestRegressionDefault(TestCase):
         """
 
         expected_file = os.path.join(os.path.dirname(__file__),
-                                     'expected_results',
+                                     '{}_expected_results'.format(platform),
                                      'default_parsers.txt')
 
         actual_file = os.path.join(os.path.dirname(__file__),
diff --git a/fitbenchmarking/systests/win32_expected_results/default_parsers.txt b/fitbenchmarking/systests/win32_expected_results/default_parsers.txt
new file mode 100644
index 00000000..ef4f52a9
--- /dev/null
+++ b/fitbenchmarking/systests/win32_expected_results/default_parsers.txt
@@ -0,0 +1,5 @@
+                                bumps                    dfo                 minuit                     scipy             scipy_ls
+                               amoeba                  dfogn                 minuit               Nelder-Mead      lm-scipy-no-jac
+cubic, Start 1  1.119e-14 (2.899e+07)  7.196e-14 (1.864e+08)  2.352e-07 (6.091e+14)  6.267e-05 (1.623e+17)[1]        3.861e-22 (1)
+cubic, Start 2  1.146e-14 (2.969e+07)  1.312e-14 (3.397e+07)  7.579e-06 (1.963e+16)     7.176e-11 (1.859e+11)        3.861e-22 (1)
+Problem Def 1   6.784e+05 (5.225e+15)          1.299e-10 (1)       4.214e-07 (3245)      3.837e-09 (29.55)[1]  0.03966 (3.055e+08)
diff --git a/fitbenchmarking/utils/tests/test_options_generic.py b/fitbenchmarking/utils/tests/test_options_generic.py
index caca4bbb..d73f31b4 100644
--- a/fitbenchmarking/utils/tests/test_options_generic.py
+++ b/fitbenchmarking/utils/tests/test_options_generic.py
@@ -83,6 +83,40 @@ class OptionsWriteTests(unittest.TestCase):
 
         self.assertDictEqual(options.__dict__, new_options.__dict__)
 
+    def test_write_to_stream(self):
+        """
+        Test that the stream options writer works.
+        """
+        options = Options(file_name=self.options_file)
+        new_file_name = 'copy_of_{}'.format(self.options_file)
+
+        # open stream, write to it and close it
+        f = open(new_file_name, 'w')
+        options.write_to_stream(f)
+        f.close()
+
+        new_options = Options(new_file_name)
+
+        assert options.stored_file_name == self.options_file
+        assert new_options.stored_file_name == new_file_name
+
+        # Overwrite file names
+        options.stored_file_name = ""
+        new_options.stored_file_name = ""
+
+        os.remove(new_file_name)
+        self.assertDictEqual(options.__dict__, new_options.__dict__)
+
+    def test_create_config(self):
+        """
+        Test that created config object contains all valid sections.
+        """
+        options = Options(file_name=self.options_file)
+
+        config = options._create_config()
+        for section in options.VALID_SECTIONS:
+            self.assertTrue(config.has_section(section))
+
     def test_user_section_valid(self):
         """
         Tests that the user defined sections are correct.