Repository: fitbenchmarking/fitbenchmarking

Issue:
Results directory location setting
The location of the results directory is set by editing the options file.  Currently this is under `[PLOTTING]`, which is non-intuitive.

I think this should be moved, and potentially the best place for it would be in the command line call, as an optional flag.  At the very least, the instructions output:
![image](https://user-images.githubusercontent.com/13645545/117789141-ed5e5c00-b23f-11eb-8385-2958904f5144.png)
should be more informative on where this option can be set.

Discussion:
robertapplin: @AndrewLister-STFC maybe we want an `OUTPUT` section and we can put the `results_dir` under there?
AndrewLister-STFC: > 
> 
> @AndrewLister-STFC maybe we want an `OUTPUT` section and we can put the `results_dir` under there? (in addition to the cmd line argument of course)

Sounds like a plan :)
robertapplin: This should now be ready for more testing @AndrewLister-STFC 

Patch:
diff --git a/docs/source/users/options/index.rst b/docs/source/users/options/index.rst
index 306ef330..3609b854 100644
--- a/docs/source/users/options/index.rst
+++ b/docs/source/users/options/index.rst
@@ -17,6 +17,7 @@ and how to override them, are given in the pages below.
     minimizer_option
     jacobian_option
     plotting_option
+    output_option
     logging_option
 
 
diff --git a/docs/source/users/options/output_option.rst b/docs/source/users/options/output_option.rst
new file mode 100644
index 00000000..b7959399
--- /dev/null
+++ b/docs/source/users/options/output_option.rst
@@ -0,0 +1,23 @@
+.. _output_option:
+
+##############
+Output Options
+##############
+
+The output section contains options to control how results are outputted.
+
+.. _results_directory_option:
+
+Results directory (:code:`results_dir`)
+---------------------------------------
+
+This is used to select where the output should be saved. If the 
+:ref:`results directory command line argument <change_results_directory>` 
+is provided, this option is overridden.
+
+Default is ``fitbenchmarking_results``
+
+.. code-block:: rst
+
+    [OUTPUT]
+    results_dir: fitbenchmarking_results
\ No newline at end of file
diff --git a/docs/source/users/options/plotting_option.rst b/docs/source/users/options/plotting_option.rst
index ee63dbee..18da40ad 100644
--- a/docs/source/users/options/plotting_option.rst
+++ b/docs/source/users/options/plotting_option.rst
@@ -124,16 +124,3 @@ Default is ``acc``, ``runtime``, ``compare``, and ``local_min``.
                 runtime
                 compare
                 local_min
-
-
-Results directory (:code:`results_dir`)
----------------------------------------
-
-This is used to select where the output should be saved
-
-Default is ``fitbenchmarking_results``
-
-.. code-block:: rst
-
-    [PLOTTING]
-    results_dir: fitbenchmarking_results
diff --git a/docs/source/users/running.rst b/docs/source/users/running.rst
index 2798fb0c..3e5c7912 100644
--- a/docs/source/users/running.rst
+++ b/docs/source/users/running.rst
@@ -17,7 +17,7 @@ Running alternative problems
 
 Other problems written in a :ref:`supported file format <problem_def>`
 can be analyzed with FitBenchmarking by
-passing the path using the ``--problem-sets`` (or ``-p``) option.
+passing the path using the ``-p`` or ``--problem-sets`` argument.
 Example problems can be downloaded from
 :ref:`BenchmarkProblems`, and they can also be found in the
 ``fitbenchmarking/examples`` directory of the code.
@@ -32,8 +32,8 @@ of the source, type into the terminal:
 Changing the options
 --------------------
    
-An options file can also be passed with the ``-o`` argument.  For example,
-the template file can be run by issuing the command
+An options file can also be passed with the ``-o`` or ``--options-file`` argument. 
+For example, the template file can be run by issuing the command
 
 .. code-block:: bash
 
@@ -42,5 +42,17 @@ the template file can be run by issuing the command
 
 Details about how the options file must be formatted are given in :ref:`options`.
 
-Running ``fitbenchmarking -h`` will give more guidance about available commands,
-including examples of how to run multiple problem sets.
+.. _change_results_directory:
+
+Changing the results directory
+------------------------------
+
+The default directory where the results are saved can be changed using the ``-r``
+or ``--results-dir`` argument. The :ref:`results directory option <results_directory_option>`
+can also be changed in the options file.
+
+.. code-block:: bash
+
+   fitbenchmarking -r new_results/
+
+The default results directory is ``fitbenchmarking_results``.
diff --git a/examples/options_template.ini b/examples/options_template.ini
index 76870e8f..6e8b43e1 100644
--- a/examples/options_template.ini
+++ b/examples/options_template.ini
@@ -1,9 +1,9 @@
 [FITTING]
 
 software: bumps
-	  dfo
-	  scipy_ls
-	  minuit
+          dfo
+          scipy_ls
+          minuit
 
 num_runs: 1
 max_runtime: 600
@@ -24,6 +24,10 @@ ralfit: gn
 
 make_plots: no
 
+[OUTPUT]
+
+results_dir: new_results/
+
 [LOGGING]
 
 external_output: log_only
diff --git a/fitbenchmarking/cli/main.py b/fitbenchmarking/cli/main.py
index 034c1132..1efff23a 100755
--- a/fitbenchmarking/cli/main.py
+++ b/fitbenchmarking/cli/main.py
@@ -73,6 +73,10 @@ of the Fitbenchmarking docs. '''
                                                        'NIST',
                                                        'average_difficulty')),
                         help='Paths to directories containing problem sets.')
+    parser.add_argument('-r', '--results-dir',
+                        metavar='RESULTS_DIR',
+                        default='',
+                        help='The directory to store resulting files in.')
     parser.add_argument('-d', '--debug-mode',
                         default=False,
                         action='store_true',
@@ -82,7 +86,7 @@ of the Fitbenchmarking docs. '''
 
 
 @exception_handler
-def run(problem_sets, options_file='', debug=False):
+def run(problem_sets, results_directory, options_file='', debug=False):
     # pylint: disable=unused-argument
     """
     Run benchmarking for the problems sets and options file given.
@@ -90,6 +94,8 @@ def run(problem_sets, options_file='', debug=False):
 
     :param problem_sets: The paths to directories containing problem_sets
     :type problem_sets: list of str
+    :param results_directory: The directory to store the resulting files in
+    :type results_directory: str
     :param options_file: The path to an options file, defaults to ''
     :type options_file: str, optional
     :param debug: Enable debugging output
@@ -107,9 +113,10 @@ def run(problem_sets, options_file='', debug=False):
         if not options_file.endswith(".ini"):
             raise OptionsError('Options file must be a ".ini" file')
 
-        options = Options(glob_options_file)
+        options = Options(file_name=glob_options_file,
+                          results_directory=results_directory)
     else:
-        options = Options()
+        options = Options(results_directory=results_directory)
 
     setup_logger(log_file=options.log_file,
                  append=options.log_append,
@@ -203,12 +210,11 @@ def run(problem_sets, options_file='', debug=False):
                   "and/or problem set then re-run FitBenchmarking"
         raise NoResultsError(message)
 
-    if os.path.basename(options.results_dir) == \
-            options.DEFAULT_PLOTTING['results_dir']:
-        LOGGER.info("\nINFO: \nThe FitBenchmarking results will be "
-                    "placed into the folder: \n\n   %s\n\nTo change this "
-                    "alter the input options "
-                    "file.\n", options.results_dir)
+    LOGGER.info("\nINFO:\nThe FitBenchmarking results will be placed into "
+                "the folder:\n\n   %s\n\nTo change this use the -r or "
+                "--results-dir optional command line argument. You can\n"
+                "also set 'results_dir' in an options file.",
+                options.results_dir)
 
     root = os.path.dirname(inspect.getfile(fitbenchmarking))
     template_dir = os.path.join(root, 'templates')
@@ -249,7 +255,9 @@ def main():
     args = parser.parse_args(sys.argv[1:])
 
     run(problem_sets=args.problem_sets,
-        options_file=args.options_file, debug=args.debug_mode)
+        results_directory=args.results_dir,
+        options_file=args.options_file,
+        debug=args.debug_mode)
 
 
 if __name__ == '__main__':
diff --git a/fitbenchmarking/utils/options.py b/fitbenchmarking/utils/options.py
index 76c22653..4d61c636 100644
--- a/fitbenchmarking/utils/options.py
+++ b/fitbenchmarking/utils/options.py
@@ -17,7 +17,7 @@ class Options:
     DEFAULTS = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                             'default_options.ini'))
     VALID_SECTIONS = ['MINIMIZERS', 'FITTING', 'JACOBIAN',
-                      'PLOTTING', 'LOGGING']
+                      'PLOTTING', 'OUTPUT', 'LOGGING']
     VALID_MINIMIZERS = \
         {'bumps': ['amoeba', 'lm-bumps', 'newton', 'de', 'mp'],
          'dfo': ['dfogn', 'dfols'],
@@ -78,6 +78,7 @@ class Options:
          'comparison_mode': ['abs', 'rel', 'both'],
          'table_type': ['acc', 'runtime', 'compare', 'local_min'],
          'colour_map': plt.colormaps()}
+    VALID_OUTPUT = {}
     VALID_LOGGING = \
         {'level': ['NOTSET', 'DEBUG', 'INFO', 'WARNING', 'ERROR',
                    'CRITICAL'],
@@ -88,6 +89,7 @@ class Options:
              'FITTING': VALID_FITTING,
              'JACOBIAN': VALID_JACOBIAN,
              'PLOTTING': VALID_PLOTTING,
+             'OUTPUT': VALID_OUTPUT,
              'LOGGING': VALID_LOGGING}
 
     DEFAULT_MINIMZERS = \
@@ -141,8 +143,9 @@ class Options:
          'colour_ulim': 100,
          'cmap_range': [0.2, 0.8],
          'comparison_mode': 'both',
-         'table_type': ['acc', 'runtime', 'compare', 'local_min'],
-         'results_dir': 'fitbenchmarking_results'}
+         'table_type': ['acc', 'runtime', 'compare', 'local_min']}
+    DEFAULT_OUTPUT = \
+        {'results_dir': 'fitbenchmarking_results'}
     DEFAULT_LOGGING = \
         {'file_name': 'fitbenchmarking.log',
          'append': False,
@@ -152,14 +155,17 @@ class Options:
                 'FITTING': DEFAULT_FITTING,
                 'JACOBIAN': DEFAULT_JACOBIAN,
                 'PLOTTING': DEFAULT_PLOTTING,
+                'OUTPUT': DEFAULT_OUTPUT,
                 'LOGGING': DEFAULT_LOGGING}
 
-    def __init__(self, file_name=None):
+    def __init__(self, file_name=None, results_directory: str = ""):
         """
         Initialise the options from a file if file is given.
         Priority is values in the file, failing that, values are taken from
         DEFAULTS (stored in ./default_options.ini)
 
+        :param results_directory: The directory to store the results in.
+        :type results_directory: str
         :param file_name: The options file to load
         :type file_name: str
         """
@@ -167,7 +173,7 @@ class Options:
         # problem groups
         self.stored_file_name = file_name
         self.error_message = []
-        self._results_dir = ''
+        self._results_dir: str = ""
         config = configparser.ConfigParser(converters={'list': read_list,
                                                        'str': str,
                                                        'rng': read_range},
@@ -230,7 +236,12 @@ class Options:
         self.comparison_mode = self.read_value(plotting.getstr,
                                                'comparison_mode')
         self.table_type = self.read_value(plotting.getlist, 'table_type')
-        self.results_dir = self.read_value(plotting.getstr, 'results_dir')
+
+        output = config['OUTPUT']
+        if results_directory == "":
+            self.results_dir = self.read_value(output.getstr, 'results_dir')
+        else:
+            self.results_dir = results_directory
 
         logging = config['LOGGING']
         self.log_append = self.read_value(logging.getboolean, 'append')
@@ -247,7 +258,7 @@ class Options:
         """
         Resets options object when running multiple problem groups.
         """
-        self.__init__(self.stored_file_name)
+        self.__init__(self.stored_file_name, self.results_dir)
 
     def read_value(self, func, option):
         """
@@ -284,15 +295,18 @@ class Options:
         return value
 
     @property
-    def results_dir(self):
+    def results_dir(self) -> str:
         """
-        Returns the path to the results directory
+        Returns the directory to store the results in.
         """
         return self._results_dir
 
     @results_dir.setter
-    def results_dir(self, value):
-        self._results_dir = os.path.abspath(value)
+    def results_dir(self, directory: str) -> None:
+        """
+        Sets the directory to store the results.
+        """
+        self._results_dir = os.path.abspath(directory)
 
     @property
     def minimizers(self):
@@ -335,8 +349,8 @@ class Options:
                               'colour_ulim': self.colour_ulim,
                               'comparison_mode': self.comparison_mode,
                               'make_plots': self.make_plots,
-                              'results_dir': self.results_dir,
                               'table_type': list_to_string(self.table_type)}
+        config['OUTPUT'] = {'results_dir': self.results_dir}
         config['LOGGING'] = {'file_name': self.log_file,
                              'level': self.log_level,
                              'append': self.log_append,


Test patch:
diff --git a/fitbenchmarking/cli/tests/test_main.py b/fitbenchmarking/cli/tests/test_main.py
index d40543a2..758eb5dd 100644
--- a/fitbenchmarking/cli/tests/test_main.py
+++ b/fitbenchmarking/cli/tests/test_main.py
@@ -44,7 +44,8 @@ class TestMain(TestCase):
         benchmark.return_value = ([], [], {}, '')
 
         with self.assertRaises(exceptions.NoResultsError):
-            main.run(['examples/benchmark_problems/simple_tests'], debug=True)
+            main.run(['examples/benchmark_problems/simple_tests'],
+                     os.path.dirname(__file__), debug=True)
 
     @patch('fitbenchmarking.cli.main.benchmark')
     def test_all_dummy_results_produced(self, benchmark):
@@ -54,7 +55,8 @@ class TestMain(TestCase):
         benchmark.side_effect = self.mock_func_call
 
         with self.assertRaises(exceptions.NoResultsError):
-            main.run(['examples/benchmark_problems/simple_tests'], debug=True)
+            main.run(['examples/benchmark_problems/simple_tests'],
+                     os.path.dirname(__file__), debug=True)
 
     def mock_func_call(self, *args, **kwargs):
         """
diff --git a/fitbenchmarking/core/tests/test_results_output.py b/fitbenchmarking/core/tests/test_results_output.py
index 474d6da9..25e19caa 100644
--- a/fitbenchmarking/core/tests/test_results_output.py
+++ b/fitbenchmarking/core/tests/test_results_output.py
@@ -63,7 +63,7 @@ def fitting_function_2(data, x1, x2):
 
 
 # pylint: enable=unused-argument
-def generate_mock_results():
+def generate_mock_results(results_directory: str):
     """
     Generates results to test against
 
@@ -74,7 +74,7 @@ def generate_mock_results():
                   Options object)
     """
     software = 'scipy_ls'
-    options = Options()
+    options = Options(results_directory=results_directory)
     options.software = [software]
     num_min = len(options.minimizers[options.software[0]])
     data_x = np.array([[1, 4, 5], [2, 1, 5]])
@@ -140,18 +140,18 @@ class SaveResultsTests(unittest.TestCase):
         """
         Setting up paths and results folders
         """
+        self.results_dir = os.path.join(
+            os.path.dirname(os.path.realpath(__file__)),
+            'fitbenchmarking_results')
         self.results, self.options, self.min_chi_sq, self.min_runtime = \
-            generate_mock_results()
-        test_path = os.path.dirname(os.path.realpath(__file__))
-        self.dirname = os.path.join(test_path, 'fitbenchmarking_results')
-        self.options.results_dir = self.dirname
-        os.mkdir(self.dirname)
+            generate_mock_results(self.results_dir)
+        os.mkdir(self.results_dir)
 
     def tearDown(self):
         """
         Clean up created folders.
         """
-        shutil.rmtree(self.dirname)
+        shutil.rmtree(self.results_dir)
 
     def test_save_results_correct(self):
         """
@@ -164,7 +164,7 @@ class SaveResultsTests(unittest.TestCase):
         group_dir = save_results(self.options, self.results, group_name,
                                  failed_problems, unselected_minimzers,
                                  cost_func_description)
-        assert group_dir == os.path.join(self.dirname, group_name)
+        assert group_dir == os.path.join(self.results_dir, group_name)
 
 
 class CreateDirectoriesTests(unittest.TestCase):
@@ -176,24 +176,23 @@ class CreateDirectoriesTests(unittest.TestCase):
         """
         Setting up paths and results folders
         """
-        self.options = Options()
         test_path = os.path.dirname(os.path.realpath(__file__))
-        self.dirname = os.path.join(test_path, 'fitbenchmarking_results')
-        os.mkdir(self.dirname)
+        self.results_dir = os.path.join(test_path, 'fitbenchmarking_results')
+        self.options = Options(results_directory=self.results_dir)
+        os.mkdir(self.results_dir)
 
     def tearDown(self):
         """
         Clean up created folders.
         """
-        shutil.rmtree(self.dirname)
+        shutil.rmtree(self.results_dir)
 
     def test_create_dirs_correct(self):
         """
         Test to check that the directories are correctly made.
         """
         group_name = 'test_group'
-        self.options.results_dir = self.dirname
-        expected_results_dir = self.dirname
+        expected_results_dir = self.results_dir
         expected_group_dir = os.path.join(expected_results_dir, group_name)
         expected_support_dir = os.path.join(expected_group_dir,
                                             'support_pages')
@@ -223,8 +222,11 @@ class PreproccessDataTests(unittest.TestCase):
         """
         Setting up paths and results folders
         """
+        self.results_dir = os.path.join(
+            os.path.dirname(os.path.realpath(__file__)),
+            'fitbenchmarking_results')
         self.results, self.options, self.min_chi_sq, self.min_runtime = \
-            generate_mock_results()
+            generate_mock_results(self.results_dir)
 
     def test_preproccess_data(self):
         """
@@ -254,9 +256,9 @@ class CreatePlotsTests(unittest.TestCase):
         Setting up paths and results folders
         """
         self.tempdir = TemporaryDirectory()
-        self.dirname = os.path.join(self.tempdir.name, 'figures_dir')
+        self.results_dir = os.path.join(self.tempdir.name, 'figures_dir')
         self.results, self.options, self.min_chi_sq, self.min_runtime = \
-            generate_mock_results()
+            generate_mock_results(self.results_dir)
         self.best_results = preproccess_data(self.results)
 
     @mock.patch('fitbenchmarking.results_processing.plots.Plot')
@@ -276,7 +278,7 @@ class CreatePlotsTests(unittest.TestCase):
         # Return the above created `plot_instance`
         plot_mock.return_value = plot_instance
         create_plots(self.options, self.results,
-                     self.best_results, self.dirname)
+                     self.best_results, self.results_dir)
         for result, best_result in zip(self.results, self.best_results):
             # Check initial guess is correctly set in results
             assert best_result.start_figure_link == expected_plot_initial_guess
@@ -305,7 +307,7 @@ class CreatePlotsTests(unittest.TestCase):
         # Return the above created `plot_instance`
         plot_mock.return_value = plot_instance
         create_plots(self.options, self.results,
-                     self.best_results, self.dirname)
+                     self.best_results, self.results_dir)
         for result, best_result in zip(self.results, self.best_results):
             # Check initial guess is correctly set in results
             assert best_result.start_figure_link == expected_plot_initial_guess
@@ -333,7 +335,7 @@ class CreatePlotsTests(unittest.TestCase):
                 side_effect=PlottingError('Faked plot')):
 
             create_plots(self.options, self.results,
-                         self.best_results, self.dirname)
+                         self.best_results, self.results_dir)
 
         expected = 'An error occurred during plotting.\nDetails: Faked plot'
         for result, best_result in zip(self.results, self.best_results):
diff --git a/fitbenchmarking/systests/test_regression.py b/fitbenchmarking/systests/test_regression.py
index 58da53b6..be52d920 100644
--- a/fitbenchmarking/systests/test_regression.py
+++ b/fitbenchmarking/systests/test_regression.py
@@ -29,6 +29,9 @@ class TestRegressionAll(TestCase):
         """
         Create an options file, run it, and get the results.
         """
+        results_dir = os.path.join(os.path.dirname(__file__),
+                                   'fitbenchmarking_results')
+
         opts = setup_options()
         opt_file = tempfile.NamedTemporaryFile(suffix='.ini',
                                                mode='w',
@@ -39,7 +42,8 @@ class TestRegressionAll(TestCase):
                                                os.pardir,
                                                'mock_problems',
                                                'all_parsers_set'))
-        run([problem], options_file=opt_file.name, debug=True)
+        run([problem], results_dir, options_file=opt_file.name,
+            debug=True)
         os.remove(opt_file.name)
         opts = setup_options(multifit=True)
         opt_file = tempfile.NamedTemporaryFile(suffix='.ini',
@@ -51,7 +55,8 @@ class TestRegressionAll(TestCase):
                                                os.pardir,
                                                'mock_problems',
                                                'multifit_set'))
-        run([problem], options_file=opt_file.name, debug=True)
+        run([problem], results_dir, options_file=opt_file.name,
+            debug=True)
         os.remove(opt_file.name)
 
     def test_results_consistent_all(self):
@@ -117,6 +122,9 @@ class TestRegressionMatlab(TestCase):
         """
         Create an options file, run it, and get the results.
         """
+        results_dir = os.path.join(os.path.dirname(__file__),
+                                   'fitbenchmarking_results')
+
         opts = setup_options()
         opt_file = tempfile.NamedTemporaryFile(suffix='.ini',
                                                mode='w',
@@ -127,7 +135,8 @@ class TestRegressionMatlab(TestCase):
                                                os.pardir,
                                                'mock_problems',
                                                'all_parsers_set'))
-        run([problem], options_file=opt_file.name, debug=True)
+        run([problem], results_dir, options_file=opt_file.name,
+            debug=True)
         os.remove(opt_file.name)
 
     def test_results_consistent_all(self):
@@ -169,6 +178,8 @@ class TestRegressionDefault(TestCase):
         """
         Create an options file, run it, and get the results.
         """
+        results_dir = os.path.join(os.path.dirname(__file__),
+                                   'fitbenchmarking_results')
 
         # Get defaults which should have minimizers for every software
         opts = setup_options()
@@ -184,7 +195,8 @@ class TestRegressionDefault(TestCase):
                                                'mock_problems',
                                                'default_parsers'))
 
-        run([problem], options_file=opt_file.name, debug=True)
+        run([problem], results_dir, options_file=opt_file.name,
+            debug=True)
         os.remove(opt_file.name)
 
     def test_results_consistent(self):
@@ -277,7 +289,4 @@ def setup_options(multifit=False):
         opts.software = ['bumps', 'scipy', 'scipy_ls']
         opts.minimizers = {s: [opts.minimizers[s][0]] for s in opts.software}
 
-    opts.results_dir = os.path.join(os.path.dirname(__file__),
-                                    'fitbenchmarking_results')
-
     return opts
diff --git a/fitbenchmarking/utils/tests/test_options_generic.py b/fitbenchmarking/utils/tests/test_options_generic.py
index b3436557..e8303156 100644
--- a/fitbenchmarking/utils/tests/test_options_generic.py
+++ b/fitbenchmarking/utils/tests/test_options_generic.py
@@ -42,7 +42,9 @@ class OptionsWriteTests(unittest.TestCase):
             comparison_mode: abs
             table_type: acc
                         runtime
-            results_dir: new_results
+
+            [OUTPUT]
+            results_dir: new_results/
 
             [LOGGING]
             file_name: THE_LOG.log
diff --git a/fitbenchmarking/utils/tests/test_options_output.py b/fitbenchmarking/utils/tests/test_options_output.py
new file mode 100644
index 00000000..025e1f1e
--- /dev/null
+++ b/fitbenchmarking/utils/tests/test_options_output.py
@@ -0,0 +1,95 @@
+'''
+Test the OUTPUT section for the options file
+'''
+import inspect
+import shutil
+import os
+import unittest
+
+from fitbenchmarking.utils.options import Options
+
+
+class OutputOptionTests(unittest.TestCase):
+    """
+    Checks that the defaults in the PLOTTING section are set correctly
+    """
+
+    def test_results_dir_default(self):
+        """
+        Checks results_dir default
+        """
+        options = Options(results_directory="")
+        expected = os.path.abspath('fitbenchmarking_results')
+        self.assertEqual(expected, options.results_dir)
+
+    def test_results_dir_not_default(self):
+        """
+        Checks results_dir default
+        """
+        options = Options(results_directory=os.path.dirname(__file__))
+        default = os.path.abspath('fitbenchmarking_results')
+        self.assertNotEqual(default, options.results_dir)
+
+
+class UserOutputOptionTests(unittest.TestCase):
+    """
+    Checks the output options in the file are set correctly or raise errors
+    """
+
+    def setUp(self):
+        """
+        Sets the directory to save the temporary ini files in
+        """
+        options_dir = os.path.dirname(inspect.getfile(Options))
+        self.test_files_dir = os.path.join(options_dir, 'tests', 'files')
+        os.mkdir(self.test_files_dir)
+
+    def tearDown(self):
+        """
+        Deletes temporary folder and results produced
+        """
+        if os.path.exists(self.test_files_dir):
+            shutil.rmtree(self.test_files_dir)
+
+    def generate_user_ini_file(self, opt_name, config_str):
+        """
+        Generates user defined ini file
+
+        :param opt_name: name of option to be set
+        :type opt_name: str
+        :param config_str: section of an ini file which sets the option
+        :type config_str: str
+
+        :return: location of temporary ini file
+        :rtype: str
+        """
+        opts_file = os.path.join(self.test_files_dir,
+                                 'test_{}_valid.ini'.format(opt_name))
+        with open(opts_file, 'w') as f:
+            f.write(config_str)
+        return opts_file
+
+    def shared_valid(self, opt_name, options_set, config_str):
+        """
+        Shared test to check that the plotting option set is valid
+
+        :param opt_name: name of option to be set
+        :type opt_name: str
+        :param options_set: option set to be tested
+        :type options_set: list
+        :param config_str: section of an ini file which sets the option
+        :type config_str: str
+        """
+        opts_file = self.generate_user_ini_file(opt_name, config_str)
+        options = Options(opts_file)
+        actual = getattr(options, opt_name)
+        self.assertEqual(options_set, actual)
+
+    def test_output_results_directory_valid(self):
+        """
+        Checks user set make_plots is valid
+        """
+        set_option = os.path.abspath("new_results")
+        config_str = \
+            "[OUTPUT]\nresults_dir: new_results/"
+        self.shared_valid('results_dir', set_option, config_str)
diff --git a/fitbenchmarking/utils/tests/test_options_plotting.py b/fitbenchmarking/utils/tests/test_options_plotting.py
index a134857a..5ca7e0d6 100644
--- a/fitbenchmarking/utils/tests/test_options_plotting.py
+++ b/fitbenchmarking/utils/tests/test_options_plotting.py
@@ -69,14 +69,6 @@ class PlottingOptionTests(unittest.TestCase):
         actual = self.options.table_type
         self.assertEqual(expected, actual)
 
-    def test_results_dir_default(self):
-        """
-        Checks results_dir default
-        """
-        expected = os.path.abspath('fitbenchmarking_results')
-        actual = self.options.results_dir
-        self.assertEqual(expected, actual)
-
 
 class UserPlottingOptionTests(unittest.TestCase):
     """
@@ -246,12 +238,3 @@ class UserPlottingOptionTests(unittest.TestCase):
         config_str = \
             "[PLOTTING]\ntable_type: chi_sq\n "
         self.shared_invalid('table_type', config_str)
-
-    def test_minimizer_results_dir_valid(self):
-        """
-        Checks user set results_dir is valid
-        """
-        set_option = os.path.abspath("results_dir")
-        config_str = \
-            "[PLOTTING]\nresults_dir: results_dir"
-        self.shared_valid('results_dir', set_option, config_str)