Repository: fitbenchmarking/fitbenchmarking

Issue:
Correct flag for MaxRuntimeError is not set when using Mantid
**Description of the error**
When the max runtime is reached using a mantid minimizer, the wrong flag is returned (3 is returned instead of 6).

**Suggested fix**
The MaxRuntimeError can sometimes be caught by the fitting software and re-raised as an ordinary Exception. So this error is caught by the 'Exception' clause and flag 3 is set.

To fix, we can check the exception message and if it contains the MaxRuntime error message then we can set the flag as 6.


Discussion:
robertapplin: I think the `nan` makes sense for the second image in the PR description because the relative performance cannot be calculated when there's a single minimizer
AndrewLister-STFC: > 
> 
> I think the `nan` makes sense for the second image in the PR description because the relative performance cannot be calculated when there's a single minimizer

Using `nan` could be an issue if the black colour clashes with a mapping someone is using, since it will always be black. I'm not sure if it's best to edit the colour mapping or use a safer divide in the result class.
robertapplin: > > I think the `nan` makes sense for the second image in the PR description because the relative performance cannot be calculated when there's a single minimizer
> 
> Using `nan` could be an issue if the black colour clashes with a mapping someone is using, since it will always be black. I'm not sure if it's best to edit the colour mapping or use a safer divide in the result class.

I see, that could be a problem. It would be good to discuss this with you tomorrow
robertapplin: After talking with @AndrewLister-STFC , we decided using inf instead of nan would also be ok (this can be changed in `fitbm_result`)
robertapplin: I ended up changing the colour of `nans` so that they use the same colour as `infs` because it was simpler and is still a good fix. 
![image](https://user-images.githubusercontent.com/40830825/135277071-797e032e-9f04-4d0e-899f-aa33adb33b8b.png)

Its ready for another look @AndrewLister-STFC 
robertapplin: Thanks @AndrewLister-STFC , its ready again

Patch:
diff --git a/docs/source/users/options/minimizer_option.rst b/docs/source/users/options/minimizer_option.rst
index ceed169d..73b31650 100644
--- a/docs/source/users/options/minimizer_option.rst
+++ b/docs/source/users/options/minimizer_option.rst
@@ -235,7 +235,7 @@ It has support for a number of minimizers, most of which are from GSL.
 
 * `Levenberg-Marquardt MD <https://docs.mantidproject.org/nightly/fitting/fitminimizers/LevenbergMarquardtMD.html>`__ (:code:`Levenberg-MarquardtMD`) - An implementation of Levenberg-Marquardt intended for MD workspaces, where work is divided into chunks to achieve a greater efficiency for a large number of data points.
 
-* `Simplex <https://docs.mantidproject.org/nightly/fitting/fitminimizers/Simplex.html>`__ (:code:`simplex`)
+* `Simplex <https://docs.mantidproject.org/nightly/fitting/fitminimizers/Simplex.html>`__ (:code:`Simplex`)
 
 * `SteepestDescent <https://docs.mantidproject.org/nightly/fitting/fitminimizers/GradientDescent.html>`__ (:code:`SteepestDescent`)
 
diff --git a/fitbenchmarking/core/fitting_benchmarking.py b/fitbenchmarking/core/fitting_benchmarking.py
index 14727a84..1605521e 100644
--- a/fitbenchmarking/core/fitting_benchmarking.py
+++ b/fitbenchmarking/core/fitting_benchmarking.py
@@ -249,7 +249,6 @@ def loop_over_fitting_software(cost_func, options, start_values_index,
     # exception and record the problem name if that is the case
     software_check = [np.isinf(v.chi_sq) for v in software_results]
     if all(software_check):
-        software_results = []
         problem_fails.append(cost_func.problem.name)
     results.extend(software_results)
 
@@ -533,12 +532,13 @@ def loop_over_hessians(controller, options, minimizer_name,
                 raise ControllerAttributeError(
                     "Either the computed runtime or chi_sq values "
                     "was a NaN.")
-        except MaxRuntimeError as ex:
-            LOGGER.warning(str(ex))
-            controller.flag = 6
         except Exception as ex:  # pylint: disable=broad-except
             LOGGER.warning(str(ex))
-            controller.flag = 3
+            # The MaxRuntimeError can sometimes be caught by the fitting
+            # software and re-raised as an ordinary Exception. So a separate
+            # except clause to set the flag will not work.
+            controller.flag = 6 if MaxRuntimeError.class_message in str(ex) \
+                else 3
 
         if controller.flag in [3, 6]:
             # If there was an exception, set the runtime and
diff --git a/fitbenchmarking/results_processing/base_table.py b/fitbenchmarking/results_processing/base_table.py
index f4e132a5..f3d67f19 100644
--- a/fitbenchmarking/results_processing/base_table.py
+++ b/fitbenchmarking/results_processing/base_table.py
@@ -510,6 +510,7 @@ class Table:
         norm_vals = (log_vals - log_llim) /\
             (log_ulim - log_llim)
         norm_vals[norm_vals > 1] = 1  # applying upper cutoff
+        norm_vals[np.isnan(norm_vals)] = 1  # deal with nans
         # trimming colour map according to default/user input
         norm_vals = cmap_range[0] + \
             norm_vals*(cmap_range[1] - cmap_range[0])
diff --git a/fitbenchmarking/results_processing/performance_profiler.py b/fitbenchmarking/results_processing/performance_profiler.py
index 4769d0a1..98e67d26 100644
--- a/fitbenchmarking/results_processing/performance_profiler.py
+++ b/fitbenchmarking/results_processing/performance_profiler.py
@@ -81,12 +81,14 @@ def plot(acc, runtime, fig_dir):
         figure_path.append(this_filename)
 
         step_values = []
+        max_value = 0.0
         for value in profile_plot.values():
-            sorted_list = np.sort(value)
+            sorted_list = np.sort(_remove_nans(value))
+            max_in_list = np.max(sorted_list) if len(sorted_list) > 0 else 0.0
+            if max_in_list > max_value:
+                max_value = max_in_list
             step_values.append(np.insert(sorted_list, 0, 0.0))
 
-        max_value = np.max([np.max(v)
-                            for v in profile_plot.values()])
         linear_upper_limit = 10
 
         use_log_plot = True
@@ -154,6 +156,13 @@ def plot(acc, runtime, fig_dir):
     return figure_path
 
 
+def _remove_nans(values: np.ndarray) -> np.ndarray:
+    """
+    Removes all the nan values from the provided numpy array.
+    """
+    return values[~np.isnan(values)]
+
+
 def create_plot(ax, step_values, solvers):
     """
     Function to draw the profile on a matplotlib axis


Test patch:
diff --git a/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py b/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py
index 96a88b38..25e1258a 100644
--- a/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py
+++ b/fitbenchmarking/core/tests/test_fitting_benchmarking_software.py
@@ -167,8 +167,8 @@ class LoopOverSoftwareTests(unittest.TestCase):
     @unittest.mock.patch('{}.loop_over_minimizers'.format(FITTING_DIR))
     def test_run_software_all_failed_minimizers(self, loop_over_minimizers):
         """
-        Tests that when all minimizers raise and exception for a problem
-        that this is reported correctly
+        Tests that when all minimizers raise an exception for a problem, it
+        is reported correctly and no results are excluded.
         """
         self.count = 0
         self.minimizer_failed = {s: self.options.minimizers[s]
@@ -182,7 +182,7 @@ class LoopOverSoftwareTests(unittest.TestCase):
                                 [fitbm_result.FittingResult(**self.result_args)
                                  for i in range(self.dfo_len)]]
         loop_over_minimizers.side_effect = self.mock_func_call
-        expected_list_len = 0
+        expected_list_len = 10
         expected_problem_fails = self.problem_fails
         expected_minimizer_failed = self.minimizer_failed
         expected_minimizer_success = self.minimizer_success