Repository: fitbenchmarking/fitbenchmarking

Issue:
Acc and runtime performance profiles show different n failures for same solver
When running the NIST/average_difficulty problems, one can see that in the performance profile plots, certain solvers are marked as having a certain number of failures based on the accuracy plot, but no failure based on the runtime plot. See 'dream [bumps]' and 'newton [bumps]' in the picture below, they are marked as having failed in the legend of the accuracy plot but not in the runtime plot.

The number of failures (to display in the plots' legends) is determined in function create_plot in performance_profiler. The upper bound to determine failures is set to 1.0e20, which might be causing the problem in these examples.

![pp](https://github.com/fitbenchmarking/fitbenchmarking/assets/28632706/dc01367f-f3ec-4789-a6fb-8f1f07517851)



Discussion:
github-actions[bot]: ## Unit & system test results
    4 files  ±0    4 suites  ±0   11m 40s :stopwatch: -24s
  569 tests ±0  559 :white_check_mark: ±0  10 :zzz: ±0  0 :x: ±0 
1 066 runs  ±0  990 :white_check_mark: ±0  76 :zzz: ±0  0 :x: ±0 

Results for commit af7e95dd. ± Comparison against base commit 05bd1e70.

[test-results]:data:application/gzip;base64,H4sIAChu/WYC/02MyQ6DIBRFf8Ww7gJk7s8YZEhIVRqGlem/F61Slue8+84OnF9sAs+BPAaQis8NTIkq+7BV5BBWUU/5OFImb5pS0fpQtFMv/64KtZfJKb9U8xc2xhAvE8uWzjljF91RKWEzvyZvky55cl/UYV19rgCU41ZSYxQcnSMIzWhk2GACMXZYzVQIpUerBfh8AcIXVtEIAQAA

coveralls: 
[![Coverage Status](https://coveralls.io/builds/70128482/badge)](https://coveralls.io/builds/70128482)

coverage: 91.237% (+0.02%) from 91.222%
when pulling **af7e95dda02ff411b1263d34033f3ab588ac2ec8 on 1240-profile_failures**
into **05bd1e7023915896fde02c3cb1754971471ea118 on master**.


Patch:
diff --git a/fitbenchmarking/results_processing/performance_profiler.py b/fitbenchmarking/results_processing/performance_profiler.py
index ffa60eb0..f50ee511 100644
--- a/fitbenchmarking/results_processing/performance_profiler.py
+++ b/fitbenchmarking/results_processing/performance_profiler.py
@@ -277,13 +277,11 @@ def adjust_values_to_plot(step_values: 'list[np.ndarray]',
         label = f"{solver}"
         plot_points = np.linspace(0.0, 1.0, solver_values.size)
         plot_points = np.append(plot_points, 1.0)
-        inf_indices = np.where(solver_values > huge)
-        solver_values[inf_indices] = huge
-        if inf_indices[0].size > 0:
-            plural_ending = "s"
-            if inf_indices[0].size == 1:
-                plural_ending = ""
-            label = f"{solver} ({len(inf_indices[0])} failure{plural_ending})"
+        failures = np.isinf(solver_values).sum()
+        huge_indices = np.where(solver_values > huge)
+        solver_values[huge_indices] = huge
+        if failures > 0:
+            label += f" ({failures} failure{'s' if failures > 1 else ''})"
         solver_values = np.append(solver_values, huge)
 
         all_labels.append(label)
@@ -342,10 +340,10 @@ def create_plot(step_values: 'list[np.ndarray]',
     avail_styles = compute_linestyle_combinations()
 
     for label, solver_values, plot_points in zip(
-                                                data_to_plot['labels'],
-                                                data_to_plot['solver_values'],
-                                                data_to_plot['plot_points']
-                                                ):
+        data_to_plot['labels'],
+        data_to_plot['solver_values'],
+        data_to_plot['plot_points']
+    ):
 
         linestyle, colour = avail_styles.pop()
 
@@ -359,7 +357,7 @@ def create_plot(step_values: 'list[np.ndarray]',
                        name=label,
                        type='scatter'
                        )
-            )
+        )
 
     return fig
 


Test patch:
diff --git a/fitbenchmarking/results_processing/tests/test_performance_profiler.py b/fitbenchmarking/results_processing/tests/test_performance_profiler.py
index 602a0a59..56c60c06 100644
--- a/fitbenchmarking/results_processing/tests/test_performance_profiler.py
+++ b/fitbenchmarking/results_processing/tests/test_performance_profiler.py
@@ -348,6 +348,38 @@ class PerformanceProfilerTests(unittest.TestCase):
         assert np.array_equal(expected_dict['plot_points'],
                               output_dict['plot_points'])
 
+    def test_adjust_values_to_plot_failures(self):
+        """
+        Test adjust_values_to_plot when failures are present.
+        """
+        solvers = ["Solver1", "Solver2", "Solver3"]
+        step_vals = [
+            np.array([0.0, 0.0123, 0.123, 1.23, 8.2]),
+            np.array([0.0, 0.12, 1e23, np.inf, np.inf]),
+            np.array([0.0, 3.8, 5.6, 1e29, np.inf]),
+        ]
+
+        labels = ["Solver1", "Solver2 (2 failures)", "Solver3 (1 failure)"]
+        solver_vals = [
+            np.array([0.0, 0.0123, 0.123, 1.23, 8.2, 1e20]),
+            np.array([0.0, 0.12, 1e20, 1e20, 1e20, 1e20]),
+            np.array([0.0, 3.8, 5.6, 1e20, 1e20, 1e20]),
+        ]
+        plot_points = [
+            np.array([0.0, 0.25, 0.5, 0.75, 1.0, 1.0]),
+            np.array([0.0, 0.25, 0.5, 0.75, 1.0, 1.0]),
+            np.array([0.0, 0.25, 0.5, 0.75, 1.0, 1.0]),
+        ]
+
+        res = performance_profiler.adjust_values_to_plot(
+            step_values=step_vals, solvers=solvers
+        )
+
+        assert res["solvers"] == solvers
+        assert res["labels"] == labels
+        assert np.array_equal(res["solver_values"], solver_vals)
+        assert np.array_equal(res["plot_points"], plot_points)
+
     def test_compute_linestyle_combinations(self):
         """
         Test compute_linestyle_combinations returns expected styles.