Repository: fitbenchmarking/fitbenchmarking

Issue:
Remove/reduce 3rd party warning messages to screen 
**To reproduce**
On Windows (and if I heard correctly @wathen also observed on other environment you tried).

Install FitBenchmarking and with default settings simply from command line type and run `fitbenchmarking`. Output observed include:

```
##################################
 Running data from: MGH17.dat 7/9
##################################
    Starting value: 1/2
        Software: SCIPY
            Minimizer: Nelder-Mead
            Minimizer: Powell
            Minimizer: CG
                Jacobian: scipy 2-point
<string>:1: RuntimeWarning: overflow encountered in exp
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
<string>:1: RuntimeWarning: overflow encountered in exp
<string>:1: RuntimeWarning: overflow encountered in exp
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
<string>:1: RuntimeWarning: overflow encountered in exp
<string>:1: RuntimeWarning: overflow encountered in exp
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
<string>:1: RuntimeWarning: overflow encountered in exp
<string>:1: RuntimeWarning: overflow encountered in exp
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
<string>:1: RuntimeWarning: overflow encountered in exp
<string>:1: RuntimeWarning: overflow encountered in exp
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
<string>:1: RuntimeWarning: overflow encountered in exp
            Minimizer: BFGS
                Jacobian: scipy 2-point
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
            Minimizer: Newton-CG
                Jacobian: scipy 2-point
            Minimizer: L-BFGS-B
                Jacobian: scipy 2-point
            Minimizer: TNC
                Jacobian: scipy 2-point
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:454: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:361: RuntimeWarning: invalid value encountered in less
  if np.any((x0 < lb) | (x0 > ub)):
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:361: RuntimeWarning: invalid value encountered in greater
  if np.any((x0 < lb) | (x0 > ub)):
c:\python37\lib\site-packages\scipy\optimize\_numdiff.py:102: RuntimeWarning: invalid value encountered in greater_equal
  sign_x0 = (x0 >= 0).astype(float) * 2 - 1
        Software: SCIPY_LS
            Minimizer: lm-scipy-no-jac
<string>:1: RuntimeWarning: overflow encountered in exp
<string>:1: RuntimeWarning: invalid value encountered in add
<string>:1: RuntimeWarning: overflow encountered in multiply
            Minimizer: lm-scipy
                Jacobian: scipy 2-point
            Minimizer: trf
                Jacobian: scipy 2-point
            Minimizer: dogbox
                Jacobian: scipy 2-point
    Starting value: 2/2
        Software: SCIPY
```  

In this issue remove these scipy\optimize from the standard console, but please ensure there are capture elsewhere in some logs.


Discussion:
jfowkes: Here's what I get when testing:

Setting `external_output` to `log_only` silences the various external error messages.

Setting `external_output` to `display` gives e.g.
```
Captured error: 
.../scipy/optimize/_numdiff.py:497: RuntimeWarning: invalid value encountered in subtract
  df = fun(x) - f0
```
I presume this is the expected behaviour?

Patch:
diff --git a/docs/source/users/options/logging_option.rst b/docs/source/users/options/logging_option.rst
index e0513faa..051760b6 100644
--- a/docs/source/users/options/logging_option.rst
+++ b/docs/source/users/options/logging_option.rst
@@ -58,13 +58,17 @@ Default is ``INFO``
 Logging external output (:code:`external_output`)
 -------------------------------------------------
 
-This toggles whether the output grabber collects the third party
-output. We note that for the Windows operating system this
-option is not available.
+This selects the amount of information displayed from third-parties.
+There are 3 options:
 
-Default is ``True`` (``yes``/``no`` can also be used)
+* ``display``: Print information from third-parties to the stdout stream
+               during a run.
+* ``log_only``: Print information to the log file but not the stdout stream.
+* ``debug``: Do not intercept third-party use of output streams.
+
+Default is ``log_only``
 
 .. code-block:: rst
 
     [LOGGING]
-    append: yes
+    append: log_only
diff --git a/fitbenchmarking/utils/options.py b/fitbenchmarking/utils/options.py
index 344a53fb..0386c643 100644
--- a/fitbenchmarking/utils/options.py
+++ b/fitbenchmarking/utils/options.py
@@ -56,7 +56,7 @@ class Options(object):
         {'level': ['NOTSET', 'DEBUG', 'INFO', 'WARNING', 'ERROR',
                    'CRITICAL'],
          'append': [True, False],
-         'external_output': [True, False]}
+         'external_output': ['debug','display','log_only']}
 
     VALID = {'MINIMIZERS': VALID_MINIMIZERS,
              'FITTING': VALID_FITTING,
@@ -106,7 +106,7 @@ class Options(object):
         {'file_name': 'fitbenchmarking.log',
          'append': False,
          'level': 'INFO',
-         'external_output': True}
+         'external_output': 'log_only'}
     DEFAULTS = {'MINIMIZERS': DEFAULT_MINIMZERS,
                 'FITTING': DEFAULT_FITTING,
                 'JACOBIAN': DEFAULT_JACOBIAN,
@@ -192,7 +192,7 @@ class Options(object):
         self.log_file = self.read_value(logging.getstr, 'file_name')
         self.log_level = self.read_value(logging.getstr, 'level')
 
-        self.external_output = self.read_value(logging.getboolean,
+        self.external_output = self.read_value(logging.getstr,
                                                'external_output')
 
         if self.error_message != []:
diff --git a/fitbenchmarking/utils/output_grabber.py b/fitbenchmarking/utils/output_grabber.py
index e1e444b7..eb840a0e 100644
--- a/fitbenchmarking/utils/output_grabber.py
+++ b/fitbenchmarking/utils/output_grabber.py
@@ -7,31 +7,59 @@ from fitbenchmarking.utils.log import get_logger
 LOGGER = get_logger()
 
 
-class OutputGrabber(object):
+class OutputGrabber:
     """
-    Class used to grab standard output or another stream.
+    Class used to grab outputs for stdout and stderr
     """
-    escape_char = "\b"
-
     def __init__(self, options):
+        self.stdout_grabber = StreamGrabber(sys.stdout)
+        self.stderr_grabber = StreamGrabber(sys.stderr)
 
-        self.origstream = sys.stdout
-        self.origstreamfd = self.origstream.fileno()
-        self.capturedtext = ""
         # From issue 500 the output grabber does not currently on windows, thus
         # we set the __enter__ and __exit__ functions to pass through for this
         # case
         self.system = platform.system() != "Windows"
+
         self.external_output = options.external_output
 
     def __enter__(self):
-        if self.system and self.external_output:
-            self.start()
+        if self.system and self.external_output != 'debug':
+            self.stdout_grabber.start()
+            self.stderr_grabber.start()
         return self
 
     def __exit__(self, type, value, traceback):
         if self.system and self.external_output:
-            self.stop()
+            self.stdout_grabber.stop()
+            self.stderr_grabber.stop()
+            if self.stdout_grabber.capturedtext:
+                self._log('Captured output: \n%s',
+                          self.stdout_grabber.capturedtext)
+
+            if self.stderr_grabber.capturedtext:
+                self._log('Captured error: \n%s',
+                          self.stderr_grabber.capturedtext)
+
+    def _log(self, str_log, *args):
+        if self.external_output == 'log_only':
+            LOGGER.debug(str_log, *args)
+        else:
+            LOGGER.info(str_log, *args)
+
+
+class StreamGrabber:
+    """
+    Class used to grab an output stream.
+    """
+    escape_char = "\b"
+
+    def __init__(self, stream):
+
+        self.origstream = stream
+        self.origstreamfd = stream.fileno()
+        self.encoding = stream.encoding
+
+        self.capturedtext = ""
 
     def start(self):
         """
@@ -40,8 +68,10 @@ class OutputGrabber(object):
         # Create a pipe so the stream can be captured:
         self.pipe_out, self.pipe_in = os.pipe()
         self.capturedtext = ""
+
         # Save a copy of the stream:
         self.streamfd = os.dup(self.origstreamfd)
+
         # Replace the original stream with our write pipe:
         os.dup2(self.pipe_in, self.origstreamfd)
 
@@ -58,15 +88,14 @@ class OutputGrabber(object):
         # Reads the output and stores it in capturedtext
         self.readOutput()
 
-        if self.capturedtext:
-            LOGGER.info('Captured output: \n%s', self.capturedtext)
-
-        # Close the pipe:
+        # Close the pipes:
         os.close(self.pipe_in)
         os.close(self.pipe_out)
-        # Restore the original stream:
+
+        # Restore the original streams:
         os.dup2(self.streamfd, self.origstreamfd)
-        # Close the duplicate stream:
+
+        # Close the duplicate streams:
         os.close(self.streamfd)
 
     def readOutput(self):
@@ -75,7 +104,7 @@ class OutputGrabber(object):
         and save the text in `capturedtext`.
         """
         while True:
-            char = os.read(self.pipe_out, 1).decode(sys.stdout.encoding)
+            char = os.read(self.pipe_out, 1).decode(self.encoding)
             if not char or self.escape_char in char:
                 break
             self.capturedtext += char


Test patch:
diff --git a/fitbenchmarking/utils/tests/test_options_generic.py b/fitbenchmarking/utils/tests/test_options_generic.py
index 22b30c26..f4cef47d 100644
--- a/fitbenchmarking/utils/tests/test_options_generic.py
+++ b/fitbenchmarking/utils/tests/test_options_generic.py
@@ -47,7 +47,7 @@ class OptionsWriteTests(unittest.TestCase):
             file_name: THE_LOG.log
             append: yes
             level: DEBUG
-            external_output: no
+            external_output: display
             """
 
         opts_file = 'test_options_tests.ini'
diff --git a/fitbenchmarking/utils/tests/test_options_logging.py b/fitbenchmarking/utils/tests/test_options_logging.py
index ff72e80b..60ab7a81 100644
--- a/fitbenchmarking/utils/tests/test_options_logging.py
+++ b/fitbenchmarking/utils/tests/test_options_logging.py
@@ -49,7 +49,7 @@ class LoggingOptionTests(unittest.TestCase):
         """
         Checks external_output default
         """
-        expected = True
+        expected = 'log_only'
         actual = self.options.external_output
         self.assertEqual(expected, actual)
 
@@ -176,9 +176,9 @@ class UserLoggingOptionTests(unittest.TestCase):
         """
         Checks user set external_output is valid
         """
-        set_option = False
+        set_option = 'debug'
         config_str = \
-            "[LOGGING]\nexternal_output: no"
+            "[LOGGING]\nexternal_output: debug"
         self.shared_valid('external_output', set_option, config_str)
 
     def test_minimizer_external_output_invalid(self):
diff --git a/fitbenchmarking/utils/tests/test_output_grabber.py b/fitbenchmarking/utils/tests/test_output_grabber.py
index 8380e47c..97c20f21 100644
--- a/fitbenchmarking/utils/tests/test_output_grabber.py
+++ b/fitbenchmarking/utils/tests/test_output_grabber.py
@@ -1,9 +1,12 @@
-from __future__ import (absolute_import, division, print_function)
-import unittest
+from __future__ import absolute_import, division, print_function
+
 import platform
+import sys
+import unittest
 
-from fitbenchmarking.utils.output_grabber import OutputGrabber
 from fitbenchmarking.utils.options import Options
+from fitbenchmarking.utils.output_grabber import OutputGrabber
+
 
 class OutputGrabberTests(unittest.TestCase):
 
@@ -11,25 +14,37 @@ class OutputGrabberTests(unittest.TestCase):
         self.options = Options()
         self.plt = platform.system()
 
-    def test_correct_output(self):
+    def test_correct_stdout(self):
+        """
+        Test that the OutputGrabber grabs the stdout stream.
+        """
         output_string = 'This is the correct output string\nSecond line'
+        error_string = 'An error has occurred:\nSome details'
+
         output = OutputGrabber(self.options)
         with output:
-            print(output_string)
+            print(output_string, end='')
+            print(error_string, end='', file=sys.stderr)
 
-        # there is currently an issue with OutputGrapper on Windows
-        # see issues for details
+        # The output grabber is not enabled for windows
         if self.plt != "Windows":
-            # print adds an extra \n
-            assert output.capturedtext == output_string + "\n"
+            assert output.stdout_grabber.capturedtext == output_string
 
-    def test_incorrect_output(self):
+    def test_correct_stderr(self):
+        """
+        Test that the OutputGrabber grabs the stderr stream.
+        """
         output_string = 'This is the correct output string\nSecond line'
-        incorrect_output_sting = 'This is the incorrect string\n'
+        error_string = 'An error has occurred:\nSome details'
+
         output = OutputGrabber(self.options)
         with output:
-            print(output_string)
-        assert output.capturedtext != incorrect_output_sting
+            print(output_string, end='')
+            print(error_string, end='', file=sys.stderr)
+
+        # The output grabber is not enabled for windows
+        if self.plt != "Windows":
+            assert output.stderr_grabber.capturedtext == error_string
 
 
 if __name__ == "__main__":