Repository: fairlearn/fairlearn

Issue:
Stop MetricFrame errors on nonscalar metrics
#### Is your feature request related to a problem? Please describe.

`MetricFrame` is quite relaxed as to the return values of the metric functions it evaluates. For example, it is perfectly possible to return a confusion matrix or a confidence bound from a metric function.

However such metrics will cause a problem if the user tries calling `group_min()`, `group_max()`, `difference()` or `ratio()` on the resultant `MetricFrame` since those operations are only well defined for scalars.

Currently, we (or rather `pandas`) will throw an exception. This is defensible if only a single metric is being computed by the `MetricFrame` but is likely to lead to a lack of user delight if there are other metrics present for which those operations do make sense. In particular, if a user has constructed a `MetricFrame` consisting of both the `accuracy_score` and `accuracy_score_confidence_interval` then they would reasonably expect `group_min()` to give a useful answer for the `accuracy_score` column.

#### Describe the solution you'd like

Catch any exceptions from `group_min()` and `group_max()` and emplace `NaN` in this case.

#### Describe alternatives you've considered, if relevant

Without this, users are forced to split up related metrics into separate `MetricFrame` objects.

#### Additional context

This is important for #235 


Comments:

Patch:
diff --git a/AUTHORS.md b/AUTHORS.md
index fb6b376..03fcc42 100644
--- a/AUTHORS.md
+++ b/AUTHORS.md
@@ -28,6 +28,7 @@ All names are sorted alphabetically by last name. Contributors, please add your
 - [Stephen Robicheaux](https://github.com/stephenrobic)
 - [Kevin Robinson](https://github.com/kevinrobinson)
 - [Mehrnoosh Sameki](https://github.com/mesameki)
+- [Bram Schut](http://github.com/bram49)
 - [Chinmay Singh](https://www.microsoft.com/en-us/research/people/chsingh/)
 - [Hanna Wallach](https://www.microsoft.com/en-us/research/people/wallach/)
 - [Hilde Weerts](https://github.com/hildeweerts)
diff --git a/docs/user_guide/installation_and_version_guide/v0.7.1.rst b/docs/user_guide/installation_and_version_guide/v0.7.1.rst
index aebdbae..b4d5102 100644
--- a/docs/user_guide/installation_and_version_guide/v0.7.1.rst
+++ b/docs/user_guide/installation_and_version_guide/v0.7.1.rst
@@ -9,3 +9,6 @@ v0.7.1
 * Relaxed checks made on :code:`X` in :code:`_validate_and_reformat_input()`
   since that is the concern of the underlying estimator and not Fairlearn.
 * Add support for Python 3.9
+* Added error handling in :code:`MetricFrame`. Methods :code:`group_max`, :code:`group_min`,
+  :code:`difference` and :code:`ratio` now accept :code:`errors` as a parameter,
+  which could either be :code:`raise` or :code:`coerce`.
diff --git a/fairlearn/metrics/_metric_frame.py b/fairlearn/metrics/_metric_frame.py
index 9826c3d..8e3ac68 100644
--- a/fairlearn/metrics/_metric_frame.py
+++ b/fairlearn/metrics/_metric_frame.py
@@ -18,6 +18,8 @@ from ._group_feature import GroupFeature
 logger = logging.getLogger(__name__)
 
 _SUBGROUP_COUNT_WARNING_THRESHOLD = 20
+_VALID_ERROR_STRING = ['raise', 'coerce']
+_VALID_GROUPING_FUNCTION = ['min', 'max']
 
 _SF_DICT_CONVERSION_FAILURE = "DataFrame.from_dict() failed on sensitive features. " \
     "Please ensure each array is strictly 1-D."
@@ -29,6 +31,12 @@ _DUPLICATE_FEATURE_NAME = "Detected duplicate feature name: '{0}'"
 _TOO_MANY_FEATURE_DIMS = "Feature array has too many dimensions"
 _SAMPLE_PARAM_KEYS_NOT_IN_FUNC_DICT = \
     "Keys in 'sample_params' do not match those in 'metric'"
+_INVALID_ERRORS_VALUE_ERROR_MESSAGE = "Invalid error value specified. " \
+                                      "Valid values are {0}".format(_VALID_ERROR_STRING)
+_INVALID_GROUPING_FUNCTION_ERROR_MESSAGE = \
+    "Invalid grouping function specified. Valid values are {0}".format(_VALID_GROUPING_FUNCTION)
+_MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE = "Metric frame contains non-scalar cells. " \
+    "Please remove non-scalar columns from your metric frame or use parameter errors='coerce'."
 
 
 def _deprecate_metric_frame_init(new_metric_frame_init):
@@ -372,29 +380,72 @@ class MetricFrame:
         """
         return self._sf_names
 
-    def group_max(self) -> Union[Any, pd.Series, pd.DataFrame]:
-        """Return the maximum value of the metric over the sensitive features.
+    def __group(self, grouping_function: str, errors: str = 'raise') \
+            -> Union[Any, pd.Series, pd.DataFrame]:
+        """Return the minimum/maximum value of the metric over the sensitive features.
 
-        This method computes the maximum value over all combinations of
-        sensitive features for each underlying metric function in the :attr:`.by_group`
-        property (it will only succeed if all the underlying metric
-        functions return scalar values). The exact return type depends on
-        whether control features are present, and whether the metric functions
-        were specified as a single callable or a dictionary.
+        This is a private method, please use .group_min() or .group_max() instead.
+
+        Parameters
+        ----------
+        grouping_function: {'min', 'max'}
+        errors: {'raise', 'coerce'}, default 'raise'
+        if 'raise', then invalid parsing will raise an exception
+        if 'coerce', then invalid parsing will be set as NaN
 
         Returns
         -------
-        typing.Any or pandas.Series or pandas.DataFrame
-            The maximum value over sensitive features. The exact type
+        typing.Any pandas.Series or pandas.DataFrame
+            The minimum value over sensitive features. The exact type
             follows the table in :attr:`.MetricFrame.overall`.
         """
+        if grouping_function not in _VALID_GROUPING_FUNCTION:
+            raise ValueError(_INVALID_GROUPING_FUNCTION_ERROR_MESSAGE)
+
+        if errors not in _VALID_ERROR_STRING:
+            raise ValueError(_INVALID_ERRORS_VALUE_ERROR_MESSAGE)
+
         if not self.control_levels:
-            result = pd.Series(index=self._by_group.columns, dtype='object')
-            for m in result.index:
-                max_val = self._by_group[m].max()
-                result[m] = max_val
+            if errors == "raise":
+                try:
+                    mf = self._by_group
+                    if grouping_function == 'min':
+                        vals = [mf[m].min() for m in mf.columns]
+                    else:
+                        vals = [mf[m].max() for m in mf.columns]
+
+                    result = pd.Series(vals, index=self._by_group.columns, dtype='object')
+                except ValueError as ve:
+                    raise ValueError(_MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE) from ve
+            elif errors == 'coerce':
+                if not self.control_levels:
+                    mf = self._by_group
+                    # Fill in the possible min/max values, else np.nan
+                    if grouping_function == 'min':
+                        vals = [mf[m].min() if np.isscalar(mf[m].values[0])
+                                else np.nan for m in mf.columns]
+                    else:
+                        vals = [mf[m].max() if np.isscalar(mf[m].values[0])
+                                else np.nan for m in mf.columns]
+
+                    result = pd.Series(vals, index=mf.columns, dtype='object')
         else:
-            result = self._by_group.groupby(level=self.control_levels).max()
+            if errors == 'raise':
+                try:
+                    if grouping_function == 'min':
+                        result = self._by_group.groupby(level=self.control_levels).min()
+                    else:
+                        result = self._by_group.groupby(level=self.control_levels).max()
+                except ValueError as ve:
+                    raise ValueError(_MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE) from ve
+            elif errors == 'coerce':
+                # Fill all impossible columns with NaN before grouping metric frame
+                mf = self._by_group.copy()
+                mf = mf.applymap(lambda x: x if np.isscalar(x) else np.nan)
+                if grouping_function == 'min':
+                    result = mf.groupby(level=self.control_levels).min()
+                else:
+                    result = mf.groupby(level=self.control_levels).max()
 
         if self._user_supplied_callable:
             if self.control_levels:
@@ -404,40 +455,57 @@ class MetricFrame:
         else:
             return result
 
-    def group_min(self) -> Union[Any, pd.Series, pd.DataFrame]:
-        """Return the minimum value of the metric over the sensitive features.
+    def group_max(self, errors: str = 'raise') -> Union[Any, pd.Series, pd.DataFrame]:
+        """Return the maximum value of the metric over the sensitive features.
 
-        This method computes the minimum value over all combinations of
+        This method computes the maximum value over all combinations of
         sensitive features for each underlying metric function in the :attr:`.by_group`
         property (it will only succeed if all the underlying metric
         functions return scalar values). The exact return type depends on
         whether control features are present, and whether the metric functions
         were specified as a single callable or a dictionary.
 
+        Parameters
+        ----------
+        errors: {'raise', 'coerce'}, default 'raise'
+            if 'raise', then invalid parsing will raise an exception
+            if 'coerce', then invalid parsing will be set as NaN
+
         Returns
         -------
-        typing.Any pandas.Series or pandas.DataFrame
-            The minimum value over sensitive features. The exact type
+        typing.Any or pandas.Series or pandas.DataFrame
+            The maximum value over sensitive features. The exact type
             follows the table in :attr:`.MetricFrame.overall`.
         """
-        if not self.control_levels:
-            result = pd.Series(index=self._by_group.columns, dtype='object')
-            for m in result.index:
-                min_val = self._by_group[m].min()
-                result[m] = min_val
-        else:
-            result = self._by_group.groupby(level=self.control_levels).min()
+        return self.__group('max', errors)
 
-        if self._user_supplied_callable:
-            if self.control_levels:
-                return result.iloc[:, 0]
-            else:
-                return result.iloc[0]
-        else:
-            return result
+    def group_min(self, errors: str = 'raise') -> Union[Any, pd.Series, pd.DataFrame]:
+        """Return the maximum value of the metric over the sensitive features.
+
+        This method computes the minimum value over all combinations of
+        sensitive features for each underlying metric function in the :attr:`.by_group`
+        property (it will only succeed if all the underlying metric
+        functions return scalar values). The exact return type depends on
+        whether control features are present, and whether the metric functions
+        were specified as a single callable or a dictionary.
+
+        Parameters
+        ----------
+        errors: {'raise', 'coerce'}, default 'raise'
+            if 'raise', then invalid parsing will raise an exception
+            if 'coerce', then invalid parsing will be set as NaN
+
+        Returns
+        -------
+        typing.Any or pandas.Series or pandas.DataFrame
+            The maximum value over sensitive features. The exact type
+            follows the table in :attr:`.MetricFrame.overall`.
+        """
+        return self.__group('min', errors)
 
     def difference(self,
-                   method: str = 'between_groups') -> Union[Any, pd.Series, pd.DataFrame]:
+                   method: str = 'between_groups',
+                   errors: str = 'coerce') -> Union[Any, pd.Series, pd.DataFrame]:
         """Return the maximum absolute difference between groups for each metric.
 
         This method calculates a scalar value for each underlying metric by
@@ -461,24 +529,38 @@ class MetricFrame:
         ----------
         method : str
             How to compute the aggregate. Default is :code:`between_groups`
+        errors: {'raise', 'coerce'}, default 'coerce'
+            if 'raise', then invalid parsing will raise an exception
+            if 'coerce', then invalid parsing will be set as NaN
 
         Returns
         -------
         typing.Any or pandas.Series or pandas.DataFrame
             The exact type follows the table in :attr:`.MetricFrame.overall`.
         """
-        subtrahend = np.nan
+        if errors not in _VALID_ERROR_STRING:
+            raise ValueError(_INVALID_ERRORS_VALUE_ERROR_MESSAGE)
+
         if method == 'between_groups':
-            subtrahend = self.group_min()
+            subtrahend = self.group_min(errors=errors)
         elif method == 'to_overall':
             subtrahend = self.overall
         else:
             raise ValueError("Unrecognised method '{0}' in difference() call".format(method))
 
-        return (self.by_group - subtrahend).abs().max(level=self.control_levels)
+        mf = self.by_group.copy()
+        # Can assume errors='coerce', else error would already have been raised in .group_min
+        # Fill all non-scalar values with NaN
+        if isinstance(mf, pd.Series):
+            mf = mf.map(lambda x: x if np.isscalar(x) else np.nan)
+        else:
+            mf = mf.applymap(lambda x: x if np.isscalar(x) else np.nan)
+
+        return (mf - subtrahend).abs().max(level=self.control_levels)
 
     def ratio(self,
-              method: str = 'between_groups') -> Union[Any, pd.Series, pd.DataFrame]:
+              method: str = 'between_groups',
+              errors: str = 'coerce') -> Union[Any, pd.Series, pd.DataFrame]:
         """Return the minimum ratio between groups for each metric.
 
         This method calculates a scalar value for each underlying metric by
@@ -504,15 +586,21 @@ class MetricFrame:
         ----------
         method : str
             How to compute the aggregate. Default is :code:`between_groups`
+        errors: {'raise', 'coerce'}, default 'coerce'
+            if 'raise', then invalid parsing will raise an exception
+            if 'coerce', then invalid parsing will be set as NaN
 
         Returns
         -------
         typing.Any or pandas.Series or pandas.DataFrame
             The exact type follows the table in :attr:`.MetricFrame.overall`.
         """
+        if errors not in _VALID_ERROR_STRING:
+            raise ValueError(_INVALID_ERRORS_VALUE_ERROR_MESSAGE)
+
         result = None
         if method == 'between_groups':
-            result = self.group_min() / self.group_max()
+            result = self.group_min(errors=errors) / self.group_max(errors=errors)
         elif method == 'to_overall':
             if self._user_supplied_callable:
                 tmp = self.by_group / self.overall


Test patch:
diff --git a/test/unit/metrics/test_metricframe_aggregates.py b/test/unit/metrics/test_metricframe_aggregates.py
index 249fa94..da3df53 100644
--- a/test/unit/metrics/test_metricframe_aggregates.py
+++ b/test/unit/metrics/test_metricframe_aggregates.py
@@ -1,5 +1,6 @@
 # Copyright (c) Microsoft Corporation and Fairlearn contributors.
 # Licensed under the MIT License.
+import functools
 
 import numpy as np
 import pandas as pd
@@ -9,7 +10,10 @@ import sklearn.metrics as skm
 import fairlearn.metrics as metrics
 
 from .data_for_test import y_t, y_p, g_1, g_2, g_3, g_4
-
+from fairlearn.metrics._metric_frame import (
+    _INVALID_ERRORS_VALUE_ERROR_MESSAGE,
+    _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE
+)
 
 metric = [skm.recall_score,
           skm.precision_score,
@@ -1040,3 +1044,670 @@ def test_2m_1sf_2cf():
         assert ratios_overall[mfn][('m', 'aa')] == min(ratio_overall_m_a)
         assert ratios_overall[mfn][('m', 'ba')] == pytest.approx(min(ratio_overall_m_b),
                                                                  rel=1e-10, abs=1e-16)
+
+
+class Test2m1sf1cfErrorHandlingCM:
+    # Metricframe containing a confusion matrix raises ValueErrors, since cells are non-scalar
+    def _prepare(self):
+        fns = {'recall': skm.recall_score, 'confusion_matrix': skm.confusion_matrix}
+        self.target = metrics.MetricFrame(metrics=fns, y_true=y_t, y_pred=y_p,
+                                          sensitive_features=g_2,
+                                          control_features=g_3)
+
+        self.expected = metrics.MetricFrame(metrics={'recall': skm.recall_score},
+                                            y_true=y_t,
+                                            y_pred=y_p,
+                                            sensitive_features=g_2,
+                                            control_features=g_3)
+
+    def test_min_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_coerce(self):
+        self._prepare()
+        target_mins = self.target.group_min(errors='coerce')
+        expected_mins = self.expected.group_min()
+        expected_mins['confusion_matrix'] = np.nan
+        assert target_mins.equals(expected_mins)
+
+    def test_min_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_coerce(self):
+        self._prepare()
+        target_maxs = self.target.group_max(errors='coerce')
+        expected_maxs = self.expected.group_max()
+        expected_maxs['confusion_matrix'] = np.nan
+        assert target_maxs.equals(expected_maxs)
+
+    def test_max_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_coerce(self):
+        self._prepare()
+        target_differences = self.target.difference(errors='coerce')
+        expected_differences = self.expected.difference()
+        expected_differences['confusion_matrix'] = np.nan
+        assert target_differences.equals(expected_differences)
+
+    def test_difference_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_differences = self.target.difference()
+        expected_differences = self.expected.difference()
+        expected_differences['confusion_matrix'] = np.nan
+        assert target_differences.equals(expected_differences)
+
+    def test_ratio_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_coerce(self):
+        self._prepare()
+        target_ratio = self.target.ratio(errors='coerce')
+        expected_ratio = self.expected.ratio()
+        expected_ratio['confusion_matrix'] = np.nan
+        assert target_ratio.equals(expected_ratio)
+
+    def test_ratio_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_ratio = self.target.ratio()
+        expected_ratio = self.expected.ratio()
+        expected_ratio['confusion_matrix'] = np.nan
+        assert target_ratio.equals(expected_ratio)
+
+
+class Test1m1sf1cfErrorHandlingCM:
+    # Metricframe containing a confusion matrix raises ValueErrors, since cells are non-scalar
+    def _prepare(self):
+        fns = {'confusion_matrix': skm.confusion_matrix}
+        self.target = metrics.MetricFrame(metrics=fns, y_true=y_t, y_pred=y_p,
+                                          sensitive_features=g_2,
+                                          control_features=g_3)
+
+        self.expected = metrics.MetricFrame(metrics={}, y_true=y_t, y_pred=y_p,
+                                            sensitive_features=g_2,
+                                            control_features=g_3)
+
+    def test_min_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_coerce(self):
+        self._prepare()
+        target_mins = self.target.group_min(errors='coerce')
+        expected_mins = self.expected.group_min()
+        expected_mins['confusion_matrix'] = np.nan
+        assert target_mins.equals(expected_mins)
+
+    def test_min_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_coerce(self):
+        self._prepare()
+        target_maxs = self.target.group_max(errors='coerce')
+        expected_maxs = self.expected.group_max()
+        expected_maxs['confusion_matrix'] = np.nan
+        assert target_maxs.equals(expected_maxs)
+
+    def test_max_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_coerce(self):
+        self._prepare()
+        target_differences = self.target.difference(errors='coerce')
+        expected_differences = self.expected.difference()
+        expected_differences['confusion_matrix'] = np.nan
+        assert target_differences.equals(expected_differences)
+
+    def test_difference_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_differences = self.target.difference()
+        expected_differences = self.expected.difference()
+        expected_differences['confusion_matrix'] = np.nan
+        assert target_differences.equals(expected_differences)
+
+    def test_ratio_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_coerce(self):
+        self._prepare()
+        target_ratio = self.target.ratio(errors='coerce')
+        expected_ratio = self.expected.ratio()
+        expected_ratio['confusion_matrix'] = np.nan
+        assert target_ratio.equals(expected_ratio)
+
+    def test_ratio_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_ratio = self.target.ratio()
+        expected_ratio = self.expected.ratio()
+        expected_ratio['confusion_matrix'] = np.nan
+        assert target_ratio.equals(expected_ratio)
+
+
+class Test2m1sf0cfErrorHandlingCM:
+    # Metricframe containing a confusion matrix raises ValueErrors, since cells are non-scalar
+    def _prepare(self):
+        fns = {'recall': skm.recall_score, 'confusion_matrix': skm.confusion_matrix}
+        self.target = metrics.MetricFrame(metrics=fns, y_true=y_t, y_pred=y_p,
+                                          sensitive_features=g_2)
+
+        self.expected = metrics.MetricFrame(metrics={'recall': skm.recall_score},
+                                            y_true=y_t,
+                                            y_pred=y_p,
+                                            sensitive_features=g_2)
+
+    def test_min_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_coerce(self):
+        self._prepare()
+        target_mins = self.target.group_min(errors='coerce')
+        expected_mins = self.expected.group_min()
+        expected_mins['confusion_matrix'] = np.nan
+        assert target_mins.equals(expected_mins)
+
+    def test_min_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_coerce(self):
+        self._prepare()
+        target_maxs = self.target.group_max(errors='coerce')
+        expected_maxs = self.expected.group_max()
+        expected_maxs['confusion_matrix'] = np.nan
+        assert target_maxs.equals(expected_maxs)
+
+    def test_max_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_coerce(self):
+        self._prepare()
+        target_differences = self.target.difference(errors='coerce')
+        expected_differences = self.expected.difference()
+        expected_differences['confusion_matrix'] = np.nan
+        assert target_differences.equals(expected_differences)
+
+    def test_difference_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_differences = self.target.difference()
+        expected_differences = self.expected.difference()
+        expected_differences['confusion_matrix'] = np.nan
+        assert target_differences.equals(expected_differences)
+
+    def test_ratio_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_coerce(self):
+        self._prepare()
+        target_ratio = self.target.ratio(errors='coerce')
+        expected_ratio = self.expected.ratio()
+        expected_ratio['confusion_matrix'] = np.nan
+        assert target_ratio.equals(expected_ratio)
+
+    def test_ratio_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_ratio = self.target.ratio()
+        expected_ratio = self.expected.ratio()
+        expected_ratio['confusion_matrix'] = np.nan
+        assert target_ratio.equals(expected_ratio)
+
+
+class Test1m1sf0cfErrorHandlingCM:
+    # Metricframe containing a confusion matrix raises ValueErrors, since cells are non-scalar
+    def _prepare(self):
+        fns = {'confusion_matrix': skm.confusion_matrix}
+        self.target = metrics.MetricFrame(metrics=fns, y_true=y_t, y_pred=y_p,
+                                          sensitive_features=g_2)
+
+    def test_min_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_coerce(self):
+        self._prepare()
+        target_mins = self.target.group_min(errors='coerce')
+        assert target_mins.isna().all()
+
+    def test_min_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_coerce(self):
+        self._prepare()
+        target_maxs = self.target.group_max(errors='coerce')
+        assert target_maxs.isna().all()
+
+    def test_max_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_coerce(self):
+        self._prepare()
+        target_differences = self.target.difference(errors='coerce')
+        assert target_differences.isna().all()
+
+    def test_difference_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_differences = self.target.difference()
+        assert target_differences.isna().all()
+
+    def test_ratio_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_coerce(self):
+        self._prepare()
+        target_ratio = self.target.ratio(errors='coerce')
+        assert target_ratio.isna().all()
+
+    def test_ratio_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_ratio = self.target.ratio()
+        assert target_ratio.isna().all()
+
+
+class Test2m1sf0cfErrorHandlingCM2:
+    # Metricframe containing a confusion matrix raises ValueErrors, since cells are non-scalar
+    # Tests for two non-scalar columns in MetricFrame.
+    def _prepare(self):
+        fns = {'confusion_matrix1': skm.confusion_matrix,
+               'confusion_matrix2': functools.partial(skm.confusion_matrix, normalize='all')}
+        self.target = metrics.MetricFrame(metrics=fns, y_true=y_t, y_pred=y_p,
+                                          sensitive_features=g_2)
+
+    def test_min_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_coerce(self):
+        self._prepare()
+        target_mins = self.target.group_min(errors='coerce')
+        assert target_mins.isna().all()
+
+    def test_min_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_coerce(self):
+        self._prepare()
+        target_maxs = self.target.group_max(errors='coerce')
+        assert target_maxs.isna().all()
+
+    def test_max_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_coerce(self):
+        self._prepare()
+        target_differences = self.target.difference(errors='coerce')
+        assert target_differences.isna().all()
+
+    def test_difference_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_differences = self.target.difference()
+        assert target_differences.isna().all()
+
+    def test_ratio_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_coerce(self):
+        self._prepare()
+        target_ratio = self.target.ratio(errors='coerce')
+        assert target_ratio.isna().all()
+
+    def test_ratio_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_ratio = self.target.ratio()
+        assert target_ratio.isna().all()
+
+
+class Test1m1sf0cfErrorHandlingSeries:
+    # Metricframe containing a confusion matrix raises ValueErrors, since cells are non-scalar
+    # Tests for two non-scalar columns in MetricFrame.
+    def _prepare(self):
+        self.target = metrics.MetricFrame(metrics=skm.confusion_matrix, y_true=y_t, y_pred=y_p,
+                                          sensitive_features=g_2)
+
+    def test_min_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_coerce(self):
+        self._prepare()
+        target_mins = self.target.group_min(errors='coerce')
+        assert np.isnan(target_mins)
+
+    def test_min_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_min_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_min(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_coerce(self):
+        self._prepare()
+        target_maxs = self.target.group_max(errors='coerce')
+        assert np.isnan(target_maxs)
+
+    def test_max_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_max_default(self):
+        # default is 'raise'
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.group_max(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_coerce(self):
+        self._prepare()
+        target_differences = self.target.difference(errors='coerce')
+        assert np.isnan(target_differences)
+
+    def test_difference_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.difference(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_difference_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_differences = self.target.difference()
+        assert np.isnan(target_differences)
+
+    def test_ratio_raise(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='raise')
+        assert _MF_CONTAINS_NON_SCALAR_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_coerce(self):
+        self._prepare()
+        target_ratio = self.target.ratio(errors='coerce')
+        assert np.isnan(target_ratio)
+
+    def test_ratio_wrong_input(self):
+        self._prepare()
+        with pytest.raises(ValueError) as exc:
+            self.target.ratio(errors='WRONG')
+        assert _INVALID_ERRORS_VALUE_ERROR_MESSAGE == exc.value.args[0]
+
+    def test_ratio_default(self):
+        # default is 'coerce'
+        self._prepare()
+        target_ratio = self.target.ratio()
+        assert np.isnan(target_ratio)