Repository: fairlearn/fairlearn

Issue:
ErrorRate.load_data does not use what is passed as argument for control_features param
The signature of `ErrorRate.load_data` is: `load_data(self, X, y, *, sensitive_features, control_features=None)`.

The `control_features` param is however not used / ignored inside the method. Should this param be deprecated or removed?

Discussion:
riedgar-ms: Thanks for looking at this; tagging @MiroDudik  who knows most about reductions. It appears that the changes to the API need propagating a little further?
taharallouche: > I would suggest to add a helper function to filter kwargs needed to load data to the objective object in _Lagrangian.__init__(), but please see this only as a suggestion. I am happy to reverse that or do it differently, when it fits better to the overall design decisions for fairlearn.

I gave it a thought and I can't find a better solution that wouldn't require redesigning the whole module :+1: 


--------

For the record, I tried another solution, namely:
- change the `Lagrangian`'s `init` signature, adding a two new arguments `sensitive_features` and `control_features` which default to `None`, and removing `**kwargs`.
- forward both of them to the constraints' `load_data`, and only `sensitive_features` to the objective's `load_data`

but it doesn't work because some constraints don't support `control_features` (`BoundedGroupLoss` for instance) :disappointed: 
MiroDudik: Hi folks. Joining a conversation a bit late, but let me first clarify a few things and then make a suggestion.

* `load_data` method is meant to be parallel to the `fit` method in estimators; we made the decision to use `X,y,**kwargs` provided to `fit()` in `ExponentiatedGradient` in a pass-through manner, so there shouldn't really be any logic (including validation or standardization) related to them until the `Moments` class uses them
* I agree that listing `control_features` and `sensitive_features` in the signature of `load_data()` in `ErrorRate` is weird if they're not being used (we might have similar issues in other moments classes?)
  * one solution would be to change the signatures of `load_data` methods to something like `load_data(X,y,*,<additional used arguments>, **kwargs)` where `**kwargs` would ensure that the method works also if a bunch of extra arguments that it doesn't use are passed to it 
  * another solution would be to use introspection in the `Lagrangian` constructor, that is `inspect.signature(obj.load_data)` and `inspect.signature(constraints.load_data)` and only pass on the `kwargs` that are listed as keyword arguments in the corresponding `load_data` method
* `gamma` actually returns a "vector of moments"
  * when the `Moment` object is used as a constraint, then "`gamma[j]`&leq;0 for all j" is used as the set of constraints (in that sense it captures constraint violation)
  * when the `Moment` object is used as an objective, then `gamma[0]` is used as the objective
* `load_data` and `gamma` are actually not designed for the end user; these are methods that need to be implemented by anybody who wants to introduce a new objective or a new set of constraints--they are implementing an internal API to support `Lagrangian` and `ExponentiatedGradient`. That said--we don't do the best job documenting this internal API (but the audience for this documentation are package developers/contributors, not necessarily package users)

My suggestion:
* I think that in this PR, we should just focus on adding an example of `ErrorRate` and fixing some related documentation issues (e.g., we could correct the description of `gamma` in the files that are touched)
* we can create a separate issue, where we discuss if we want to tweak the internal API around `load_data`
StefanieSenger: I have now removed the removing of `control_features` from this PR and it can be dealt with separately.

> gamma actually returns a "vector of moments"
> 
>     when the Moment object is used as a constraint, then "gamma[j]≤0 for all j" is used as the set of constraints (in that sense it captures constraint violation)
>     when the Moment object is used as an objective, then gamma[0] is used as the objective

Thank you for the explanations, @MiroDudik. I have used this almost literally on the docstring.

Patch:
diff --git a/docs/user_guide/mitigation/reductions.rst b/docs/user_guide/mitigation/reductions.rst
index df1707c..319ee40 100644
--- a/docs/user_guide/mitigation/reductions.rst
+++ b/docs/user_guide/mitigation/reductions.rst
@@ -194,7 +194,7 @@ the predicted labels.
     ...                                      y_pred=y_pred,
     ...                                      sensitive_features=pd.Series(sensitive_features, name="SF 0"))
     >>> selection_rate_summary.overall.item()
-        0.4
+    0.4
     >>> selection_rate_summary.by_group
     SF 0
     a    0.6
@@ -351,6 +351,70 @@ and false positive rate.
                    b           0.2380...
     dtype: float64
 
+
+.. _error_rate:
+
+Error Rate
+~~~~~~~~~~
+
+We can use the :class:`ErrorRate` either to measure performance of a trained model that does or
+does not take sensitive features into account, or as an
+objective function during the training of a fairness aware model.
+
+To measure the ErrorRate in respect to a trained estimator we use its :code:`gamma` method:
+
+.. doctest:: mitigation_reductions
+
+    >>> from fairlearn.reductions import ErrorRate
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.model_selection import train_test_split
+    >>> from sklearn.linear_model import LogisticRegression
+    >>> import numpy as np
+    >>> rng = np.random.default_rng(42) 
+    >>> X, y = make_classification(n_features=10, class_sep=0.1, random_state=42)
+    >>> X[:, -1] = rng.integers(0, 2, size=(X.shape[0],)) # defining the sensitive feature
+    >>> sensitive_features = X[:, -1]
+    >>> classifier = LogisticRegression().fit(X, y)
+    >>> costs = {"fp":0.1, "fn":0.9}
+    >>> errorrate = ErrorRate(costs=costs)
+    >>> errorrate.load_data(X, y, sensitive_features=sensitive_features)
+    >>> errorrate.gamma(classifier.predict)
+    all    0.139
+    dtype: float64
+
+
+Using :class:`ErrorRate` as part of a reductions approach for fairness mitigation is equivalent
+to a cost-sensitive classification problem.
+
+.. doctest:: mitigation_reductions
+
+    >>> from fairlearn.reductions import ErrorRate, EqualizedOdds, ExponentiatedGradient
+    >>> from fairlearn.metrics import MetricFrame
+    >>> from sklearn.metrics import accuracy_score
+    >>> import numpy as np
+    >>> rng = np.random.default_rng(42) 
+    >>> X, y = make_classification(n_features=10, class_sep=0.1, random_state=42)
+    >>> X[:, -1] = rng.integers(0, 2, size=(X.shape[0],)) # defining the sensitive feature
+    >>> sensitive_features = X[:, -1]
+    >>> objective = ErrorRate(costs={"fp":0.1, "fn":0.9})
+    >>> constraint = EqualizedOdds(difference_bound=0.01)
+    >>> classifier = LogisticRegression()
+    >>> mitigator = ExponentiatedGradient(classifier, constraint, objective=objective)
+    >>> X_train, X_test, y_train, y_test, sensitive_train, sensitive_test = train_test_split(
+    ... X, y, sensitive_features, test_size=0.33, random_state=42)
+    >>> mitigator.fit(X_train, y_train, sensitive_features=sensitive_train) # doctest: +ELLIPSIS
+    ExponentiatedGradient(...)
+    >>> y_pred = mitigator.predict(X_test)
+    >>> mf_mitigated = MetricFrame(metrics=accuracy_score, y_true=y_test, y_pred=y_pred, sensitive_features=sensitive_test)
+    >>> mf_mitigated.overall.item()
+    0.5151515151515151
+    >>> mf_mitigated.by_group
+    sensitive_feature_0
+    0.0    0.611111
+    1.0    0.400000
+    Name: accuracy_score, dtype: float64
+
+
 .. _error_rate_parity:
 
 Error Rate Parity
@@ -375,6 +439,10 @@ the overall error rate by more than the value of :code:`difference_bound`.
 
     >>> from fairlearn.reductions import ErrorRateParity
     >>> from sklearn.metrics import accuracy_score
+    >>> X                  = np.array([[0], [1], [2], [3], [4], [5], [6], [7], [8], [9]])
+    >>> y_true             = np.array([ 1 ,  1 ,  1 ,  1 ,  1,   1 ,  1 ,  0 ,  0 ,  0 ])
+    >>> y_pred             = np.array([ 1 ,  1 ,  1 ,  1 ,  0,   0 ,  0 ,  1 ,  0 ,  0 ])
+    >>> sensitive_features = np.array(["a", "b", "a", "a", "b", "a", "b", "b", "a", "b"])
     >>> accuracy_summary = MetricFrame(metrics=accuracy_score,
     ...                                y_true=y_true,
     ...                                y_pred=y_pred,
diff --git a/fairlearn/reductions/_moments/error_rate.py b/fairlearn/reductions/_moments/error_rate.py
index 5ca4a9c..81d0108 100644
--- a/fairlearn/reductions/_moments/error_rate.py
+++ b/fairlearn/reductions/_moments/error_rate.py
@@ -32,13 +32,15 @@ class ErrorRate(ClassificationMoment):
       c_{FP} P[h(X)=1, Y=0] + c_{FN} P[h(X)=0, Y=1]
 
     where :math:`c_{FP}` and :math:`c_{FN}` are the costs of false positive
-    and false negative errors respectively. The standard misclassification
+    and false negative errors respectively. The default misclassification
     error corresponds to :math:`c_{FP}=c_{FN}=1.0`.
 
+    Read more in the :ref:`User Guide <error_rate>`.
+
     Parameters
     ----------
     costs : dict
-        The dictionary with keys :code:`'fp'` and :code:`'fn'` containing the
+        Dictionary with keys :code:`'fp'` and :code:`'fn'` containing the
         costs of false positives and false negatives. If none are provided
         costs of 1.0 are assumed.
     """
@@ -62,7 +64,17 @@ class ErrorRate(ClassificationMoment):
             raise ValueError(_MESSAGE_BAD_COSTS)
 
     def load_data(self, X, y, *, sensitive_features, control_features=None) -> None:
-        """Load the specified data into the object."""
+        """Load the specified data into the object.
+
+        Parameters
+        ----------
+        X : array of shape (n_samples, n_features)
+            The feature array
+        y : array-like of shape (n_samples,)
+            The label vector
+        sensitive_features : array-like of shape (n_samples, n_sensitive_features)
+            The sensitive feature vector
+        """
         _, y_train, sf_train, _ = _validate_and_reformat_input(
             X,
             y,
@@ -70,7 +82,7 @@ class ErrorRate(ClassificationMoment):
             sensitive_features=sensitive_features,
             control_features=control_features,
         )
-        # The following uses X  so that the estimators get X untouched
+        # The following uses X so that the estimators get X untouched
         super().load_data(X, y_train, sensitive_features=sf_train)
         self._index = [_ALL]
 
@@ -80,14 +92,29 @@ class ErrorRate(ClassificationMoment):
         return self._index
 
     def gamma(self, predictor: Callable) -> pd.Series:
-        """Return the gamma values for the given predictor."""
-        pred = predictor(self.X)
-        if isinstance(pred, np.ndarray):
+        """Calculate a vector of moments.
+
+        When ErrorRate() is used as a constraint, then `gamma[j]≤0 for all j` is used as
+        the set of constraints. When ErrorRate() is used as an objective, then
+        `gamma[0]` is used as the objective.
+
+        Parameters
+        ----------
+        predictor : func
+            bound function returning predictions
+
+        Returns
+        -------
+        error : :class:`pandas.Series`
+            gamma value for the predictor
+        """
+        y_pred = predictor(self.X)
+        if isinstance(y_pred, np.ndarray):
             # TensorFlow is returning an (n,1) array, which results
             # in the subtraction in the 'error =' line generating an
             # (n,n) array
-            pred = np.squeeze(pred)
-        signed_errors = self.tags[_LABEL] - pred
+            y_pred = np.squeeze(y_pred)
+        signed_errors = self.tags[_LABEL] - y_pred
         total_fn_cost = np.sum(signed_errors[signed_errors > 0] * self.fn_cost)
         total_fp_cost = np.sum(-signed_errors[signed_errors < 0] * self.fp_cost)
         error_value = (total_fn_cost + total_fp_cost) / self.total_samples
diff --git a/fairlearn/utils/_input_validation.py b/fairlearn/utils/_input_validation.py
index 73865d6..901b410 100644
--- a/fairlearn/utils/_input_validation.py
+++ b/fairlearn/utils/_input_validation.py
@@ -16,7 +16,7 @@ _KW_SENSITIVE_FEATURES = "sensitive_features"
 _KW_CONTROL_FEATURES = "control_features"
 
 _MESSAGE_X_NONE = "Must supply X"
-_MESSAGE_Y_NONE = "Must supply y"
+_MESSAGE_Y_NONE = "Must supply nonempty y"
 _MESSAGE_SENSITIVE_FEATURES_NONE = "Must specify {0} (for now)".format(_KW_SENSITIVE_FEATURES)
 _MESSAGE_X_Y_ROWS = "X and y must have same number of rows"
 _MESSAGE_X_SENSITIVE_ROWS = "X and the sensitive features must have same number of rows"
@@ -56,10 +56,11 @@ def _validate_and_reformat_input(
     ----------
     X : numpy.ndarray, pandas.DataFrame
         The feature matrix
-    y : numpy.ndarray, pandas.DataFrame, pandas.Series, or list
-        The label vector
-    expect_y : bool
-        If True y needs to be provided, otherwise ignores the argument; default True
+    y : numpy.ndarray, pandas.DataFrame, pandas.Series, list, or None
+        The label vector. Must be of shape (n,) or (n,1).
+    expect_y : bool, default=True
+        If True, y needs to be provided and non-empty (np.asarray(y).size > 0),
+        otherwise ignores the argument
     expect_sensitive_features : bool
         If true, sensitive_features must be provided to the call to this function.
         This is the default setting.
@@ -77,25 +78,25 @@ def _validate_and_reformat_input(
         estimator.
 
     """
-
-    if y is not None:
-        # calling check_X_y with a 2-dimensional y causes a warning, so ensure it is 1-dimensional
-        if isinstance(y, np.ndarray) and len(y.shape) == 2 and y.shape[1] == 1:
-            y = y.reshape(-1)
-        elif isinstance(y, pd.DataFrame) and y.shape[1] == 1:
-            y = y.to_numpy().reshape(-1)
-
-        # Using an adapted version of check_array to avoid a warning in sklearn version < 1.6
-        y = check_array(y, ensure_2d=False, dtype="numeric", ensure_all_finite=False)
+    if expect_y:
+        if y is None:
+            raise ValueError(_MESSAGE_Y_NONE + f", got y={y}.")
+        y = np.asarray(y)
+        if y.size == 0:
+            raise ValueError(_MESSAGE_Y_NONE + f", got y={y}.")
+        if not (y.ndim == 1 or (y.ndim == 2 and y.shape[1] == 1)):
+            raise ValueError(f"`y` must be of shape (n,) or (n,1), got y of shape=({y.shape}).")
         if enforce_binary_labels and not set(np.unique(y)).issubset(set([0, 1])):
             raise ValueError(_LABELS_NOT_0_1_ERROR_MESSAGE)
-    elif expect_y:
-        raise ValueError(_MESSAGE_Y_NONE)
+        y = check_array(y.reshape(-1), ensure_2d=False, dtype="numeric", ensure_all_finite=False)
 
     result_X = check_array(X, dtype=None, ensure_all_finite=False, allow_nd=True)
     if isinstance(X, pd.DataFrame):
         result_X = pd.DataFrame(result_X)
 
+    if (y is not None) and y.shape[0] != result_X.shape[0]:
+        raise ValueError(_MESSAGE_X_Y_ROWS)
+
     sensitive_features = kwargs.get(_KW_SENSITIVE_FEATURES)
     if sensitive_features is not None:
         check_consistent_length(X, sensitive_features)


Test patch:
diff --git a/test/unit/fixes.py b/test/unit/fixes.py
deleted file mode 100644
index 72f6841..0000000
--- a/test/unit/fixes.py
+++ /dev/null
@@ -1,13 +0,0 @@
-# Copyright (c) Microsoft Corporation and Fairlearn contributors.
-# Licensed under the MIT License.
-
-import sklearn
-
-
-def get_sklearn_expected_1d_message():
-    # Handle change of message for sklearn
-    if sklearn.__version__ < "0.23.0":
-        expected_message = "bad input shape"
-    else:
-        expected_message = "Data must be 1-dimensional"
-    return expected_message
diff --git a/test/unit/postprocessing/test_threshold_optimization.py b/test/unit/postprocessing/test_threshold_optimization.py
index dec99ed..57b742d 100644
--- a/test/unit/postprocessing/test_threshold_optimization.py
+++ b/test/unit/postprocessing/test_threshold_optimization.py
@@ -1,6 +1,7 @@
 # Copyright (c) Microsoft Corporation and Fairlearn contributors.
 # Licensed under the MIT License.
 
+import re
 from contextlib import nullcontext as does_not_raise
 from copy import deepcopy
 from test.unit.input_convertors import _map_into_single_column
@@ -22,6 +23,7 @@ from fairlearn.postprocessing._tradeoff_curve_utilities import (
 from fairlearn.utils._input_validation import (
     _LABELS_NOT_0_1_ERROR_MESSAGE,
     _MESSAGE_SENSITIVE_FEATURES_NONE,
+    _MESSAGE_X_Y_ROWS,
     _MESSAGE_Y_NONE,
 )
 
@@ -90,17 +92,14 @@ def test_none_input_data(X, y, sensitive_features, constraints):
     )
 
     if y is None:
-        with pytest.raises(ValueError) as exception:
+        with pytest.raises(ValueError, match=re.escape(_MESSAGE_Y_NONE)):
             adjusted_predictor.fit(X, y, sensitive_features=sensitive_features)
-        assert str(exception.value) == _MESSAGE_Y_NONE
     elif X is None:
-        with pytest.raises(ValueError) as exception:
+        with pytest.raises(ValueError, match="Expected 2D array, got scalar array instead"):
             adjusted_predictor.fit(X, y, sensitive_features=sensitive_features)
-        assert "Expected 2D array, got scalar array instead" in str(exception.value)
     elif sensitive_features is None:
-        with pytest.raises(ValueError) as exception:
+        with pytest.raises(ValueError, match=re.escape(_MESSAGE_SENSITIVE_FEATURES_NONE)):
             adjusted_predictor.fit(X, y, sensitive_features=sensitive_features)
-        assert str(exception.value) == _MESSAGE_SENSITIVE_FEATURES_NONE
     else:
         # skip since no arguments are None
         pass
@@ -157,11 +156,13 @@ def test_threshold_optimization_different_input_lengths(data_X_y_sf, constraints
     inconsistent_exception_messages = {
         "inconsistent_sklearn": "Found input variables with inconsistent numbers of samples",
         "inconsistent_pandas": "All arrays must be of the same length",
+        "inconsistent_fairlearn": "X and y must have same number of rows",
     }
 
     empty_exception_messages = {
         "empty_sklearn": "Found array with 0 sample(s) (shape=(0,)) while a minimum of 1 is required.",
         "empty_pandas": "Found array with 0 sample",
+        "empty_fairlearn": _MESSAGE_Y_NONE,
     }
 
     for permutation in [(0, 1), (1, 0)]:
@@ -503,15 +504,14 @@ def test_predict_different_argument_lengths(data_X_y_sf, constraints):
     )
 
     with pytest.raises(
-        ValueError, match="Found input variables with inconsistent numbers of samples"
+        ValueError,
+        match=re.escape("Found input variables with inconsistent numbers of samples: [20, 19]"),
     ):
         adjusted_predictor.predict(
             data_X_y_sf.X, sensitive_features=data_X_y_sf.sensitive_features[:-1]
         )
 
-    with pytest.raises(
-        ValueError, match="Found input variables with inconsistent numbers of samples"
-    ):
+    with pytest.raises(ValueError, match=_MESSAGE_X_Y_ROWS):
         adjusted_predictor.predict(
             data_X_y_sf.X[:-1], sensitive_features=data_X_y_sf.sensitive_features
         )
diff --git a/test/unit/reductions/grid_search/test_grid_search_arguments.py b/test/unit/reductions/grid_search/test_grid_search_arguments.py
index fc60fc0..4119e43 100644
--- a/test/unit/reductions/grid_search/test_grid_search_arguments.py
+++ b/test/unit/reductions/grid_search/test_grid_search_arguments.py
@@ -2,7 +2,7 @@
 # Licensed under the MIT License.
 
 import logging
-from test.unit.fixes import get_sklearn_expected_1d_message
+import re
 from test.unit.input_convertors import (
     conversions_for_1d,
     ensure_dataframe,
@@ -37,6 +37,7 @@ from fairlearn.reductions._grid_search._grid_generator import (
 )
 from fairlearn.utils._input_validation import (
     _LABELS_NOT_0_1_ERROR_MESSAGE,
+    _MESSAGE_X_Y_ROWS,
     _MESSAGE_Y_NONE,
 )
 
@@ -108,11 +109,9 @@ class ArgumentTests:
         )
         X, _, A = _quick_data()
 
-        with pytest.raises(ValueError) as execInfo:
+        with pytest.raises(ValueError, match=_MESSAGE_Y_NONE):
             gs.fit(transformX(X), None, sensitive_features=transformA(A))
 
-        assert _MESSAGE_Y_NONE == execInfo.value.args[0]
-
     # ----------------------------
 
     @pytest.mark.parametrize("transformA", candidate_A_transforms)
@@ -129,12 +128,9 @@ class ArgumentTests:
         X, _, A = _quick_data()
         Y = np.random.randint(2, size=len(A) + 1)
 
-        with pytest.raises(ValueError) as execInfo:
+        with pytest.raises(ValueError, match=_MESSAGE_X_Y_ROWS):
             gs.fit(transformX(X), transformY(Y), sensitive_features=transformA(A))
 
-        expected_exception_message = "Found input variables with inconsistent numbers of samples"
-        assert expected_exception_message in execInfo.value.args[0]
-
     @pytest.mark.parametrize("transformA", candidate_A_transforms)
     @pytest.mark.parametrize("transformY", candidate_Y_transforms)
     @pytest.mark.parametrize("transformX", candidate_X_transforms)
@@ -319,9 +315,13 @@ class ArgumentTests:
         X, Y, A = _quick_data(A_two_dim)
 
         Y_two_col_df = pd.DataFrame({"a": Y, "b": Y})
-        with pytest.raises(ValueError) as execInfo:
+        with pytest.raises(
+            ValueError,
+            match=re.escape(
+                f"`y` must be of shape (n,) or (n,1), got y of shape=({Y_two_col_df.shape})."
+            ),
+        ):
             gs.fit(transformX(X), Y_two_col_df, sensitive_features=transformA(A))
-        assert get_sklearn_expected_1d_message() in execInfo.value.args[0]
 
     @pytest.mark.parametrize("transformA", candidate_A_transforms)
     @pytest.mark.parametrize("transformX", candidate_X_transforms)
@@ -336,9 +336,13 @@ class ArgumentTests:
         X, Y, A = _quick_data(A_two_dim)
 
         Y_two_col_ndarray = np.stack((Y, Y), -1)
-        with pytest.raises(ValueError) as execInfo:
+        with pytest.raises(
+            ValueError,
+            match=re.escape(
+                f"`y` must be of shape (n,) or (n,1), got y of shape=({Y_two_col_ndarray.shape})."
+            ),
+        ):
             gs.fit(transformX(X), Y_two_col_ndarray, sensitive_features=transformA(A))
-        assert get_sklearn_expected_1d_message() in execInfo.value.args[0]
 
     # ----------------------------
 
diff --git a/test/unit/utils/test_utils.py b/test/unit/utils/test_utils.py
index 5c38bab..2e78a1c 100644
--- a/test/unit/utils/test_utils.py
+++ b/test/unit/utils/test_utils.py
@@ -96,3 +96,17 @@ def test_merge_columns_value_error():
         ValueError, match=r"Received argument of type list instead of expected numpy\.ndarray"
     ):
         iv._merge_columns([["A", "1"], ["B", "2"]])
+
+
+@pytest.mark.parametrize(
+    "y", [[], np.asarray([]), pd.Series(dtype="float64"), pd.DataFrame(), None]
+)
+def test_validate_and_reformat_input_empty_y(y):
+    """Test that _validate_and_reformat_input raises as expected when y is expected, but
+    passed as an empty list, nd.array, series or dataframe or None."""
+    X = pd.DataFrame.from_dict({"alpha": ["a", "a", "b"], "beta": [1, 2, 1]})
+
+    with pytest.raises(ValueError, match=iv._MESSAGE_Y_NONE):
+        X, y, _, _ = iv._validate_and_reformat_input(
+            X=X, y=y, expect_y=True, expect_sensitive_features=False
+        )