Sample Selection for Fair and Robust Training

Abstract: Fairness and robustness are critical elements of Trustworthy AI that need to be addressed together. Fairness is about learning an unbiased model while robustness is about learning from corrupted data, and it is known that addressing only one of them may have an adverse affect on the other. In this work, we propose a sample selection-based algorithm for fair and robust training. To this end, we formulate a combinatorial optimization problem for the unbiased selection of samples in the presence of data corruption. Observing that solving this optimization problem is strongly NP-hard, we propose a greedy algorithm that is efficient and effective in practice. Experiments show that our method obtains fairness and robustness that are better than or comparable to the state-of-the-art technique, both on synthetic and benchmark real datasets. Moreover, unlike other fair and robust training baselines, our algorithm can be used by only modifying the sampling step in batch selection without changing the training algorithm or leveraging additional clean data.


Reviews and author responses:

This paper proposes a new framework of fair sample selection. The paper introduces fairness constraints into clean selection and formalize the optimization problem in the same format as the knapsack problem. Then the authors propose an algorithm iteratively uses FairBatch to decide the fraction of the sample size of each group, uses a Greedy algorithm approximate algorithm (for the knapsack problem) to select samples, and update model parameters. The experiments show that the proposed algorithm achieves better fairness performance and sometimes also achieves higher accuracy.  Originality: Improving fairness and robustness by an adaptive sample selection approach is new. The method proposed in this paper combines some well-established methods, including ITLM, FairBatch and a greedy algorithm for the knapsack problem. 

Quality: The submission is technically sound. The results of fairness are well supported. For the robustness, it is unclear to me what type of attacks (e.g., adversarial perturbation, data poisoning) can the proposed method deal with. In the experiments, the authors consider injecting noise into a fraction of training data (or a specific group). Is this the only type of attack that the algorithm is robust to?

Clarity: The submission is well written and organized. As mentioned above, more information on robustness should be addressed.

Significance: The proposed method performs competitive to the SOTA w.r.t. accuracy and outperforms the SOTA w.r.t. fairness.

Other comments:
typo: in Eq. (6), the absolute value symbol should be removed.

=====================================================
After reading the authors' response and others' comments, the contribution w.r.t. robustness is not convincing to me. Thus, I changed my score. Yes.

 We appreciate your thoughtful comments. We give our responses here and will add discussions in the revision.

Q1. Cascade (two-step) approach

A1.
Indeed, the FR-Train paper [1] has considered two-step approaches using L2R [2] (i.e., eliminate noise with clean data, then do fair training), but the result shows large performance degradations when the training data is poisoned. Although L2R can handle some types of biases like class imbalance [2], fairness-related data include more complicated biases regarding the sensitive groups [3]. Thus, we suspect that robust training even with additional data like L2R may also discard samples unnecessarily, as it cannot distinguish poisoned samples that need to be removed from biased samples that should not be removed. 

The results on the two-step approach are mainly empirical, and we do not have a full theoretical justification yet. We made a best effort to choose hyperparameters for the two-step baselines. Looking at our results along with the FR-Train paper's results using L2R, we believe noise and bias are not easily separable. Theoretically investigating the correlation between bias and noise is an interesting future work.

[1] Roh et al., “FR-Train: A Mutual Information-Based Approach to Fair and Robust Training”, ICML 2020.

[2] Ren et al., “Learning to Reweight Examples for Robust Deep Learning”, ICML 2018.

[3] Mehrabi et al., “A Survey on Bias and Fairness in Machine Learning”, ACM Computing Surveys 2021.

Q2. Extending FairBatch to use a validation set

A2.
We agree that extending FairBatch to use additional validation data is an interesting direction. As you said, there are two possible approaches: (1) build a new two-step approach using L2R and (2) make FairBatch itself use additional data for fair and robust training.

The first approach eliminates noise using L2R and then runs FairBatch. We suspect the performance will be similar to the two-step approach results in the FR-Train paper (e.g., Tables 1, 2, and 3 in [1]) where there is a worse tradeoff between accuracy and fairness compared to FR-Train.

The second approach is promising, but we would like to note that constructing an additional clean dataset can be a major limitation. The definitions of bias and noise are not always clear cut and may even change over time. Coming up with a clean dataset for each notion of bias and noise is challenging. We thus explicitly avoid relying on a clean dataset, although it would help if we had one. We suspect the performance will be comparable to the “FR-Train with clean val. set” result, Table 3 in our paper. We will clarify these points in our revision.


 Thanks for the response to my questions. Below are some clarifications I would like to seek:

1. Your claim that fairness and robustness cannot distinguish between bias and noise is a little confusing. Most robust learning approaches will attempt to fix label noise even if the data is biased. Consider, for example, approaches like CRUST[1] or L2R[2]. They rely on ideas like coresets (i.e. selecting subsets of the dataset) or small held-out validation sets to select samples. I can imagine that an approach like ITLM might suffer -- though, I must ask whether is there a theoretical justification or is your finding mainly empirical, and secondly, in the cascaded approach where you do robustness followed by fairness, did you tune the hyper-parameter properly to ensure the best performance? However, if you have access to a small clean & unbiased dataset, you could easily do an approach like L2R (which does a bi-level optimization) followed by a fair cleaning. Again, I am not expecting that you have an answer to everything and that you have run every possible experiment, but is there a more principled and theoretical reason as to why the cascaded approach will not work? I would be skeptical about fairness -> robustness cascade but I think robustness -> fairness seems perfectly doable. 
2. My second point related to the above is why not extend FairBatch (which anyways does a bi-level optimization approach for fairness) to robust learning setting if we have a small clean validation dataset? This seems like the first thing I would try (in addition to the cascade described above) but it also seems like a reasonable baseline to investigate.

Finally, I'm still a little unhappy with the approach as it simply combines two existing known ideas of ITLM and FairBatch and does not consider an approach like FairBatch with a clean validation set (I understand that this is extra work but not something infeasible because the validation set required in approaches like L2R and FairBatch can often be small). 


[1] https://cs.stanford.edu/people/jure/pubs/crust-neurips20.pdf
[2] https://arxiv.org/abs/1803.09050

 We thank you for the insightful comments, which we will address in our revision.

Q1. Novelty of the approach

A1. 
We clarify that the integration of ITML and FairBatch itself is non-trivial, and the end performance is indeed better than FR-Train (explained in A2 below) thanks to the integration. Another takeaway is that simple two-step approaches (e.g., performing data cleaning and then fair training) fail to attain both fairness and robustness because individual techniques cannot distinguish bias from noise. We believe that fair and robust training is a nascent, but rapidly growing field [1]. 

[1] Lee et al. “Machine Learning Robustness, Fairness, and their Convergence”, SIGKDD 2021


Q2. Trade-off curves for FR-Train and our algorithm

A2. 
As per your great comment, we performed more experiments to draw trade-off curves of FR-Train and our algorithm. The table below shows results using the synthetic dataset w.r.t. equalized odds (EO) disparity where the experimental settings are identical to Table 1 in the paper:

\------------------------------------------------ $\ \ \ \ \ \ \ \ \ \ \   $ \------------------------------------------------\
$\ \ \ \ \ \ \ \ \ \ \ \ \ \  $  Our algorithm $\ \ \ \ \  \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $    FR-Train\
\------------------------------------------------ $\ \ \ \ \ \ \ \ \ \ \  $ \------------------------------------------------\
$\ \ \ \ \ \$ Accuracy $\ \ \ \ \ \ \$  EO Disparity  $\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $ Accuracy $\ \ \ \ \ \ \$  EO Disparity \
\------------------------------------------------ $\ \ \ \ \ \ \ \ \ \ \  $ \------------------------------------------------\
$\ \ \ \ \$.726 ± .001$\ \ \ \ \ \ \ \ \$.039 ± .018 $\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $  .710 ± .002 $\ \ \ \ \ \ \ \ \$.046 ± .016       
$\ \ \ \ \$.727 ± .002$\ \ \ \ \ \ \ \ \$.064 ± .005 $\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $  .714 ± .008 $\ \ \ \ \ \ \ \ \$.079 ± .005       
$\ \ \ \ \$.729 ± .001$\ \ \ \ \ \ \ \ \$.082 ± .007 $\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $  .722 ± .005 $\ \ \ \ \ \ \ \ \$.078 ± .010       
$\ \ \ \ \$.732 ± .000$\ \ \ \ \ \ \ \ \$.119 ± .008 $\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $  .730 ± .000 $\ \ \ \ \ \ \ \ \$.088 ± .008       
$\ \ \ \ \$.732 ± .000$\ \ \ \ \ \ \ \ \$.127 ± .001 $\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  $  .711 ± .007 $\ \ \ \ \ \ \ \ \$.139 ± .026       
\------------------------------------------------ $\ \ \ \ \ \ \ \ \ \ \  $ \------------------------------------------------

The trends are consistent with the other results in the paper, where our algorithm usually shows better fairness (i.e., lower disparity) than FR-Train when the accuracy is similar. Another observation is that FR-Train’s trade-off curve is noisy due to its adversarial training. We will add these trade-off curves in our revision.


Q3. Relative differences in measures

A3. 
Thanks for raising this point, and we would like to clarify our approach. In our experiments, we made a best effort to align the accuracy values as much as possible and then compare the fairness values. Even so, we agree there are still ambiguous cases like your example. A common approach is to then compare the change percentages of accuracy and fairness (e.g., [2]). In your example, if there are two (accuracy-unfairness) results A:(0.8, 0.1) and B:(0.82, 0.13), B’s accuracy is 2.5% higher than A, but the unfairness is 30% worse, so A's result is better. We will add this discussion in our revision.

[2] Zemel et al., “Learning Fair Representations”, ICML 2013.


 We thank you for the insightful comments, which we will address in our revision.

Q1. Quality -- type of attack

A1. 
We currently focus on handling label noises including random flipping and poisoning attacks. However, ITLM is also known to handle adversarial perturbation on the features, so we believe our techniques can be extended to other types of attacks as well. We will add these points in our revision.


Q2. Other comments -- typo

A2. 
Thanks for pointing out the typo. We will fix it in our revision.


 We thank you for the insightful comments. We will address the comments in the Main Review and Limitations grouping similar questions together.

Q1. Target robustness

A1. 
We clarify that our current contribution is addressing label noise issues including random flipping and poisoning attacks. We agree that the robustness literature is vast, and we believe that learning from noisy labels is also a major branch of research as this recent survey [1] shows. However, our algorithm can be extended to other robustness tasks as ITLM (the robust training method we build on top of) is also known to perform well against adversarial perturbation on features. Inspired by your suggestion, we will clarify our current research scope and also suggest possible extensions to address adversarial perturbations on features and distributional robustness in our revision.

[1] Song et al., "Learning from Noisy Labels with Deep Neural Networks: A Survey," ArXiv 2020.


Q2. Related work

A2. 
We appreciate the papers [2,3], which study distributional robustness. As we mentioned in A1, we will discuss how to extend our work for distributional robustness and cite these papers in our revision.

[2] Mandal et al., “Ensuring Fairness Beyond the Training Data”, NeurIPS 2020.

[3] Rezaei et al., “Robust Fairness under Covariate Shift”, AAAI 2021.


Q3. Design choice of our algorithm

A3. 
We clarify why our algorithm design is suitable for robustness against label noises (our current focus as explained in A1). Although our algorithm can be categorized as in-processing, it actually does not require the modification of the model or the data, but only the sampling step of batch selection. As a result, our algorithm can be deployed easily with minimal changes. Going back to your suggestion, we agree that if the goal is distributional robustness, then learning fair representations may be the better approach as it is necessary to fix the data. We will add this discussion in our revision.


Q4. ITLM setting in Figure 2

A4. 
We use the full iterative method where ITML iteratively selects clean samples during the training. We will clarify this point in our revision.


Q5. Convergence

A5. 
Thanks for pointing out the convergence problem and suggesting the average prediction strategy. Currently, we do not have theoretical guarantees for convergence. However, both ITLM and FairBatch do have convergence guarantees under some assumptions. Hence, we suspect that our algorithm will converge under reasonable circumstances as well. In our experiments, we did not run into convergence issues so far. Nevertheless, we agree the average prediction strategy is a good solution in general. We will discuss these points in our revision.


 We thank you for the insightful comments. We will address the comments grouping them into three categories: algorithm, experiments, and related work.


Q1. Algorithm

A1. 
We would like to clarify (1) why the joint approach is essential and (2) why we chose to combine ITLM and FairBatch.

The joint optimization is necessary because fairness or robustness techniques cannot distinguish bias and noise and thus perform poorly individually. In your scenario of performing data cleaning before fair training, the data cleaning method is likely to treat bias just like noise and incorrectly fix it before the fair training can do anything. Our experiments also confirm that two-step approaches (eliminate noise, then do fair training) perform far worse than our holistic approach as shown in Tables 1, 2, 5, and 6 (see the two-step approaches ITLM->FairBatch and ITLM->Penalty). Similar observations have been made in the FR-Train work. We thus need to eliminate noise and bias together in a single framework. 

We choose ITLM and FairBatch because they are representative loss-based sampling techniques that enhance robustness and fairness, respectively. Here, loss-based sampling has a significant implementation advantage where it can easily be deployed as a batch selection method, which does not require modifying the model architecture or data processing. We took the natural first step of combining the representative fair and robust training techniques and certainly expect our techniques to generalize for other combinations as well. 

We will clarify these points in our revision.


Q2. Experiments -- Multi-label classification with deep models

A2.
We appreciate the future directions and would like to clarify why we started from a binary label setting. Many important fairness applications (e.g., loan approval and recidivism prediction) use binary labels and group fairness metrics such as demographic parity. Here we solved the non-trivial problem of jointly optimizing fairness and robustness as explained in A1. We totally agree the next step is to extend our methods to multi-label tasks using deep models and will add a discussion in our revision.


Q3. Related work

A3.
We appreciate the paper (RFLearn). Indeed, RFLearn also performs fair and robust training, but solves a different problem: RFLearn mitigates sample selection bias only whereas our algorithm also handles label noises and thus assumes a more adversarial setting. We will cite the paper and add the comparison in our revision.



This paper presents an approach that does fair learning while ensuring it is also robust to label noise. The authors do so by proposing a combinatorial optimization approach that can be solved using a greedy algorithm. They compare their approach to a few baselines and show the superiority of their technique.  I will divide my review into a list of pros and cons.

Pros of this paper:
- The paper proposes a sample selection framework that is aimed at both robustness and fairness. From the empirical results, it is shown that the proposed method outperforms all the baselines considered both in terms of fairness and accuracy measures.
- It is an interesting approach to integrate robustness criteria with fairness constraints and solving it by posing it as a multi-dimensional knapsack problem.
- The authors also provided a detailed empirical analysis of the impact of various factors like noise ratio, fairness constraints, and ERM weights on the model’s performance.
- The authors show the NP-hardness of the optimization problem.

Cons of this work:
- From the novelty perspective, the approach proposed in this paper is not very novel and is highly derived from previous works like ITLM and FB. Apart from this, there are no significant theoretical contributions from the work. 
- This paper seems like combining two well-known ideas, i.e. ITLM for robustness and FairBatch from fairness. This seems a little incremental in my opinion. 
- The experiments were done on binary label datasets like COMPAS, AdultCENSUS using a logistic regression model as classifier. It would be interesting to see the performance and the intricacies of the proposed approach using multi-label classification datasets.
- Furthermore, in order to understand the scalability and effectiveness of the proposed approach in a deep learning setting, it would be interesting to see some experiments on datasets like UTKFace, FairFace using deep models.
- Also, how does the algorithm work when applied to deep models as classifiers? The authors can consider CELEBA task (e.g. smile detection) with gender as a protected attribute, or age/gender recognition in UTKFace with race as a protected attribute. 
- The most important issue in my mind with this paper is why do a joint optimization (like the way proposed here) in the first place. There are literary "N" robust learning algorithms and "M" fair learning approaches (either pre-processing, in-processing and post-processing based), where N and M are large numbers growing by the year. Obviously, it is not possible to compare to all NxM combinations, but why will a joint learning approach perform better than an approach, that first, say filters out noisy examples and then does fair learning? 
- Related to the above and also the second con, the paper basically combines two well-known ideas, i.e. ITLM for robustness and FairBatch from fairness. Why only these two? Even if one could do a joint optimization (like the way proposed), we could easily formulate an approach that can "combine" these in a principled manner. Again, it is not possible to explore every possible combination, but I would have liked the authors to discuss why these two approaches would work well together and why the proposed idea would qualitatively be more principled compared to another such approach. Sorry, this comment is a little vague but I would have just liked some more elaboration on this.
- For example, the authors mention "we are not aware of a general data cleaning method that is designed to explicitly improve model fairness and robustness together". I do not think one needs to do this together. One can clean the data first (there seem to be many stable and off-the-shelf approaches to do so) and then do an in-processing or post-processing approach. Though this would change the "dataset", it is still fine because in the end it is applying fairness only on the useful dataset.
- Also the authors can check out a recent paper (https://arxiv.org/pdf/2105.11570.pdf). There seems to be some similarities in the approach (again, I don't think this is published work so it might be ok to not compare against them but it would be good to comment on the differences).

Overall, I like the idea of doing fairness and robustness jointly. ML papers are often focused on only one desiderata along with accuracy (i.e. either only robustness and fairness), so this paper is solving the "right" problem and is headed in the right direction. However, I do think there are issues with this paper. One of the major one is why does it not work to just blindly (and sequentially) combine robustness and fairness together (e.g. noise filtration followed by either an in-processing or post-processing fairness approach). Why joint learning is important and why is the proposed combination the only way? See also some of my other comments.
 Overall, fairness is an important problem as is robust learning. I do not really see as many "negative societal impacts" of this work.

This paper designs a classifier that is fair and robust. The authors choose sample selection as a means to ensure robustness. Such sample selection strategy eliminates points that have large loss. Fairness constraints are enforced by explicitly adding these constraints during training. 

The overall optimization problem is setup as a bi-level optimization problem where the outer optimization problem selects the hypothesis and the inner optimization problem selects a subset of samples. Additionally, Lagrangian multipliers are introduced for fairness constraints and are updated according to a heuristic. Therefore, in my view, the entire optimization problem looks like a min-max optimization problem ($\min_{\theta, S} \max_{\lambda} \ell(\theta,S) + \lambda \textrm{Fairness}(\theta, S)$) although the authors didn't explicitly mention this formulation.  Originality: There have been several works on designing fair and robust classifiers (1. Ensuring fairness beyond the training data, Neurips-2020, 2. Robust fairness under covariate shift, AAAI-2021). Unfortunately, they are neither mentioned in the related work and nor compared against in the experiment section. 

The authors are mainly concerned about robustness against label flipping in the experimental section. This is a very restricted form of robustness compared to distributional or adversarial robustness. Moreover, if one only cares about label noise, then probably a better approach is to consider better loss functions.

Clarity: I could follow the overall algorithm, and the various choices made in the design. However, the presentation needs to improve in several places. 
1. The authors should be upfront about what kind of robustness they are targeting and why the proposed strategy is the best fit for that objective. 
2. The authors made a choice in directly encoding the fairness constraints during training. This is an in-processing approach and why should we prefer such an approach rather than learning a fair representation of the selected subset $S$ of samples.
3. Please clarify how ILTM is run for figure 2. Do you run a full iterative method or just discard samples based on an already trained model according to the optimization problem defined on page 3 (first equation)?
4. Finally, the authors should discuss why the proposed algorithm converge or why is it supposed to work. From the overall algorithm, it seems that the $\lambda$-variables are updated through gradient ascent and the $(\theta,S)$ parameters are just best responses. There is no reason to believe that last iterate of this procedure converges. However, the average prediction over all the iterates is the right choice and have been used in reduction-based approaches to fair classification.

 Although the design of fair and robust classifier is an interesting problem, there have been already several work in this area. Moreover, the paper has several limitations.
1. The choice of sample selection strategy is not well-motivated.
2. The kind of robustness guarantees provided is weak and there are already several work in designing fair classifier with adversarial and distributional type of robustness.
3. There is no formal guarantee of the proposed algorithm and moreover this need not be the right approach (see my earlier comments).
4. Several related work on fair and robust classifiers are missing and the authors haven't compared the proposed classifier with these existing methods.

This paper proposes an approach for training machine learning models which are simultaneously fair (with respect to conventional group fairness definitions) and robust to a fraction of the training data being corrupted. The idea is to, over the course of training, select a subset of instances to focus on which simultaneously trim off instances with high loss (likely outliers) while also up-weighting selected sensitive groups. Experimental results are shown for two standard datasets.
  Combining fairness and robustness is a worthwhile goal -- the example in Fig 2 is a nice illustration of how naively applying methods aimed at robustness could worsen fairness. A strength of the method proposed in this paper is that it is simple and efficient to implement, and can be applied to almost any ML model. The experimental results also show some promise. 

There are two main weaknesses:

First, the method considered is also somewhat incremental, more or less combining an existing approach for robustness and an existing approach for fairness. It's not clear if there is a big conceptual lesson learned from this.

Second, the experimental results are also somewhat mixed compared to previous work. Compared to the existing FR-Train method, the proposed method seems to sometimes improve on fairness while sometimes worsening accuracy a bit. A couple of suggestions: (a) showing a complete tradeoff curve between accuracy and fairness for both methods, to get a more complete idea of the Pareto frontier and when each method might be more appropriate (b) it could be helpful to provide some context for the magnitude of the relative differences in each measure. E.g., what does it mean for accuracy to change by 0.02 but EO to worsen by 0.03? Is this typically a good tradeoff?  

All together, given that the method is relatively incremental, it would be better to see a cleaner experimental evaluation to suggest that it will have a big impact in practice.  yes

Meta-review: Reviewers broadly appreciated the focus of the paper, and agree that the intersection of fairness and robust is a rich and relatively underexplored space.  Yet, reviewers found the work lacking in scope (e.g., with respect to the specific types of robustness considered and the characterization of the methods/approach) and novelty (e.g., again with respect to robustness, reviewers found that this was a very straightforward application of a known method).  Our private discussion largely followed the public commentary that occurred between Reviewer 1QFb and the authors, so I would encourage the authors to take that sentiment and those comments into account moving forward.  Broadly speaking, all reviewers (and the AC) feel that this work is promising but, at present, underdeveloped.

Decision: accept