On Demonstration Selection for Improving Language Model Fairness
Authors: Song Wang, Peng Wang, Yushun Dong, Lu Cheng, Yangfeng Ji, Jundong Li


Reviewer paper summaries: The work proposes the use of something akin to stratified sampling with fairness considerations for
picking examples for in-context learning, that is, the examples given in an LLM prompt to improve
its ability to solve the instructed task. Tasks here are classification tasks which are measured in
terms of group bias metrics (demographic groups get similar shares of positive outcomes). The
method expands on prior "Fair ICL" (which I could describe in the same way as above) by making use
of sentence embeddings to find stratified (in terms of label and demographic/sensitive group
membership) examples similar to the one being classified. This requires having a dataset but
instead of storing the entire dataset, the data is clustered via its embedding which then serves as
a distilled form of the dataset for use in the examples lookup.

The paper does some experiments on two datasets/tasks with 3 baselines: no examples baseline, ICL
without any stratification baseline, "Fair ICL" with demographic/fairness stratification baseline.
The results show that the proposed method is better at producing less biased results.

Fairness has emerged as a critical issue due to the recent advancements and widespread application of large language models (LLMs). The authors propose leveraging in-context learning (ICL) techniques to tackle this challenge, introducing Fairness-Aware Demonstration Selection (FADS). Specifically, FADS systematically selects training examples following predefined fairness criteria, including data bias mitigation, model bias mitigation, and final demonstration selection. Comparative analysis on two benchmarks demonstrates that FADS consistently yields fairer outcomes than vanilla ICL and outperforms the established baseline, Fair ICL, in most cases. Additionally, the experiments include a zero-shot evaluation of GPT-3.5 and GPT-4 on fairness. An ablation study and analysis further confirm the effectiveness of the proposed FADS approach.
Summary of strengths: + Addresses an important and topical problem.

+ FADS examples picked by the proposed methods do help with biased classification.

+ Fairly well written and mostly easy to follow work.

1. Given the rapid advancement of LLMs, fairness has emerged as an urgent issue. Research aimed at addressing this challenge, such as this paper, is valuable to the community.
2. The proposed method, built upon ICL, inherits several advantages, such as sidestepping the need for model training. The concept is straightforward and practical to implement. The steps taken to mitigate data and model bias separately appear sound and methodically organized.
3. Experiments on two benchmarks demonstrate the effectiveness of FADS, showcasing its capability to produce fairer outcomes than the baseline ICL methods. An ablation study further confirms the significant contribution of the data/model bias mitigation step to enhancing fairness.
4. A valuable by-product of this research is the zero-shot evaluation of GPT-3.5 and GPT-4 on two benchmarks in terms of fairness, providing useful references for future studies in this area.
Summary of weaknesses: My primary concern is a too narrow framing of the proposed methods relative to similar methods or
alternatives with similar requirements.

- Baseline comparisons do not cover alternatives that share the same requirements as the proposed
  methods. This is most visible in the experimentation with the adult dataset and the structured
  data classification problem derived from it. In such tasks, it is not clear whether an LLM should
  be used at all. The data which the methodology uses to create unbiased examples for ICL might as
  well be used to train a simple classification model with or without debiasing.

  Suggestions:

    1) add to experimentation the accuracy and bias results of classification models with
    and without data debiasing.

    2) argue/discuss that the data requirements of the proposed method are sufficiently smaller or
    sufficiently different that the same data cannot be alternatively used to train a more
    effective or at least comparable model for the task at hand.

- The proposed example construction is in many ways similar to data debiasing and thus could be
  compared to other data debiasing methods as applied to data debiasing or as applied to unbiased
  example construction for ICL.

  Suggestions:

    1) compare the proposed example pickers to data debiasing methods and motivations. That is,
    compare ICL unbiased example construction in terms of its performance as a data debiasing
    method and compare data debiasing methods in terms of their effectiveness on picking unbiased
    examples for ICL. A discussion of how example picking is different/similar to data debiasing
    would be worthwhile here as well depending on the outcomes of the aforementioned
    comparisons/experiments.

    One such method which to some degree the paper has already employed for a
    different purpose is counterfactual data augmentation (Lu et al. 2018. Gender bias in neural
    natural language processing).

Small things:

- This work covers fairness only in non-generative applications of LLM or "decision making tasks"
  which are classification tasks. This is a minor issue with the title.

  Suggestions: adjust the title so it is clearer that the fairness being addressed is
  classification/decision task fairness and not fairness of generative outputs as would be the
  first impression from "Language Model Fairness".

- Discussion of fine-tuning or training on debiasing data mentions the drawbacks such as data and
  resource requirements. What is not mentioned is the computational drawback of including examples
  in LLM prompts. Typical usage charges for models such as those from openai are measured in terms
  of prompt tokens which means including examples will incur additional costs.

  Suggestions: Mention prompt costs in discussion and include in the results the number of prompt
  tokens required as a proxy for computational/actual costs in the various baselines and the
  proposed method.

- Similarly to above, the proposed method has significant overhead over just a classification model
  due to the stratified example sampling.

  Suggestions: Discuss the computational storage overheads of the example picking methods. There is
  a tradeoff involved in the number of clusters stored up to the entire dataset, and these
  overheads which would be nice to discuss and/or demonstrate as well.

1. Technically, this work primarily extends an existing technique (i.e., ICL) for application purposes, which may render the methodological contributions and novelty marginal. As highlighted, data bias, whether in training or demonstration data, is a critical issue for fairness. The efficacy of the proposed FADS method also depends on the quality and diversity of the initial data used for demonstration selection. If this initial dataset lacks representativeness or diversity, the ability of FADS to enhance fairness might also be undermined.
2. The experimental results are not as significant and consistent as claimed in the paper. For instance, from Table 1, it is evident that the simpler Fair ICL baseline not only produces fairer results than the other two baselines but also outperforms the proposed FADS in certain cases. Moreover, it seems that FADS tends to compromise performance (i.e., accuracy) in most comparisons with other baselines. This trade-off between fairness and accuracy is, however, not thoroughly discussed or analyzed.
3. Although an ablation study is conducted with two variants of FADS included, results from their main experiments are absent. While figures are presented effectively, the omission of complete results raises concerns about potential cherry-picking. Also, a specific case or error analysis after the experiments is required to bring more deep findings.
4. While the proposed FADS method shows some effectiveness on two benchmarks, broader validation across diverse datasets or scenarios would be helpful. The superior performance of FADS in toxic classification is attributed to speculation rather than robust analysis. Additionally, varying performance across GPT-3 and GPT-4 raises concerns about the generalizability of FADS across different LLMs and more varied datasets.
Comments, suggestions and typos: - Consider using bias metrics without the absolute value so you can show in which direction the
  bias tends. Is it stereotypical bias or otherwise?

- "counterfactual fairness" in the first paragraph of Section 3 could be better called "individual
  fairness" as that better distinguishes it from group fairness which this paper addresses.

- Another problem with using the Adult dataset is that there may be better instance to instance
  distance functions definable as it is tabular data instead of using embedding distances of the
  text representations. Consider using something like L_2 distance in the numerical representations
  of the Adult dataset instances and compare that to the embedding distance in terms of produced
  clusters and/or final bias metrics.

- With regards to simpler things that could work as well is that demonstration of fairness can be
  produced by flipping the sensitive attributes in instances while keeping the label fixed. Please
  consider a baseline method that presents pairs of examples that differ in sensitive attribute but
  nothing else. This is something already done for another purpose in the paper. There are some
  objections to such procedures in literature. The objection is that changing an attribute might
  produce an instance outside of the data manifold, i.e. it may be an impossible or unrealistic
  instance. In the adult dataset, for example, the relationship status also encodes gender so
  flipping gender without adjusting relationship status will produce "impossible" instances.

- Line 503 states that accuracy has improved due to ICL as compared to vanilla but the results do
  not show this. It is the opposite except in one or 2 cases. Please explain.

- The parameter settings around line 479 are not useful unless explained. If not explained,
  consider moving them to the appendix.

- Please include 0% flipping in the results of Figure 4 .

- Citation on line 734 is malformatted.

1. The paper focuses on comparing FADS against baseline ICL methods. Are there other fairness approaches, as mentioned in the related work, to include for a more comprehensive analysis?
2. The model bias mitigation step relies on "fairer model predictions" for each sub-cluster. How these "fairer model predictions" are defined and obtained is unclear. Does this involve feeding the entire dataset to the model for predictions before FADS?
Overall assessment scores: 2.5

2.5