On Demonstration Selection for Language Model Fairness in Decision-Making
Authors: Song Wang, Peng Wang, Yushun Dong, Tong Zhou, Lu Cheng, Yangfeng Ji, Jundong Li


Reviewer paper summaries: NOTE: This is an update to a review of an earlier revision of this work in the April cycle here:
  https://openreview.net/forum?id=9IUfu0F50J . I include the original review and comment on updates
  to points as necessary.

The work proposes the use of something akin to stratified sampling with fairness considerations for
picking examples for in-context learning, that is, the examples given in an LLM prompt to improve
its ability to solve the instructed task. Tasks here are classification tasks which are measured in
terms of group bias metrics (demographic groups get similar shares of positive outcomes). The
method expands on prior "Fair ICL" (which I could describe in the same way as above) by making use
of sentence embeddings to find stratified (in terms of label and demographic/sensitive group
membership) examples similar to the one being classified. This requires having a dataset but
instead of storing the entire dataset, the data is clustered via its embedding which then serves as
a distilled form of the dataset for use in the examples lookup.

UPDATE: My original reading did not appreciate the justifications for the elements of the
  stratified sampling used for demonstration selection which are aimed at addressing
  (demonstrations set) data bias and model bias, one each.

The paper does some experiments on two datasets/tasks with 3 baselines: no examples baseline, ICL
without any stratification baseline, "Fair ICL" with demographic/fairness stratification baseline.
The results show that the proposed method is better at producing less biased results.

UPDATE: The new paper includes some experiments on the impact on the number of demonstrations which
  show a strange result that for large enough set, the performance in all metrics gets worse.
  Further, the selection of which results from the earlier revision are presented in the new one
  needs some explanation which I ask about further in this review.
Summary of strengths: + Addresses an important and topical problem.

+ FADS examples picked by the proposed methods do help with biased classification.

+ Fairly well written and mostly easy to follow work.

UPDATES: Still all true. Feels even better written than before but this might be due to being
  already familiar with it from the earlier revision.
Summary of weaknesses: My primary concern is a too narrow framing of the proposed methods relative to similar methods or
alternatives with similar requirements.

UPDATE: This weaknesses listed below have not been addressed or even acknowledged in the revision.
  More updates follow.

- Baseline comparisons do not cover alternatives that share the same requirements as the proposed
  methods. This is most visible in the experimentation with the adult dataset and the structured
  data classification problem derived from it. In such tasks, it is not clear whether an LLM should
  be used at all. The data which the methodology uses to create unbiased examples for ICL might as
  well be used to train a simple classification model with or without debiasing.

  UPDATE: The minimal change would have been to note that the available data alone is insufficient
    for training but sufficient for demonstrations. It is not clear though whether the requirements
    of the proposed methods are on the level that would have made it more preferable to train a
    classifier for the task instead of using the data for trying to debias an LLM with examples.

  Suggestions:

    1) add to experimentation the accuracy and bias results of classification models with
    and without data debiasing.

    2) argue/discuss that the data requirements of the proposed method are sufficiently smaller or
    sufficiently different that the same data cannot be alternatively used to train a more
    effective or at least comparable model for the task at hand.

- The proposed example construction is in many ways similar to data debiasing and thus could be
  compared to other data debiasing methods as applied to data debiasing or as applied to unbiased
  example construction for ICL.

  UPDATE: Not addressed or acknowledged.

  Suggestions:

    1) compare the proposed example pickers to data debiasing methods and motivations. That is,
    compare ICL unbiased example construction in terms of its performance as a data debiasing
    method and compare data debiasing methods in terms of their effectiveness on picking unbiased
    examples for ICL. A discussion of how example picking is different/similar to data debiasing
    would be worthwhile here as well depending on the outcomes of the aforementioned
    comparisons/experiments.

    One such method which to some degree the paper has already employed for a
    different purpose is counterfactual data augmentation (Lu et al. 2018. Gender bias in neural
    natural language processing).

UPDATE: Given the major weaknesses have not been addressed, my assessment has not changed.

Small things:

- This work covers fairness only in non-generative applications of LLM or "decision making tasks"
  which are classification tasks. This is a minor issue with the title.

  UPDATE: The title might have gotten worse with this respect. Perhaps using the word
    "classification" in the first sentence of the abstract instead of "decision" would help.

  Suggestions: adjust the title so it is clearer that the fairness being addressed is
  classification/decision task fairness and not fairness of generative outputs as would be the
  first impression from "Language Model Fairness".

- Discussion of fine-tuning or training on debiasing data mentions the drawbacks such as data and
  resource requirements. What is not mentioned is the computational drawback of including examples
  in LLM prompts. Typical usage charges for models such as those from openai are measured in terms
  of prompt tokens which means including examples will incur additional costs.

  UPDATE: Not addressed.

  Suggestions: Mention prompt costs in discussion and include in the results the number of prompt
  tokens required as a proxy for computational/actual costs in the various baselines and the
  proposed method.

- Similarly to above, the proposed method has significant overhead over just a classification model
  due to the stratified example sampling.

  UPDATE: Also not addressed.

  Suggestions: Discuss the computational storage overheads of the example picking methods. There is
  a tradeoff involved in the number of clusters stored up to the entire dataset, and these
  overheads which would be nice to discuss and/or demonstrate as well.
Comments, suggestions and typos: - Consider using bias metrics without the absolute value so you can show in which direction the
  bias tends. Is it stereotypical bias or otherwise?

  UPDATE: Not considered, not a big deal.

- "counterfactual fairness" in the first paragraph of Section 3 could be better called "individual
  fairness" as that better distinguishes it from group fairness which this paper addresses.

  UPDATE: Suggestion still stands; "individual fairness" is the dual to covered "group fairness",
    and "counterfactual fairness" is a type of "individual fairness".

- Another problem with using the Adult dataset is that there may be better instance to instance
  distance functions definable as it is tabular data instead of using embedding distances of the
  text representations. Consider using something like L_2 distance in the numerical representations
  of the Adult dataset instances and compare that to the embedding distance in terms of produced
  clusters and/or final bias metrics.

  UPDATE: Not considered or compared. This could have been covered by the transfer learning
    scenario that absolutely needs to use a distance metric that is closer to LLM's view of these
    instances but the use of a separate bert encoder independent of the LLM used for classification
    (and "transferring" from) is problematic when arguing for this.

- With regards to simpler things that could work as well is that demonstration of fairness can be
  produced by flipping the sensitive attributes in instances while keeping the label fixed. Please
  consider a baseline method that presents pairs of examples that differ in sensitive attribute but
  nothing else. This is something already done for another purpose in the paper. There are some
  objections to such procedures in literature. The objection is that changing an attribute might
  produce an instance outside of the data manifold, i.e. it may be an impossible or unrealistic
  instance. In the adult dataset, for example, the relationship status also encodes gender so
  flipping gender without adjusting relationship status will produce "impossible" instances.

  UPDATE: Some of this was discussed in openreview comments but given that counterfactual results
    are in the revision, some note about impossible counterfactual needs to be included. I.e.
    changing just the gender in the adult dataset may produce impossible instances as gender is
    also represented in relationship status.

- Line 503 states that accuracy has improved due to ICL as compared to vanilla but the results do
  not show this. It is the opposite except in one or 2 cases. Please explain.

  UPDATE: There are more result issues which I include in the new comments below.

- The parameter settings around line 479 are not useful unless explained. If not explained,
  consider moving them to the appendix.

  UPDATE: Still there but a minor point.

- Please include 0% flipping in the results of Figure 4 .

  UPDATE: Added.

- Citation on line 734 is malformatted.

  UPDATE: Fixed.

UPDATES (new comments/questions):

- 098 bias. Consequently, LLMs may still yield biased
  099 outputs even if unbiased demonstrations are se-
  100 lected as input.

  Couldn't the proposed methodology be described as finding unbiased demonstrations? Do you mean to
  say that FADS does more than just find unbiased demonstrations unlike tools like FairICL?

- Line 351 mentions that the filtering to account for "model bias" queries the LLM. How is this
  query structured? Is it identical to the eventual application of the LLM except without
  demonstrations? Is it possible that how a model views an instance in whatever presentation is
  given here is different to how it views it as a demonstration? A demonstration, after all, comes
  with a label.

- Please highlight/bold top results for Acc in Tables like the other columns are.

- Why does Table 1 not have error ranges while Table 2 does?

- Comparing results of Table 1 in the revision to the originals points out that different
  parameters were used for GPT-4 than for GPT-3.5. It looks like the new table has the old GPT3.5
  16-shot and the old GPT4 8-shot results. Why? I hope it is not to avoid presenting the not so
  good results in the old GPT-4 16-shot rows.

- Old Figure 4 presented the error bars in a slightly better way. It is much harder to read the
  error bars now.

- The results of Table 3 hint that there is something unexpected happening. If more FADS
  demonstrations is not always better, what does that apply to the various aspects of the FADS
  design? Can you continue to see how the trend evolves further and maybe include error bars in
  case this table is just a fluke?

- "Adult involves two types of sensitive attributes: gender and France." ?

- The Limitations section:

  591 ... If
  592 these demonstrations do not adequately represent
  593 the actual query samples in real-world scenarios,
  594 the model may still exhibit biased behavior. ...

  This sounds like a limitation of ICL (i.e. examples must be relevant). Does your methodology make
  the problem worse than just the ICL problem? That is, if you used plain ICL with irrelevant
  demonstrations dataset, you'd be better off than if you used FADS with the same dataset?
Overall assessment scores: 2.5