FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models
Authors: Dahyun Jung, Seungyoon Lee, Hyeonseok Moon, Chanjun Park, Heuiseok Lim


Reviewer paper summaries: Authors propose a benchmark within Question Answering to assess LLMs w.r.t. their ability to maintain fair behaviors under extreme scenarios modeled as adversarial prompts explicitly designed to elicit biased responses.

This paper studies the robustness of LLMs to jailbreak attacks regarding fairness. It shows that LLMs, when prompted in a typical way, are more vulnerable.
Summary of strengths: * The work positions itself critically and constructively in relation to previous existing NLP fairness datasets, making the novelty of the approach evident. 
* The description of the methodology and the assumptions on which it is built are straightforward to grasp, follow, and understand. 
* The experiments are conducted on a variety of recent LLMs.
* In punctually describing the quantitative results, authors draw general implications and recommendations for the research community.

1. The paper is well-written and easy to follow.

2. The curated dataset seems useful, as a complement to existing robustness benchmarks. The resulting finding is interesting.
Summary of weaknesses: 1. Add an explicit definition of social bias in the context of your contribution. 
2. Clarify (in lines 292-293 and before) how you "select and allocate the most effective method for constructing the dataset", i.e., how the most critical prompt is chosen, according to which criteria (e.g., is it quantitative or qualitative? How it is evaluated?). Specularly, clarify the same aspect for "the single most vulnerable prompt" (line 335).   
3. Clarify the use/choice as "reference" LM of GPT-3.5 in 3.2.2.

1. The paper lacks a more comprehensive and structured related work section. The current one is very vague and misses a few critical aspects, like fairness in LLMs, attacks/jailbreaks in LLMs, etc.

2. Can the paper also test if self-consistency can help LLMs to achieve better robustness in this scenario. In figure, can the experiment be conducted on more shot settings? 

3.  The presentation of tables can be improved.
Comments, suggestions and typos: 1. Explicit the three categories in line 090. 
2. Some references are repeated. 
3. Consider rephrasing the expression "white-box LLMs" because it is at least controversial.  
4. Clarify Language Model Evaluation Harness (line 373).  
5. 4.5 Challenges in Few-Shot Seting > Setting 
6. Add the final . in various captions, e.g., starting from Figure 4 and the tables in the Appendix.

For each first sentence, phrase, or word at the first part of the paragraph, a period should be added to their right, even if they are bold.
Overall assessment scores: 3.5

2.5