A Benchmark Method for Assessing Bias and Fairness in Large Language Models
Authors: Audrey Emmely Rodrigues Vasconcelos, Baldoino Fonseca dos Santos Neto, Tiago Vieira, Davy de Medeiros Baia, Márcio de Medeiros Ribeiro, Nathalia Nascimento, Paulo Alencar


Reviewer paper summaries: The authors conduct a bias evaluation study leveraging two fairness datasets, CrowS-PairS and StereoSet, adopting three well-established fairness measures.

This paper proposes a benchmark for assessing fairness of large language models. The topic is interesting. However, the paper has evident weakness in terms of writing, presentation, and technical contribution.
Summary of strengths: * The paper is self-contained and details the necessary background knowledge to comprehend the proposal: the resources adopted, i.e., datasets and metrics, are carefully described. 
* The choice of the datasets and models is well-motivated.

The studied topic is interesting
Summary of weaknesses: 1. The novelty is hidden. The description of the contributions in the Introduction is vague. Authors are applying existing datasets and leveraging existing metrics, basically conducting a simple application, not a novel comprehensive framework. Fig. 2 reports just a standard ML/NLP pipeline. Where's the novelty? The paper needs a different, more fitted narrative to the actual contribution.   
2. There's currently an overlap between the study design's presentation and the framework's instantiation, e.g., the LMs tested (lines 157-158). Please divide the two aspects to enhance the generalizability of the approach and the overall clarity and structure of the paper. Additionally, the Section Study Design presents Related Work.
3. Unclear line 193: does your work build upon Kaneko and Bollegala 2022? Moreover, the Study Design section is not the right section to report implementation and cod-ish details. 
4. Restructure Section 3.2 by adding formulas to make the presentation of the metrics more precise and concise. Cite all the metrics. 
5. Authors claim multiple times throughout the paper that the focus is on LLMs while using just BERT, RoBERTa, and DeBERTa, which is inconsistent with current, more recent available LLMs. Also, the criticism of universal/one-size-fits-all datasets is not discussed enough; finally, it is stated that the datasets adopted are two (defining the contribution as a comprehensive benchmark needs a rephrasing; see point 1 of weaknesses).

1. The writing is very vague. After reading abstract and introduction, I still don't understand what is the limitation in current methods/models/benchmarks, and what is the unique contribution of this paper. For example, in the abstract, the paper claims to "introduce a comprehensive benchmark and will advance SOTA fairness assessment", but specifically what is added, or introduced to make the benchmark comprehensive? In the final paragraph of the introduction, the paper claim to "explore several relationship in assessment", how? Many contents in the paper are redundant, like the last sentence of the abstract and the first paragraph in the intro. They are too detailed. 

2. The technical depth is insufficient. The paper only run three small LLM, which are BERT-based. Can the author also consider the more recent generative LLMs, like llama, GPT that are mentioned in the introduction? There is no novel dataset provided, no novel metric provided, all the findings are already known and not interesting.

3. The figure are not professional. Some are even redundant, e.g., fig1, fig2.

4. The presentation of tables are arbitrary.
Comments, suggestions and typos: 1. Lines 060-062 and in other sentences throughout the whole paper: use \citet to display the name of the authors directly with the reference, avoiding repetitions. 
2. Clarify the interpretation of the expression "contextual nature of bias" and your definition of bias and practical harms arising from LMs in your context of investigation.
3. The recommender systems mentioned are confusing in the Related Work: the paper does not focus on that specific application. Therefore, the references are not well-framed/motivated.  
4. Introduce the "unmasked sentence sub-task" mentioned for the first time in line 148.     
5. Remove the Literature Review Papers as the first stage of the flowchart in Fig. 1 (and in general in the text). It's just the first stage of every academic work, and since the contribution is not a review, it is really odd to have it in the workflow.    
6. How does paragraph 180-188 fit since it discusses fairness in ML and not NLP? Is it needed?   
7. Consider moving 4.1 in the Appendix: the details are related to tiny technical aspects of the implementation that are important to document but not interesting enough for the main body. 
8. Avoid cod-ish details and terms such log_softmax in line 533. 
9. Add some bolds to the tables to help the reader identify the best results. 
10. The template style is old: adopt the most recent ACL one.

The writing changed between "LLM" and "large language models" many times.
Overall assessment scores: 1

1