SEFair: Benchmarking Fairness in New Setups, Domains and Languages Using Real Community Data
Authors: Yftah Ziser, Yifu QIU, Shun Shao, Anna Korhonen, Shay B Cohen


Reviewer paper summaries: The contribution provides Stack Exchange Fairness (SEFair), a benchmark dataset to assess extrinsic fairness, gathering user-submitted contents associated with protected attributes, e.g., location.
Summary of strengths: * SEFair allows for evaluating previously neglected dimensions, e.g., investigating multi-class protected attributes and intersectional biases.
* The dataset is not limited to the English language.
Summary of weaknesses: 1. Add details in the Introduction to clarify the findings' scope (from line 077).  
2. Motivate better the relevance and the reliance of the application on Stack Exchange, outlining the aspects of the methodology and findings that are generalizable and applicable to other contexts. 
3. Is adopting usefulness prediction as Y a standard practice also adopted in other datasets gathering posts from similar platforms? The novelty of the approach at the moment is partially hidden. 
4. Unclear if the dataset is built upon existing ones (e.g., line 248): this aspect should be made explicit from the start. Details on the data collection phase are partially missing/omitted.  
5. Clarify exactly the notion/meaning of "domain adaptation" as well as bias/harm in the context of the work. These dimensions are currently unclear and prevent the reader from fully understanding the discussion.  
6. Motivate the deployment of the Bias Removal Methods chosen in 4.1.1. 
7. Explicit the text encoders employed.
Comments, suggestions and typos: 1. Avoid a long list of references without stating what is discussed (e.g., lines 114-118). 
2. How large is large? How low is low? Unclear details in lines 235-236. 
3. Unify the formatting of Xtrain in 4.1.  
4. Consider adding a thorough data statement to the appendix (Bender and Friedman, 2018). 
5. Consider adding some textual comments in Appendices C, D, and E, as well as bolds to the numbers in the tables (also the ones in the main body) to support and guide the reader in interpreting the results.
Overall assessment scores: 2