Venue: NeurIPS 2022 Workshop TSRML

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a technique to generate similar sentence pairs that vary along a sensitive attribute for the purposes of evaluating fairness. Their method relies on LLM's zero shot style transfer capabilities, replacing demographic attributes using hardcoded rules and expaning on them with GPT3. They also validate their method using human studies where they ensure the sentences mean similar things but differ along a sensitive attribute.
strengths: - Important problem, well written, well executed design and study

weaknesses: I don't have any strong weaknesses with the work to raise.

It would be interesting to apply this technique for further problems (NLI, QA, etc.)  in the future. 

The applications for studying fairness are interesting, but I'm not entirely sure the method needs to be just fairness specific. Could this technique reduce any "bias" in the dataset (e.g., any artifact that might exist?)
overall_recommendation: Good paper!
review_title: Import Direction & Well Executed Paper

final_score: rating: 7: Good paper, accept
review_content: summary: Summary.

This paper is dedicated to designing generative intuitive fairness specifications for natural language processing. They point out the gap between human intuition about these perturbations and the formal similarity specifications. The authors leverage unsupervised style transfer and GPT-3's zero-shot capabilities to generate expressive candidate pairs of semantically similar sentences. 
strengths: Pros.

1. I acknowledge the contribution of this paper's specification to the fairness community.

2. Extensive empirical are conducted.

3. The paper is well-written and easy to follow. 
weaknesses: Cons.

1. Why table 2 and the last column in table 3 do not have the error bar?

2. As shown in Table 3, the relabel technique only has marginal benefits, since the error bar is highly overlapped.
overall_recommendation: Accept. Details are referred to above sections.
review_title: A good paper, making sufficient contributions.

Decision: Accept