Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper describes the OCCGEN toolkit, which extracts gender-balanced multilingual parallel data, and evaluates three multilingual models in high-resource and low-resource gender-balanced datasets. The proposed toolkit extracts entities of all the occupations from wiki data and obtains the corresponding gender and text information. Furthermore, it acquires parallel sentences through sentence embedding from the multilingual sentence encoder and parallel sentence alignment following the margin-based criterion. Finally, the dataset balancing algorithm generates the gender-balanced dataset through entity categorisation and gender balancing within occupations. The authors report two use-cases of the OCCGEN toolkit on high-resource languages and low-resource languages. Experiment results show that the multilingual machine translation models perform better on male entity samples. 
strengths: 1. The paper topic is very important.
2. The paper describes the OCCGEN toolkit, which generates multilingual balanced datasets in gender within occupations. 
3. The experiment result shows the multilingual machine translation models perform better on male entity samples, which is intuitive.

weaknesses: ***Main Weakness***  
0. This paper presents the OccGen toolkit that builds multilingual parallel data sets balanced in gender within occupations. However, this track is ***Datasets and Benchmarks***, so why not just release the refined multilingual parallel data sets to the audiences? The current dataset is too tiny, which only consists of nearly 1K sentences . Furthermore, all the experiments and analyses are conducted with the tiny dataset, which can not well support the paper's claim. 

The other weaknesses:
1. The proposed gender balancing algorithm performs gender balancing through data selection, which operates well in binary gender. However, it can not handle the minority genders. For instance, in Figure 2, if we consider the ‘transgender female,’ according to the gender-balance hypothesis, much data from male and female entities could be discarded. 
2. Similarly to the main weakness, the experiment is inconsistent with the motivation. In Line 21, the motivation is to create a balanced dataset for training and evaluation. However, the experiment section fails to show the proposed balance dataset’s effectiveness in reducing the gender bias of the translation model. 

Overall, I would like to see directly released training data.

correctness: Yes.
clarity: Good.
relation to prior work: Yes.
additional feedback: 1. How does OCCGEN help to refine the training data?
2. What's the performance of the translation model trained with the refined data? Will the overall translation quality decrease?
review_title: More Data Please!

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper contributes a toolkit that is capable of extracting multilingual parallel data balanced in gender within occupations from real-world data. Specifically, the pipeline of this toolkit consists of three main stages: data collection, datasets alignment, and dataset balancing, where the balancing methodology is customizable in both gender and occupations. To show the effectiveness of the proposed methodology, the authors create the evaluation benchmarks of machine translation (MT) for 4 high-resource languages (Arabic, English, Russian, Spanish) and one low-resource language (Swahili-English). Besides, it conducts experiments and discusses the translation performance of different languages in terms of gender. 
To conclude, the contributions of this work are:
1. Proposing a methodology that can create multilingual parallel data balanced in gender within occupations.
2. Contributing five English-centric parallel datasets for both high-resource and low-resources languages, which are created by the proposed balancing methodology.
3. Provide a preliminary analysis of three pre-trained MT models regarding the translation behaviors of gender bias. 

strengths: 1. **Significance**: Compared to other work, the released toolkit aims to extract real-world data, ensuring the reliability of data sources. Besides, it is customizable for different gender/occupation options, which is considerable for non-binary gender.
2. **Accessibility**: Code and data are hosted in the open-source platform and can be accessed instantly.
3. **Implications**: The released toolkit can be used to extract additional multilingual parallel data without the above-mentioned bias. The community may directly utilize the released benchmarks to analyze MT behavior in terms of binary gender. With customized settings, it also can help the researcher in the field of MT to create more benchmarks to study the other bias of MT models. 

weaknesses: 1. **Relevance to the broader research community**: This work only evaluates the performance of pre-trained MT models using the released benchmarks. It should also study diverse MT models including vanilla Transformer to extend the scope of experiments.
2. **Contribution**: The contribution of this work is a little bit limited. The proposed pipeline highly relies on the existing methods (collection and alignment). From the perspective of methodology, it only contributes a customizable balancing algorithm. 
correctness: 1. The balancing method is sound, and the released datasets are constructed by the proposed method. 
2. **Models**: The pre-trained MT models used in the experiments are trained on the additional data. The authors should also discuss the influence of pre-training data regarding the bias evaluation. Besides, if we train a Transformer model from scratch using a balanced dataset, will the bias problem be alleviated? Is the proposed benchmarks able to correctly evaluate this “unbiased” model under this circumstance?
3. **Evaluation Metric**: BLEU is a sentence-level metric that evaluates the overlap between reference and the model-generated text. It may not completely reflect the bias problem since the gender/occupation bias may be related to some specific terms (like the examples mentioned in Line 15~17). 
----------
***Update:*** The authors have added the human evaluation results in terms of translation bias (binary gender). 
clarity: 1. Line 25: The abbreviation “MT” is used without definition. It is defined in Line 31.
2. Some references are not cited in a proper way (parenthetical citation and narrative citation). For example, in Line 14, “with one gender Bolukbasi et al. [2016]” should be rewritten as “with one gender [Bolukbasi et al., 2016]”.
3. Line 68: Table 2 is missing.
4. Line 76: Appendix citation is missing.
5. It would be clear if the submission provide a figure to clearly explain the relationship between category, entity, and occupation. 

relation to prior work: The related papers are cited and dicussed.
additional feedback: 1. BLEU score is a general-purpose evaluation metric. It would be better to use a metric like F-score to evaluate the bias in the analysis part, focusing on the translation of special terms related to gender. 
2. According to the submission instructions, there should be additional pages containing the paper checklist (https://neurips.cc/Conferences/2022/CallForDatasetsBenchmarks).
review_title: Great work on building the parallel dataset balanced in gender within occupations.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper suggests OCCGEN for the extraction of multilingual parallel data balanced in gender within occupations. 
strengths: - Occupation is an important attribute across languages that concern gender balance since the output trained with or analyzed by such balance may show unwanted inductive biases that harms e.g., the end users of products
- Authors select high and low resource language families and the representatives to check how the pipeline works and how the result comes out with such languages
weaknesses: - Some questionable approach regarding multilingual alignment (detail in Correctness section)
- It seems that checking only BLEU may not highlight the utility of the proposed method (detail in Prior Work section), and the analysis of the evaluation result is weak; there is little to learn except that the performance of mBART is significantly inferior compared to other candidates
correctness: It seems that the claims are correct overall, and the resulting dataset seems to be constructed in a sound way with an appropriate evaluation method, but some approaches seem sub-optimal. For instance, in multilingual alignment, using pivot language is an intuitive approach that makes the problem much easier, but I think authors need to explain more on using such approach; if the number of languages that are used are not large enough, why is not alignment made in pairwise way? This does not necessarily harm the quality of the paper or other claims, but I think it regards correctness of the claims.
clarity: This paper is well-written and has a nice structure, but some typos and missing contents are observed.
relation to prior work: There seems some missing works in the evaluation using machine translation. Using only BLEU may not shed light to the effectiveness of the proposed extraction scheme. Does the level of BLEU indicates the gender-unbiasedness of the translation result? I think, to check the balancedness of the augmented multilingual dataset, one would have to fine-tune a translation model with the augmented dataset and check the model correctly or unbiasedly infer some occupation-related statements suggested in previous work on translation gender bias (I think the authors may be well aware of them). Since this is just a suggestion and may go beyond the scope of the proposed dataset extraction scheme, so I do not see this as a limitation; however, if authors decided to use BLEU as the only evaluation metric of MT, I suggest the addition of why not using other fairness-related metrics.
additional feedback: Comments

- One of my concerns is that, though authors claim that the proposed extraction scheme is suitable for gender-balanced corpus that regards various aspects of gender, but it is currently a claim and is not shown by the evaluation. I wonder if there is any evaluation methodology that helps the users of the proposed method can see if the corpus is well extracted regarding various gender spectrum in Figure 2.

- Next, to be honest, Discussion section (especially the first paragraph) is difficult to understand due to its conceptual usage of terms and it would be much better if there are some example sentences of improvement regarding some of the languages. Especially, what does the author mean by line 263: 'English and Spanish have a solid language generation'? if it is due to the script-level similarity with the source language (English), doesn't it hide the effectiveness of the proposed evaluation methodology?  Also, is there any authors' suggestion on line 287?

- Last, this is just for a suggestion; I think it may be nice to tell readers how to pronounce the datset; e.g., 'oxygen'?

Addiotionally, I suggest some typos or missing contents here.

- I think citation formats are all with authors' names, which makes it difficult for readers to tell the citation form the text body.
- '. While' to ', while' in line 22
- line 68; where is Table 2?
- Appendix ?? in line 75
review_title: A timely dataset for gender and occupation concerning multilingualism and gender balance

final_score: rating: 3: Clear rejection
review_content: summary and contributions: In this article, researchers present a dataset and toolkit called OccGen, which uses a selection of entities from Wikidata to generate a "gender balanced" dataset that can be used for machine translation. The authors use their sample dataset to demonstrate differences in translation outputs based on binary gender.
strengths: The paper is very clearly written, and the motivation well outlined. The techniques used seem generalizable to other contexts.
weaknesses: The major weakness of this paper is related to their misrepresentations of gender and their misleading claims regarding gender diversity. The authors claim in the abstract that "OccGen can extract datasets that reflect gender diversity (beyond binary) more fairly in society." The authors repeat this in the introduction. However, in the actual article itself, the authors only use binary gender (and incorrectly call these "male and female" instead of man and woman.) The authors go on and state that they are "exempt" from any responsibility about this decision since it is out of scope. 

This is honestly confusing to me because the authors' stated goal of this work is to address harms of gender bias related to allocation and representation. This contribution replicates harms it claims to be combating in its exclusion of non-binary people. This is especially significant as this is a datasets and benchmarks track. Have the authors considered how this (lack of) representation harms non-binary people? The authors made the decisions about the construction of the dataset. I do not see this decision as "out of scope" in any meaningful way. Non-binary people were even included in the data presented in Figure 2 (albeit in much smaller numbers), but the authors chose not to include any of them at all. 

Of course, not every paper can be everything to everyone. But it is clear that the authors are familiar with the relevant literature on gender bias and gender diversity, having cited many articles I might have suggested. The authors had a real opportunity to engage gender bias in a meaningful way but instead chose to ignore it, call it "out of scope" and link out to "wonderful online resources" that they do not engage with. At the very least, the authors need to meaningfully engage with the limitations of their work with regard to use of binary gender over other representations. But, the nature of the work presented, even if it could be generalized and might otherwise have meaningful technical contributions, is grounded in binary gender and gender "balance" so I cannot support it in its current form.

In addition to the harms and ethical implications, a focus on binary gender limits the potential impact such a contribution could have. There could have been an interesting discussion on gender bias in each language, but that seems to have been reduced to a single sentence (279-280). There also could have been a discussion about how the technique could be meaningfully expanded beyond a representation of binary gender, or what kind of dataset requirements would be necessary to improve results for more/all genders. 
correctness: When referring to binary gender, the terms "man/woman" are the correct terms to use, not "male/female." Of course, this does not account for other genders.
clarity: I think there is something missing in the sentence on lines 22-23. Maybe remove the "while"? Otherwise, the paper is clearly written.
relation to prior work: Yes.
additional feedback: Reference to the Appendix is missing (??), line 76.
review_title: Contribution chooses to ignore non-binary people

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper presents a toolkit, OccGen, to collect a multilingual parallel and balanced dataset that contain a similar number of sentences per gender within each occupation, from Wikipedia. The pipeline-style toolkit consists of data collection (including information extraction on entities, occupations, gender and language information, biography scraping on entities and data preprocessing), dataset alignment that aligns collected sentences of different languages into parallel data, and dataset balancing that attempts to include a similar number of sentences for each category. In order to showcase the toolkit, the paper further presents two case studies on high-resource languages (i.e., English, Arabic, Spanish and Russian) and low-resource languages (Swahili-English). Implementation details of the toolkit on the cases are presented. Two datasets are hence collected and experimental results on the two collected datasets are reported with gender-oriented analysis. The major contributions of the paper are hence the presented toolkit for constructing gender-balanced parallel data and two case studies on the two collected datasets. 

strengths: 1. Gender bias is an important issue in machine translation. The paper presents a toolkit to collect datasets for evaluating this issue.

2. The methodology of the toolkit together with implementation details in high/low-resource languages are presented.

3. Two case studies with experiment results are reported.


weaknesses: 1. The sizes of the presented datasets for high/low-resource languages are small, which makes the sufficiency of using the collected data to evaluate gender bias in machine translation skeptical. It would be much better if the paper could discuss the size issue, e.g., which factor has a great impact of the final size of the collected dataset, whether the toolkit can be used to collect sufficient instances for languages that do not have many entities in Wikipedia. 

2. Although details of the toolkit are presented, OccGen seems to be quite sophisticated and manual postedition is required to build the final benchmark dataset, which could make it difficult to adapt the toolkit to other languages.

3. It is not clear how the work presented in the paper is suitable to this track which prioritizes datasets and benchmarks while the paper seems to focus on a toolkit. Intrinsic analysis on the collected datasets and clear definition of benchmark tasks on the collected datasets seem to be missing.

4. The paper states that it attempts to address both evaluation and mitigation issues. However, it is not clear how gender bias is mitigated in machine translation with a small collected balanced dataset.

5. The analysis presented in experiments seems to be language-dependent. It is hence not clear whether the conclusion from this analysis could be extended to other languages. 

6. The analysis on the more/less attention to the source sentence and words in the target sentence seems to be short of empirical evidence. Visualization or deep look into NMT models on gender generation could be used as the support to this conclusion.


correctness: 1. The claim on mitigation with the collected datasets is not clearly discussed.

2. Benchmark tasks and evaluation metrics on the collected datasets are not either clearly defined. Although the paper states that they try to evaluate gender performance with standard evaluation methods, this is not clearly discussed and supported. 

clarity: The paper is not clearly written on some key issues. Please refer to comments before. 

relation to prior work: Although the paper theoretically compare with previous work in the section of related work, it could be better if detailed comparisons to previous datasets on gender bias, e.g., size of dataset, aspect involved in the built dataset, are presented. 
additional feedback: Writing of the paper could be improved, e.g., line 24 & 25.
review_title: The paper mainly presents a toolkit and two case studies while datasets and benchmark tasks, evaluation metrics are not clearly presented and discussed. 

Meta-review: metareview: This paper describes a toolkit for building multilingual datasets that are balanced across preselected gender categories in occupations, drawing from source datasets such as Wikipedia. The toolkit performs data collection, aligns data, and balances data in a customizable way. Case studies are performed on a machine translation task for five languages, and translation performance is discussed in terms of gender identity.

We appreciate the engaged discussion by reviewers and authors for this paper. The topic under consideration is in an important area of study, and it is clear substantial effort has gone into the paper and underlying work. However, given that the goal of the paper is to address gender measurement, it is crucial that questions of non-binary gender identification be correctly addressed. The authors need to be clear and direct about the limitations of their work and the limitations imposed by the use of a dataset with an almost entirely binary selection of genders, rather than simply describing non-binary gender questions as “out of scope.” Additionally, the paper needs to be written to use preferred terminology with respect to gender in all areas.

This paper is being recommended for a *conditional acceptance*. In order for this paper to be accepted, the final camera-ready version of this paper will be checked to ensure that it:

- More appropriately represents the toolkit and dataset by removing inappropriate claims as to the inclusiveness of the dataset.
- Discusses the limitations of the presented work as it relates to the under-representation of non-binary people, and by explicitly engaging in the related cited work. 
- Acknowledges the limitations of the gender taxonomy being inherited from Wikipedia.
- Ensures that all references to gender use the correct terminology.

Strengths:

- The paper is in a very important research area and is clearly written.
- The multilingual machine translation experiments demonstrate both the effectiveness of the toolkit in high- and low-resource languages, and the importance of gender in the task.
- Code and data have been made available for the community.

Weaknesses:

- The paper needs to handle questions of gender in a more inclusive way, lest it perpetuate harms associated with restricting gender to a binary.
- The experiments conducted do not demonstrate reducing the gender bias of the translation model--mitigation is discussed as a motivation but it is not clear that it is actually a major aspect of the work as currently presented.
- The primary novelty of the work is limited to the balancing algorithm provided, as the collection and alignment stages take advantage of existing approaches.
- Evaluation metrics depend heavily on BLEU scores, which may not be sufficiently informative for this kind of work.
confidence: 3: The area chair is somewhat confident
Decision: Accept