Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper tackles the problem of abusive language detection in social media with focus on Indic languages, which are traditionally not resource rich in terms of datasets. To that effect, the paper makes the following contributions:
1. Creates an annotated dataset of ~150k comments taken from an Indic social media platform, i.e., ShareChat
2. Presents experiments with several state of the art multilingual models to demonstrate the efficacy of their domain adapted version of XLM-R, i.e., AbuseXLMR.
strengths: The paper has the following strengths:
1. It tackles the very relevant problem of abuse detection on social media, which is of consequence to a lot of stakeholders, e.g., regulators, users, social media companies, etc.
2. The dataset released is of a good size with focus on Indic languages, which traditionally don’t have a lot of good datasets available.
3. The dataset collection and sampling strategy is robust in terms of ensuring that some users don’t dominate the dataset and that the dataset is not simply composed based on keywords. Other datasets in the field of abuse detection have been marred by such issues.
4. The experiments conducted are of a good breadth with few shot and zero shot setups taken into account. Results are reported in a systematic manner.
weaknesses: The paper has the following weaknesses:
1. The main weakness of the paper is that it really doesn’t build upon recent research in the field, along multiple facets in fact. The annotation guidelines are such that they really encourage non-binary labeling; there is a significant amount of subjectivity in what is considered abuse when it comes to beliefs, practices, and religion. Recent research (https://arxiv.org/pdf/2106.05664.pdf) has used best-worst scaling to account for such subjectivity in annotations; the authors have neither cited the paper nor taken learnings from it in creating their dataset.
2. Furthermore, while the authors create a good size dataset from a social media platform, they could have gone further to make the dataset richer in terms of also providing the social graph information of users; there is no discussion of this even though state of the art abusive language detection methods (https://arxiv.org/pdf/1904.04073.pdf) hugely benefit from social graph information, specially when these datasets come from social media platforms.
3. The AbuseXLMR model that the authors release already achieves a ~90% performance on the dataset. Keeping the inter-annotator agreement in view, I wonder what is left to be achieved on the benchmark anyway? The benchmark doesn’t offer a significant challenge. Additionally, the authors mention that misspellings and obfuscation is a problem for the models but they don’t bring into the discussion any past work on character-level composition to tackle the problem (https://arxiv.org/pdf/1809.00378.pdf).
4. ~Authors are releasing a balanced dataset, which they claim is a desirable trait as per them. I do not necessarily see the proper reasoning for this; abusive language is rather sparse in natural settings, so the dataset is far from what would be considered a natural distribution. Why is balancing a good thing here? Some more motivation around it would have been good to have.~ This point was addressed, authors provided results on splits with near-natural distribution.
correctness: I found the dataset sampling strategy to be sound and the experiments to be well-conducted.
clarity: The paper is very well-written and easy to follow.
relation to prior work: See weaknesses section; there is quite a lot of work that authors could have built upon to enhance the quality of the dataset they are releasing.
additional feedback: None
review_title: Abusive language dataset with rather limited utility

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a competitive multilingual dataset for abusive comment detection for Indic language and an effective model for solving multilingual abusive detection. The dataset has a significantly larger scale than previous datasets. The annotation of the dataset is convincible as experts are claimed to be involved in the work and the measurement of annotator agreement proves the consistency of annotation. The proposed approach has novel parts-pretrained over social media data. The authors conduct extensive experiments over the MACD dataset and evaluate the AbuseXLMR model.
strengths: 1. This paper provides a large-scale and well-annotated multilingual dataset.
2. The proposed AbuseXLMR performance is better against baselines.
3. This paper discusses all-side experiment settings.
weaknesses: 1. The relationship between diversity and user distribution is not clear. 
2. Age distribution of annotators is concentrated in young people while abusive language is likely to be different between different age groups.
3. The AbuseXLMR is pretrained with social media data while it is pretrained with a larger scale of data. I hope the authors may discuss the effects of the scale of pretrained data and different domains of pretrained data.
4. In the error analysis section, is the XLM-R pretrained with the social media data. If not, why not discuss the proposed AbuseXLMR.
correctness: The submission is highly likely to be constructed in a sound way. As experts are claimed to provide the annotation and the analysis of dataset is comprehensive and convincible.
clarity: This paper is well-written, clearly structured and overall a pleasure to read.
relation to prior work: This paper discusses previous work well and provides statistical data of previous work.
additional feedback: None.
review_title: MACD: Multilingual Abusive Comment Detection at Scale for Indic Languages

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The work makes the following contributions:

- A new large dataset in 5 Indic languages for abusive comment classification
- A new model called AbuseXLMR pretrained (continually on XMLR) on 15 Indic languages on private dataset
- Extensive benchmarking and baseline (incl. SOTA) results for the new MACD dataset


strengths: Interesting paper with thorough analysis across various axes.

Some strengths:
- Significant amount of data open-sourced for 5 important Indic languages for the abuse detection task
- Releases a good model (Abuse-XLMR) pretrained on large amount of real social media
- Interesting analyses across different dimensions like few-shot, zero-shot, cross-dataset, etc.

The paper was overall very well-written and easy to follow. Enjoyed reading it in detail.
weaknesses: I see the following couple of major weakness:

- Literature Survey for similar datasets is not well done. There seem to be atleast double the existing datasets mentioned in this paper.
- Choice of XLMR over other pretrained models is not very clear (could probably be addressed with more explanation)

There are a few other questions / comments, which I have put in the "Clarity" section.
correctness: Mostly, all the experiments are well thought-through and does a pretty good job of benchmarking. (although there are some concerns mentioned in the below section)

One minor comment:  
Please follow proper ISO 639-1 language code wherever applicable. This is the proper international standard.
For example, Kannada is "kn", not "Ka".
clarity: Overall, the paper was clear enough. I have put some of my questions, concerns, comments as well as things to clarify below:

1. L48: What is "data source limit"? (and how is it related to not enabling modeling by combining all open datasets?)

2. L56: For the sake on non-Indian audience, would be helpful to explain in detail why is there is only 1 language from the Indo-Aryan family and 4 from Dravidian, when Indo-Aryan is the second-most spoken-language family in the world (after European).

3. Fig. 2.d. shows 6 categories of abuse. Will that also be released? If not, please clearly mention that the dataset will only have pairs of comments and 0/1 to denote abuse/safe.

4. L173: Why not 80:10:10 ? (especially since we are working with low resource languages where more training is essential). 10% of 150k is 15k, which is already quite a significant number for test set (around 3k per lang). Would be helpful to explain why 60:20:20 was preferred or if possible, release an alternate standard split with 80:10:10. Although it is understandable that many researchers just use both training & val sets for training for practical purposes, still I believe 80:10:10 would be helpful.

5. L178: Why not AbuseMuRIL or Abuse-mBERT or AbuseIndicBERT ? Please mention what was the decision behind the choice of XLM-R to continue pretraining. Although I see in table 3 that XLM-R performs better than MuRIL in 3 langs, in the other 2 langs MuRIL is better, but footnote 6 says footnote requires paired corpora (hence difficult to pretrain). If this is correct, please explicitly mention so.

6. L183 (footnote 5): Since it was already mentioned that assuming language based on user profile is not reliable, why not use a language identification model or atleast determine the language based on the dominant unicode character ranges used whenever applicable? (like it was mentioned earlier). For example, for langs like Hindi, Nepali, Marathi, etc. since they all use Devanagari, a character set based match will not imply the language. But in other langs with dedicated scripts (like Dravidian languages, Gujarati, Panjabi, Odia, etc.), this could work very good. Would be great if you can share why this was not considered.

7. L198 (ref to table 3): L102 mentions a model called IndicBERT. Why was that not evaluated?

8. L207: Interesting study! Some language-wise interpretability analysis could make it more interesting. Like for example, why are different source languages helpful for different target languages in different models? (That is, for example, why does Kannada provide different best results in the 3 different models?)

9. L226: Do you mean you combine the train and val sets, and use that for training the joint model? If so, why?

10. L234: Since the continued pretraining was multilingual, why not also perform a multilingual fine-tuning experiment to see if it is better than the __joint__ results? Basically, why not try pr-mono+ft-joint?

11. Appendix Table 10: Interesting to see that there is not even 4% code-mixing. Was the comments in the dataset picked somehow such that this is kept as low as possible? Because predominantly, South Asian people on social media use roman text to write comments in their native language. Would be helpful to provide more info about this.

relation to prior work: When searching for related abuse detection datasets for Indic langs, came across this previous work by the same organization:
https://www.kaggle.com/competitions/iiitd-abuse-detection-challenge/overview

Why is this not mentioned anywhere? How is this work different from that?
Does it have any overlap? Please explain in detail, this is quite essential.
Especially because it seems to have more than double the no. of languages covered in this work.

---

Also, came across another dataset which covers 3 of the 5 Indic langs evaluated in this work:
https://arxiv.org/abs/2106.09460

Why is this not mentioned at all? It seems to be of significant size (60k samples).
Please cover this as well.
additional feedback: Very good paper overall, and valuable contribution to the Indic community ! ~But I have rated it 6 due to quite a many questions for which the answer is unclear/unknown.
I would be glad to increase my rating significantly if all (or atleast most) points mentioned in the review are addressed / explained in the paper.~
review_title: Interesting dataset for Abuse Binary Classification task for 5 Indic languages

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper releases a large-scale and multilingual Indic language dataset called Multilingual Abusive Comment Detection dataset (MACD) curated from a popular social media for the research community of abusive comment detection. Compared to the existing dataset, MACD is large-scale (150k textual comments) and multilingual (5 Indic languages). MACD is one of the largest abusive speech datasets for Indic languages. Further, the authors propose AbuseXLR for abusive comment detection. The experiments show the effectiveness of the AbuseXLR.
strengths: - One of the largest abusive speech datasets for Indic languages, filling in the gap in the community of abusive comment detection.
- The AbuseXLR is a powerful social media dataset.
- The experiments are comprehensive and valid.
- Error analysis is interesting and has interesting insights

weaknesses: In the joint training experiment, the authors claim “the joint training (joint) improves the performance for all the languages over the monolingual (mono) training”. However, XLM-R (joint) does not outperform XLM-R (mono) in Hindi (Acc and F1) and Telugu (Acc). This should be justified.
correctness: The claims in the paper are generally adequate
clarity: This paper is overall clearly written
relation to prior work: Yes, the differences are clearly stated and justified
additional feedback: Please see strengths and weaknesses
review_title: This paper releases a large-scale and multilingual Indic language dataset

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The first half of the paper presents a collection of 150k social media posts in 5 indic languages annotated by experts based on their toxicity. The second half of the paper presents the results for binary classification using a pre-trained XLM-R model on abusive content detection.
strengths: The paper presents 75k social media comments annotated with labels of abusive content classes (plus 75k non-abusive comments).
This is the largest dataset for indic languages for toxic comments so far.
weaknesses: The non-abusive comments were sampled randomly. This has two problems: 1. It is unclear whether you checked these comments manually to guarantee that they are not abusive. 2. Difficult or ambiguous cases are most likely not part of this set.

Given good approaches for transfer learning exist, as well as high quality machine translation, it is unclear, whether we really need a lot of annotated data in all languages for all tasks to perform machine learning.

Half of the paper is about binary abusive language classification. The only connection to the dataset is that the test split was used for evaluation. It is unclear, what the contribution of this second half is. In particular, it is very strange that no multi-class classification setup was chosen given the multi-class labels of the presented dataset.

The AbuseXMLR method is just an XML-R model pre-trained on different data. This is not really a research contribution...


correctness: The non-abusive comments were sampled randomly. This has two problems: 1. It is unclear whether you checked these comments manually to guarantee that they are not abusive. 2. Difficult or ambiguous cases are most likely not part of this set.

The dataset splits should have been done chronologically - not random. Language and topics evolve and a time-based splitting would be more realistic for simulating an in-use system.
clarity: The first half of the paper is well written, but many crucial deals are missing in the paper (they are in the supp. materials, though).
relation to prior work: Authors mention that different formats/schemes could be problematic when joining multiple datasets.
There exists work, which does exactly this:

Julian Risch, Philipp Schmidt, and Ralf Krestel. 2021. Data Integration for Toxic Comment Classification: Making More Than 40 Datasets Easily Accessible in One Unified Format. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 157–163, Online. Association for Computational Linguistics.

In general, there is a lot of work on hate speech/toxicity/abuse/etc. datasets, especially in non-indic languages. Therefore, transfer learning approaches or approaches using machine translation should have been mentioned and it should have been demonstrated that they are significantly worse compared to a native indic annotated dataset.
additional feedback: There are many arxiv references. The authors should cite peer-reviewed versions instead, e.g., [18] was published at ACL.
review_title: Improved after rebuttal

Meta-review: metareview: This paper creates MACD, a large-scale (150K), human-annotated, multilingual (5 languages) dataset for abusive language detection in social media. The dataset has a focus on Indic languages, which are low-resource for this task. The paper presents in detail the sampling and annotation process. In addition, the paper presents AbuseXLM-R, a domain adapted model that achieves state of the art. The paper is well written and contains detailed experimentation. Although the paper uses binary labeling, it includes several annotators, to attenuate subjective bias and disagreement. The size and diversity of the dataset make it a valuable contribution to the community.

*pros*

* the paper takes the challenging problem of online abuse in social media, a very relevant topic.
* the dataset size is large and focused on low-resource languages.
* the dataset has been sampled with care, to ensure its representativity
* the dataset is carefully anonymized to remove PII


*cons*
* the dataset is presented in a "balanced" distribution, which is not representative of the skewness and low-prevalence of real-life abusive content.
* the dataset focuses on binary labels whereas recent research suggests that continuous scales are more resilient to bias.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept