Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors publish and analyse the largest annotated 10 German abusive language comment datasets. They also conduct a thorough crowd-based annotation study that complements professional moderators’ decisions. 
strengths: Largest dataset in German Language so far for abusive comment
Experimental evaluation includes traditional and state-of-the-art models
weaknesses: Contents from a single newspaper outlet. Also the commenters have to comment on the partner’s website. This also largely creates an inherent population bias in the dataset. 
labeling schema is applied directly from another paper [36]
Even if extensive quality check is placed, the Krippendorff’s alpha [27] for all of the 177 85,000 comments we find low agreement (α = 0.19). 
correctness: Yes. The dataset construction process is described nicely. The evaluation setup may  be improved further. 
clarity: Yes
relation to prior work: Yes
additional feedback: Nice paper.
review_title: The authors publish and analyse the largest annotated 10 German abusive language comment datasets

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work introduced a German abusive language comment dataset. This dataset includes both professional moderators’ and crowd-based annotations. The author presented characteristics of the dataset and the demographic properties of crowd-study participants. Experiments show that this dataset can serve as training data for abusive language detection models.
strengths: * This dataset can suffice the need for a German news comment dataset. Each comment in the dataset is annotated by both professional moderators and 5 crowd-sourcing workers. Multiple annotations allow for creating more accurate ground truth.

* The authors have elaborate on dataset characteristics and present many valuable findings related to rejection tendency.

* Extensive experiments show that the dataset is a very useful resource for training abusive comment detection models.
weaknesses: This dataset, of course, can be an important corpus for the abusive comment detection domain. However, the contribution is limited to this small domain. The classification task proposed in the paper is relatively standard.
I also have ethical concerns about the annotation collection procedure (See Ethics)

correctness: Yes, the dataset is constructed soundly and the evaluation methods and experiment design are appropriate and performed correctly.
clarity: Yes, the paper is well written.
relation to prior work: Yes, it is clearly discussed how the paper differs from previous contributions.
additional feedback: I would suggest that the authors deleted page 14.
review_title: Important dataset, but the contribution is limited and incremental

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present a novel abusive comment dataset for German, consisting of article's comments of the German newspaper Rheinische Post (RP), labelled by both moderators (Mod) and crowdworkers (Crowd), resulting in a dataset of 85000 annotated comments. Five crowd workers labeled each comment using a labeling schema of 7 classes, with several quality asssurance methods put in place, such as pre-screenings and continuous control by interleaving gold annotations. The paper analyses dataset characteristics such as comment rejection tendency based on the annotator demographics, showing they largely align with literature, but also highlighting differences between Mod and Crowd annotations, the latter rejecting comments more often. The dataset is evaluated in a binary setting (abusive, non-abusive) on several baselines and a more sophisticated line of transformer models, and compared to the most related dataset (DerStandard), indicating that the larger size of the proposed dataset helps to generalize to unseen domains, as a classifier solely trained on this dataset performs much better when tested on DerStandard than vice versa.
strengths: - Well motivated paper and useful dataset, with two separate profiles of classifying offensive comments.
- Dataset construction and analysis appears sound, with sensible labelling schema, annotation strategy, and quality assurance. The demographic overview is highly appreciated and differences in the annotation profiles are insightful.
- Large number of baselines were explored, in three different evaluation settings, and experimental comparison to the DerStandard dataset was appreciated, highlighting the benefits of the proposed one. 
weaknesses: - As the authors pointed out, the IAA is very low with 0.19 and while they explored dataset versions with different agreement thresholds, according to Figure 2, the agreement remains very low. It would have been nice if the authors compared their agreement with relevant literature, putting the numbers better into perspective for this specific task.
- It is not entirely clear to me: the labels assigned by the Mods are also multiclass, but the authors do not compare these with the ones they have chosen for the crowdsource annotators. More insights would be appreciated.
- Further insights on the text form of the comments would be nice: are they colloquial with emojis, similarly to tweets and the likes or use standard German.
- I find 40% of the paper to be for the experiments a bit much, considering it is a dataset paper, I'd rather get more insights on the dataset specifics in the main paper, instead of the appendix. For instance, Figure 4 can be moved to the appendix and the joint task objective can be described more concisely. 
- On that note, I would be curious to know whether the authors have considered using more task-relevant auxiliary objectives, e.g. sentiment classification (of tweets). 
- The authors should motivate better their use of the employed metric: are we really interested in the entire spectrum of false negatives/positives? In many real-life applications, we probably want to minimize the number of abusive comments that are not flagged. 
correctness: Sound, but see metric comment above.
clarity: Very well written and nice to read. 
relation to prior work: NA.
additional feedback: l.135 quality insurance -> quality assurance
review_title: Promising new dataset not without flaws

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors offer a new data set for comment moderation, in German. The data is partially crowd-sourced with the usual drawbacks (annotator reliability). An initial experimental study is conducted with baseline classifiers.
There may be an ethical problem, as the crowdworkers were apparently instructed that their answers are *only* used by the university research project, and now they are made publicly available. Furthermore, it is unclear if the actual comments are all legal, as abusive and problematic comments can, for example, include copyright violations and personally identifiable information.
There is also an evaluation problem (too much about optimizing some evaluation score, not about solving an actual problem), but that part of the paper is not central.
strengths: New data set, not English.
weaknesses: I am not sure if the authors paid enough attention to the reliability of crowd workers.
They certainly followed standard protocols: attention checks and gold standard questions.
Nevertheless it is IMHO a known fact that crowdworkers will have a strong tendency to choose the route of least resistance, i.e. to provide easy answers. This is a big issue in current NLP. For example when generating question-answering data sets, you are likely to get trivial questions such as "Who did x" that take the least effort to formulate.
This needs to be taken into account here. For example, a post can be abusive in multiple ways - the current assignment only allowed for single tagging. It also mixes two questions: (1) in which way the post is abusive, and (2) whether it passes the threshold that it should be withheld by the moderators. A post could be abusive (e.g., Angela Merkel is <insult>) but it may not be necessary to remove it.
Hence (see also the comments below) I am not convinced that the data quality of this data set is good enough to be "suitable for training models to identify abusive language in a binary classification setting" - the models trained on this data perform poorly on the DerStandard data set, i.e., it does not appear to transfer to other data sets. It may be a suitable data set for experimentation with such models, but there may be a high risk of overfitting to particularities of this data set.

One may well argue that this is up to the user of the data set to decide and experimentally verify, though. And in general I remain in favor of accepting this data set paper, if it weren't for the problems discussed in the ethics section.

Page 7, line 248: you use PR and ROC curves, but then give only one AUC value. Below which of the two curves?
Was there any train-test split used in this experiment?

Page 7, line 253. Here you effectively compare AUC scores on two different data sets, RP-Crowd-2 and RP-Crowd-3. But if you change the threshold when comments are considered abusive or not, then it means the data distributions changes. The same holds for RP-Mod - comparing apples and oranges.
More precisely, it is very likely that RP-Crowd-2, RP-Crowd-3 and RP-Mod agree on easy instances, and differ on difficult cases; and changing the threshold can make the data set easier or harder. Because of this issue, it may be insightful to cross-evaluate: how well does the RP-Crowd-2 models perform with RP-Crowd-3 and RP-Mod evaluation.

correctness: Page 8, the P-R-Curves clearly are incorrectly generated. First of all, the curves must not be assumed to begin at (0,1) - in fact, the value at 0 is undefined - as this will lead to overestimation of the performance. The best guess for a starting point (0,y) is the y of the first actual sample. Secondly, the curves must not be linearly interpolated, instead an exponential interpolation is necessary. In particularly the blue lines are affected by this.

For further details on incorrect interpolation on PR-curves, see: Davis, Jesse, and Mark Goadrich. "The relationship between Precision-Recall and ROC curves." Proceedings of the 23rd international conference on Machine learning. 2006.

Also note that AUC is not adjusted for chance. For ROC, a random rankings scores 0.5, hence the difference between 0.78 and 0.67 is after adjustment to chance the difference between 0.56 and 0.34, a quite substantial drop.

Do not just follow a simple number aggregate such as the AUC. It is too coarse, and it does not relate to the problem that you are trying to solve. The task that you are solving is to filter out abusive comments. This means that you will likely need to allow a quite small false positive rate, while you still need a fairly high true positive rate.
The AUC aggregates average over all possible recall respectively FPR values. But high FPR values or low recall values are useless systems.
Consider that we have 10% abusive comments, and we want to falsely flag at most 10% of comments incorrectly. This means that for the ROC curve, only the leftmost part of the curve (up to 10%) is of interest, any performance further to the right is moot. Figure 5a clearly indicates that all the RP-trained classifiers perform little different from random in this range. It is interesting to see that 5b differs very clearly, the DerStandard trained models seem to work okayish for the RP data. Why?
In line 285 you claim that the models trained on RP-Crowd-2 performed best on the RP-Crowd-3 data. But this exactly due to the evaluation problem! If you consider the PR-Curve on the left side of 5b, you can see that the RP-Crowd-3 model performed significantly better up to a recall of 0.25. But these early detections are exactly where such a system would be potentially useful, not the long tail on the right!

In line 287, you argue that "the underlying reason for poor cross-evaluation performance of models trained on DerStandard is ..." - I beg to disagree: the DerStandard trained model transferred to new data much better than the models trained on RP-Crowd, which performed really poor on the DerStandard data. This conclusion is wrong, in my opinion. And while I agree that there is a high importance for additional training data, my concern is that the provided crowd-sourced-data may not have sufficient quality to help much: Training on the new data set does not transfer to DerStandard data, hence the models may have learned particular data-set specific signals (e.g., if it contains "MAGA", then it is likely bad).

DUPLICATES have been assigned to different folds. This causes overestimation of the quality. Sometimes they also have different labels. Probably not a huge amount. Some 500 to 1000 (near-) duplicates I estimate, so 1%-2%. Nevertheless, these pairs should always be assigned the same fold.
clarity: The authors are unclear on some issues in their evaluation procedure, such as which AUC they used, and how this measure relates to the problem to solve.

I think the annotation process by the crowdworkers should have been documented much better in the actual paper, not only the appendix.
- were crowd-workers asked a binary "problematic or not" question, or did they have to choose the label? i.e., the "full" labeling scheme in Table 2, was it used for the crowd workers, or not?
- if annotators used different labels from this table, how was a final labeling reached as there might not be a majority label?
- were any crowd worked labels discarded, because they were identified as unreliable? For example the "outlier" crowd workers that flagged up to 85% of comments as abusive likely should NOT be used. Is this information available for users, or does the published data set only include the labels obtained via pre-defined thresholds?
- it was not clear to me when reading the paper that RP-2 and RP-3 were besides setting the threshold differently, also re-balanced independently.
relation to prior work: Good.
additional feedback: The authors should reconsider the placement of Figure 3 in between of two paragraphs, neither of which relates to Figure 3.

For the format of this submission, I would have expected more detailed documentation of the data labeling, and less focus on the preliminary experiments (which also are not a strength of the paper...)
review_title: Problems in the evaluation part and possible ethical problems

Meta-review: This work constructs the largest annotated 10 German abusive language comment datasets and performs a comprehensive analysis. Although reviewers raise their concerns from different aspects, the authors have made convincing replies. I recommend it is accepted, and authors should continue to refine the draft based on the rebuttal comments.
Decision: Accept