SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)
Year: 2023

Abstract:
" The paper contains examples which are offensive in nature. nlp datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the nlp community has come to realize that the approach of ‘reconciling’ these different subjective interpretations is inappropriate. Many nlp researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve them–indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus creation for nlp has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing nlp models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on nlp, instead of both nlp and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements–as training with aggregated labels for subjective nlp tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers.

heading: 1 Introduction
text: In recent years, the assumption that natural language (nl) expressions have a single and clearly identifiable interpretation in a given context is increasingly recognized as just a convenient idealization. Virtually every large-scale annotation project in the last twenty years has found widespread evi-
dence of disagreement (Poesio and Artstein, 2005; Versley, 2008; Recasens et al., 2011; Passonneau et al., 2012; Plank et al., 2014b; Martínez Alonso et al., 2016; Dumitrache et al., 2019). Plenty of evidence has also been found of disagreements over the inferences that underlie much of semantic interpretation (Pavlick and Kwiatkowski, 2019). More recently, the increasing focus in nlp on tasks depending on subjective judgments such as sentiment analysis (Kenyon-Dean et al., 2018), humour and sarcasm identification (Simpson et al., 2019) and offensive language detection (Cercas Curry et al., 2021; Leonardelli et al., 2021; Akhtar et al., 2021) led to the realization that in such tasks ‘reconciling’ different subjective interpretations does not make much sense (Basile, 2020; Basile et al., 2021).
Many ai researchers concluded therefore that rather than eliminating disagreements from annotated corpora, we should preserve them–indeed, some researchers have argued that corpora should aim to preserve all interpretations produced by annotators (e.g. Poesio and Artstein, 2005; Aroyo and Welty, 2015; Kenyon-Dean et al., 2018; Pavlick and Kwiatkowski, 2019; Uma et al., 2021b; Davani et al., 2022; Abercrombie et al., 2022; Plank, 2022). As a result, a number of corpora with these characteristics now exist, and more are created every year (Plank et al., 2014a; White et al., 2018; Dumitrache et al., 2019; Poesio et al., 2019; Cercas Curry et al., 2021; Leonardelli et al., 2021; Akhtar et al., 2021; Almanea and Poesio, 2022). Yet this approach to corpus creation is still not widely adopted.
One reason for this is that the usefulness of such resources has been questioned. In response, much recent research has investigated whether corpora of this type, besides being more accurate characterizations of the linguistic reality of language interpretation, are also useful resources for training nlp models, and if so, what is the best way for exploiting disagreements. For example, Beigman Klebanov
2304
and Beigman (2009) used information about disagreements to exclude items on which judgments are unclear (‘hard’ items). In CrowdTruth (Aroyo and Welty, 2015; Dumitrache et al., 2019), information about disagreement is used to weigh the items used for training. Plank et al. (2014a) proposed to use the information about disagreement to supplement gold labels during training. Finally, a number of methods for training directly from data with disagreements without first obtaining an aggregated label, have been proposed (Sheng et al., 2008; Rodrigues and Pereira, 2018; Uma et al., 2020; Fornaciari et al., 2021; Davani et al., 2022). This research in turn led to questions about how such models can be evaluated (Basile et al., 2021; Uma et al., 2021b; Gordon et al., 2021a; Fornaciari et al., 2022). A succinct overview of the literature on how the problem affects data, modeling and evaluation in nlp is given in Plank (2022), and an extensive survey can be found in Uma et al. (2021b).
The considerations above led to the proposal of the first semeval Shared Task on Learning With Disagreement (LeWiDi) (Uma et al., 2021a). The objective was to provide a unified testing framework for learning from disagreements, and evaluating with such datasets. LeWiDi 2021 created a benchmark consisting of 6 existing and widely used corpora containing information about semantic and inference disagreements and classifying images which raised a great deal of interest in the community–the benchmark was downloaded by more than 100 research groups worldwide–but in the end very few groups submitted runs for the evaluation phase, perhaps because the state of the art systems developed for Uma et al. (2021b) proved too hard a baseline, or maybe due to the need for expertise in both nlp and computer vision. Moreover, the prior benchmark only covered one subjective task, humour detection (Simpson et al., 2019), and only one language, English. Since then, there has been an increased interest in learning from disagreements with several new models and datasets published over the past year (e.g. Fornaciari et al., 2021; Davani et al., 2022; Gordon et al., 2021b).
We therefore proposed a second shared task on learning with disagreement, which differs from the first edition in two crucial respects: (i) It focuses entirely on subjective tasks, where training with aggregated labels is obviously inappropriate. The datasets are entirely new, and many not yet publicly available; and we include Arabic as a non-English
language. (ii) For evaluation, we concentrate on soft approaches to evaluation (Uma et al., 2021b). We believe that a shared task thus reformulated is extremely timely, given the current high degree of interest in subjective tasks such as offensive language detection, and in particular given the growing interest in research on the issue of disagreements in such data (Basile et al., 2021; Leonardelli et al., 2021; Akhtar et al., 2021; Davani et al., 2022; Plank, 2022; Uma et al., 2022; Almanea and Poesio, 2022). This belief was supported by the much greater participation in this second edition, with over 130 groups registered, 29 submitting a run and 13 system description papers.

heading: 2 Task organization
text: In order to provide a thorough benchmark, we employed four textual datasets of subjective tasks, all characterized by providing a multiplicity of labels for each instance and by having a size sufficient to train state-of-the-art models. Both hard and soft evaluation metrics were employed (Uma et al., 2021b), but with the priority given to soft metrics. In this section, we briefly introduce the datasets and our evaluation criteria. We also elaborate on the setup of the shared task.

heading: 2.1 Data
text: There are by now many datasets preserving disagreements in subjective tasks. We leveraged this diversity, by covering subjective tasks (i) in a variety of genres including both social media and conversations, (ii) in variety of languages including both English and Arabic (iii) in a variety of tasks (misogyny, hate speech, offensiveness detection) and (vi) in a variety of data collection procedurex(experts, specific demographic groups, Amazon Mechanical Turk (AMT)-crowd).
The characteristics of the selected datasets are summarized in Table 1, while Table 2 shows examples of items from each dataset.
2.1.1 Hate Speech on Brexit (HS-Brexit) This dataset, proposed by Akhtar et al. (2021), consists of 1,120 English tweets collected with keywords related to immigration and Brexit. The dataset was annotated with hate speech (in particular xenophobia and islamophobia), aggressiveness, offensiveness, and stereotype, by six annotators belonging to two distinct groups: a target group of three Muslim immigrants in the UK, and a control
group of three other individuals. All the annotations are binary and the dataset is unbalanced towards the negative class across all four dimensions: between 7% of instances annotated with the positive class for aggressiveness and 18% for offensiveness. An analysis of the disaggregated annotation revealed interesting patterns in this dataset. In particular, many tweets where the target and control group completely disagree with each other contain strongly connotated hashtags such as #illegals and #rapists. Moreover, in all cases of total disagreement between the two groups, the target group indicated the presence of hate and the control group indicated its absence, but never the other way round. This dataset has not been previously distributed and so was novel to participants.
2.1.2 Arabic Misogyny and Sexism (ArMIS)
ArMIS v. 1 (Almanea and Poesio, 2022) is a dataset of Arabic tweets with binary labels created to study the effect on sexism judgments of bias–in particular, on where judges stand on the axis from conservative to liberal. The data was annotated by three people, one self-identifying as a conservative male, one as a moderate female, the last as a liberal female. The annotators labelled the tweets for sexism using the ami guidelines from Anzovino et al. (2018).
The dataset was used in Almanea and Poesio (2022) to compare models for sexism detection from disagreement based on soft-loss training (Uma et al., 2020) with models based on the ‘radical perspectivist’ approach in Akhtar et al. (2021). It was also used in Uma et al. (2022) to analyze the difference between subjective bias, bias due to ambiguity, and bias due to noise.
2.1.3 The ConvAbuse dataset ConvAbuse (Cercas Curry et al., 2021) is a dataset of 4,185 English dialogues conducted between users and two conversational agents. The user utterances have been annotated by experts in gender studies using a hierarchical labelling scheme that includes the following categories: Abuse binary, Abuse severity (1,0,-1,-2,-3; 2); Directedness (explicit, implicit); Target (group, individual–system, individual–3rd party); Type (general, sexist, sexual harassment, homophobic, racist, transphobic, ableist, intellectual). Each example was labelled by at least three annotators, and around 20% of the examples were found to be abusive.
2.1.4 The Multi-Domain Agreement dataset (MD-Agreement)
Leonardelli et al. (2021) created a dataset of around 10,000 English tweets from three domains (BlackLivesMatter, Election2020, Covid-19). Each tweet was annotated for offensiveness by 5 annotators via AMT. Particular focus was put on pre-selecting tweets to be annotated that are likely to lead to disagreement. Indeed, almost 30% of the dataset has then been annotated with a 2 vs 3 annotators disagreement, while almost another 30% of the dataset has an agreement of 1 vs 4 judgments.

heading: 2.2 Evaluation metrics
text: Although the need for preserving disagreements between judges is by now widely recognized, particularly for tasks such as those included in our benchmark, in practice much research on learning from disagreement still employs ‘hard’ measures such as accuracy or F1. But a genuine shift towards a new paradigm in which disagreements are not
ignored requires new, ‘soft’ metrics (Basile et al., 2021; Uma et al., 2021b).
A number of such metrics were tested in Uma et al. (2021b). The simplest metric of this type is to evaluate ambiguity-aware models by treating the probability distribution of labels they produce as a soft label, and comparing that to the full distribution produced by annotators, using, for example, crossentropy. This approach was adopted in, inter alia, Peterson et al. (2019); Uma et al. (2020). So in the first edition of LeWiDi, two evaluations were carried out:
1. a hard evaluation of how well the model predicts gold labels, using micro F1.
2. a soft evaluation of how similar the soft label (label distribution) produced as a response is to the soft label extracted from the raw annotation, measured using cross-entropy (CE).
In this second LeWiDi edition focusing on subjective disagreement, soft evaluation using crossentropy was used as primary evaluation metrics, e.g., in our results tables given that the existence of a ‘truth’ cannot be assumed. We also provided
results with F1, but only as extra information.

heading: 2.3 Task definition
text: The principal goal of our task is to develop methods for capturing agreements/disagreements. Given the diverse nature of the datasets, participants may develop unique methods for each dataset, however they were encouraged to use a single (or similar) crowd learning methodology or framework across all datasets, rather than the best model for a specific dataset.
Each dataset was released as a json file, with harmonized fields across all datasets to facilitate access. Each item of each dataset was characterized by the following fields: item_id, text, task, number of annotations, disaggregated annotations, annotator IDs, language, hard label, soft labels, split, other info. Only the latter field other info varies across dataset, as it contains subfields with information that is specific to each dataset, i.e. the
1
. H@Qº JÖÏ @ Qå áÓ @ Yëð H@Q ¯A ZA éJ ¯ ©¢ ®ÖÏ @ àB ½ JªK. A JÓ A ªË @ Õ æK ¬ñ P Y J«@ ú k@
2
¼PYÓ àñºK à @ É ¯AªË@ ú «@ñË@ àA B@ úÎ«ð è @QÖÏ @ð Ég. QË@ á K. ðQ ®Ë @ áK Qº J K ½J ¯AÓð
Éêk. AÓ@ : áK QÓ B ñê ¯ éÓñÊªÖÏ @ è Yë á« I . J « áÓ AªJ.£ð àAÓ Pð àA¾Ó É¿ ú ¯ QÓ B@ @ YêË
. @Q«@ ð é JÓ @Q .º K AÓ@ ð ú æ A ®Ë @ éK ñ Ë @ ÉJ J.¢ ð ÐC¾Ë á ®K Y ð
annotators’ info and additional annotations columns, as ilustrated in Table 1.
CodaLab was the designated site for hosting semeval-2023 competitions, with participants needing to register in order to obtain data and submit their results through the platform.3
The second edition of LeWiDi had two main phases:
Practice phase. In this phase, participants were encouraged to develop novel approaches for model training using the annotator labels. For each dataset, they were given the training and development data, with the complete set of metadata (disaggregated crowd annotations and the additional information specific to each dataset).
Participants could test the performance of their models on the development set by making predictions on the given development input data and then uploading their submissions to CodaLab for preliminary testing. We permitted up to 999 submissions in this phase. The ‘leader board’ was made public to allow participants not only to see how their models performed, but also to compare the performance of their model to those submitted by other participants.
Evaluation phase. The evaluation phase was the official testing phase of the competition. In this phase, we released test data, without aggregated nor disaggregated labels, but equipped with all the other available metadata. The number of submissions for this phase was limited to five submissions per participant to prevent the participants from fine-tuning their models on the test data. Each submission could include one to four datasets (at least one). The leader board was also kept public in this phase and each participant could decide whether or not to publish their score. For calculating general rankings across datasets (Table 3), we decided if a team failed to submit predictions for all datasets, then for the missing dataset(s) the team was assigned the same position as achieved by the organizer’s baseline (see Section 2.4).
Post-campaign evaluation. As our aim is to make this benchmark available beyond the competition to support researchers developing disagreement-aware models, we included a third, post-evaluation phase to allow lifetime access to the data. Researchers participating in this phase will be able to access
3Our competition can be found at https://codalab.lisn. upsaclay.fr/competitions/6146
the same data as in the evaluation phase and test their models on the test data for the various tasks. In this phase the test labels have been released and the data are now made available also through our webpage4.

heading: 2.4 Baselines
text: We provided a majority class baseline, where the prediction of every instance of a dataset has been labeled with the class that is the most frequent within each dataset. These baselines were deliberately kept simple in order not to deter participation, unlike in the previous edition of the shared task.

heading: 3 Participating systems
text: The LeWiDi 2 edition of the shared task was much more successful than the first one (Uma et al., 2021a). More then 130 different teams subscribed to our competition page and 30 teams participated in the evaluation phase of our task submitting their predictions for at least a dataset. The majority of the teams (21) submitted predictions for all datasets, while one team submitted for three datasets and two teams for two datasets. A few teams (6) adopted a different strategy submitting their predictions for only one dataset.
Among the participants in the evaluation phase, 13 teams submitted a system paper (Cui, 2023; Gajewska, 2023; Grötzinger et al., 2023; GarcíaDíaz et al., 2023; Kohli and Tiwari, 2023; Maity et al., 2023; Rizzi et al., 2023; Shahriar and Solorio, 2023; Sullivan et al., 2023; van der Goot, 2023; Vitsakis et al., 2023; Wan and Badillo-Urquiola, 2023). Moreover, after the competition ended, the participants were asked to fill a short survey to provide key information about their strategies and systems. Thus, in total (between surveys and system papers) we have retrieved information about the work of 18 teams.

heading: 4 Results and discussion

heading: 4.1 Overall cross-entropy results and general considerations
text: The overall results according to our main metric, cross-entropy (CE), are shown in Table 3. We concentrate on the results with this metric in this Section. As each team was allowed to submit up to 5 predictions for each dataset, the shown ranking is based on lowest CE obtained for each dataset.
4Our website: https://le-wi-di.github.io/
Unfortunately, we do not have papers for the 7 best performing systems, and only partial descriptions for three of the top teams (duxy, chengcheng and PALI), so the discussion of the results is partial. Nevertheless, we believe some general conclusions can still be drawn.
First of all, we would like to highlight the great diversity of the proposed solutions, much more than normally seen in semeval text classification tasks such as HatEval, OffensEval. Although most systems relied on a pre-trained large language model (llm), with the exception of MIND and XiaCui, most went beyond simple fine-tuning of such models. This points to the fact that this task is novel and thus cannot be simply addressed with off-theshelf solutions, and that no “standard" solution has emerged yet. Also, the willingness to develop new systems–in fact, in many cases, up to 4 systems for the different datasets–suggests the community is interested in the problem. Almost all participating systems beat our (simple) baseline model although there were clear differences in performance between systems. A variety of approaches to learning from disagreement were also considered, covering nearly all those discussed in the recent survey by Uma et al. (2021b), as well as new ones. Almost all systems trained directly from data containing disagreement, most notably using soft loss training (Uma et al., 2020), including the top 2 systems (duxy and chengcheng), but also Repeated Labelling (Sheng et al., 2008), instance weighting (Cui, 2023), and the annotator-specific multi-task learning approach recently proposed by Davani et al. (2022). The majority of systems used different methods for training with disagreement for different datasets. A number of systems (e.g., University at Buffalo, SafeWebUH) used the ‘radical perspectivist’ approach of training separate models for each annotator (e.g. Akhtar et al., 2021), at least for the tasks with a fixed number of annotators (HS-Brexit and ArMIS). A few systems used aggregated labels, but not many, with many of the papers arguing that this approach is not appropriate for the tasks at hand. One team (MIND) first predicted hard labels trough neural networks and
than re-elaborated the prediction space to map the predicted results to a disagreement value. However, it was not just the method that mattered, but the implementation as well: e.g., both the top performing systems and some of the worst performing systems appear to use soft loss training, and there was a substantial difference in performance between the different systems using the approach of Davani et al. (2022), i.e. iLab and morlikowski.
In terms of architecture, we should mention that although many teams developed different systems for the different datasets, many others took the difference between the datasets as an extra challenge. In particular, 5 systems used multi-task learning for sharing parameters across tasks. One team, MaChAmp, took the challenge to the extreme by training a single model for 11 distinct semeval shared tasks. Mostly variants of bert were employed, perhaps because our tasks are discriminative rather than generative. Another element emerging from the analysis of the results from the participant systems is the role of additional information in the benchmark data. The systems that leveraged either explicit information about the annotators or extra layers of annotation performed better in general.

heading: 4.2 Individual datasets results
text: As already observed for the overall results, also for singular datasets ranking, none of the systems that performed best submitted a paper, but we can still make a few observations. In terms of specific ranking, we can observe a certain degree of variability across datasets, although generally each team ranked similarly across the datasets (average standard deviation is of two positions).
HS-Brexit HS-Brexit is one of the datasets for which explicit metadata on the annotators is provided, i.e., the demographic group as either muslim immigrant or control group. HS-Brexit also has a ‘dense’ annotation, i.e., all annotators annotated every instance. Some systems, e.g., University at Buffalo, reported only being able to work in this setting. SafeWebUH (Shahriar and Solorio, 2023), the system ranked 9th for this dataset, implemented a system that leverages the annotator metadata, showing an improvement when this information is taken into account. Moreover, SafeWebUH found post-aggregation to outperform their alternative soft-label system in terms of cross-entropy score.
The annotator-level information is also lever-
aged by the system ranking 10th on this dataset (University at Buffalo) (Sullivan et al., 2023), in the form of annotator embeddings.
While the main task for this dataset is binary classification on the ‘hate’ label, the provided extra labels ‘offensiveness’ and ‘aggressiveness’ were used by some systems (e.g., IRELMaity et al. 2023), as extra information at the same level as annotator metadata.
ArMIS ArMIS was a challenge as the only nonEnglish dataset, meaning that many groups had to use a different llm for this task (e.g., AraBERT). The most distinctive characteristic of ArMIS from the annotation perspective is that all the data were annotated by three annotators chosen to represent the whole range from conservative to liberal on the political spectrum. For this reason, it made sense to adopt a ‘perspectivist’ approach to this dataset, modelling the individual annotators and then extracting a soft label from these annotations - as done, e.g., by the Buffalo group (University at Buffalo), or Heriot-Watt (iLab) (Vitsakis et al., 2023).
The characteristics of ArMIS also provided motivation for the development of the sinusoidal functions approach by team Lon-eå (QMUL) (Hosseini et al., 2023).
ConvAbuse The best two systems by both metrics (Duxy, chencheng498) used multi-task learning and fine-tuning pretrained models (BERT, DeBERTa). But another team (iLab) used similar methods (fine-tuning MLT) with poorer performance, which they put down to their approach to loss calculation, aimed at preserving all annotators’ perspectives at inference time.
ConvAbuse is the only dataset with dialogue context (agent and user turns). Some report using full dialogues (eevvgg by Gajewska (2023), University at Buffalo by Sullivan et al. (2023)) and others only the target final user turn (iLab by Vitsakis et al. (2023)). Most, including all the best performing systems, do not provide this information.
Some teams discuss not being able to use their primary proposed innovations due to the fact that not all annotators label all items (e.g., SafeWebUH). University at Buffalo, was one such team, and developed a distinct method for this dataset. This consisted of adding annotator embeddings and modelling (supposed) interactions and discussions between annotators with cross multi-head attention layers (Shahriar and Solorio, 2023). They showed that this produced a significant performance boost on both metrics compared to a model without this modeling.
MD-Agreement At the top of the leader board, 3 systems are tied (Duxy, chencheng498 and king001). Unluckily information available for the systems is scarce as they did not submit a full report, although two of them used multi-task learning and fine-tuning pretrained models.
In terms of leveraging individual annotators behavior, the MD-Agreement dataset has been collected via AMT, and consists of >50,000 annotations form a high number of different annotators (> 800) that are sparsely represented and for which the only information available is the (anonymous) ID of the rater. This poses challenges and requires implementation of a different strategy with respect to the other datasets, leading several teams to disregard this information for this dataset. Some interesting approaches have been used though. The eevvgg (Gajewska, 2023) team characterized each annotator with respect to the answers the others gave, modelling reliability of individual raters against the opinion of the majority. Moreover, University at Buffalo (Sullivan et al., 2023) used annotators agreement to calculate what they call ‘polarization’ of a topic (there are three topics present in the dataset), i.e. how much a certain topic is judged uniformly or dividing the annotators. Interestingly, these two teams that leveraged the annotator information, positioned well in the MD-Agreement leader board (6th and 7th), and for both teams this
comprised their best ranking obtained among the four datasets.

heading: 4.3 Selected systems overview
text: In this Section we briefly highlight some interesting ideas introduced by the participant systems.
University of Buffalo (Sullivan et al., 2023) tried three different approaches for the four tasks, based on an analysis of the characteristics of the datasets. We already discussed their approaches for the two datasets in which not all annotators labelled all items, MD-Agreement and ConvAbuse. For HS-Brexit and ArMIS, separate models were trained for each annotator, then combined using ensemble methods.
Lon-eå (Hosseini et al., 2023) tested three approaches to predict the soft label: besides a normal sigmoid function, they also tried a sinusoidal function and a step function taking the number of annotators into account. The intuition was that the possible soft values depend on number of annotators: e.g., in ArMIS, with 3 annotators, the probability of a label can only take the values 0.33, 0.66, and 1. Unfortunately the sinusoidal function approach turned out to not work very well, except for hard evaluation on ArMIS, where the system
obtained first place.
iLab Vitsakis et al. (2023) did not place highly on the leaderboards, but were among the teams that took the most strongly ‘perspectivist’ approaches (Cabitza et al., 2023). Their method focused on preserving the distinction between each annotator throughout optimisation and prediction by fine-tuning a multi-task model in which they predicted each annotator’s labels with a separate classification layer. They argue that this type of approach avoids the potential for minoritised annotator voices to be ignored or subsumed when optimising to aggregated or distributional metrics.

heading: 4.4 Comparison and limitations of F1 and CE
text: Although we believe that a soft form of evaluation is clearly more appropriate for subjective tasks such as in our campaign, we do provide the leader board according to the more canonical hard evaluation F1 metric (Table 8 in the Appendix), for a comparison. Comparing the rankings in Table 8 with those for CE (Table 3) we can see that they are not very dissimilar, but there are some differences. To better understand the origin of these differences, for each of the four datasets, we split the test predictions submitted the 14 best submitted systems into three subsets of data, divided according to three levels of annotators’ agreement (low, mid, high)5 and calculated separate F1. The results in Figure 1 shows a dramatic drop in F1 performance with the decrease of agreement. It is interesting to see how with highly uncertain, or ‘difficult’ items - the items with an agreement between 40% and 60% - the hard label produced by a system is wrong 46% of the times. Moreover, we observed a moderate correlation between F1 and CE scores calculated on high- and mid-agreement subsets, whereas only a weak correlation exists between CE and F1 scores on data with low agreement.
We also can observe how learning from disagreement in training improved performance in terms of CE with respect to F1. For example, the performance of UMUTEAM that did not include disagreement in training and ranked 13 in F1, while dropping 6 positions in CE ranking. Or the performance of eevvgg and University at Buffalo which obtained their best performance in MD-Agreement
5High-agreement: more than 83.3% agreement on the same label, either "0" or "1". Mid-agreement: 66.6% to 83.3% agreement, either on "0" or "1" label. Low-agreement: 33.3% to 66.6% agreement on both labels
dataset, improving their average position with respect to their rankings for the other datasets, likely due to the fact that for this dataset they were the only team using information about annotators. All in all, these observations suggest that the observed differences between F1 and CE team rankings are likely to arise from the best ability of CE to represent data with disagreements and how CE is a more appropriate measure than F1 to include disagreements observations in subjective nl tasks, despite some intrinsic limitations.
One observation emerging from Table 3 is that systems generally obtained a higher CE score with ArMIS and MD-Agreement, and a lower CE score with HS-Brexit and ConvAbuse. As pointed out also by team MIND (Rizzi et al., 2023) in an in-depth discussion about evaluation metrics, this depends on the intrinsic entropy of the agreement distribution of each dataset, which in our case is higher for ArMIS and MD-Agreement. Indeed CE can be as low as the total entropy of the distribution. As a side consequence of this behavior, CE scores are not directly comparable across different datasets. Moreover, the relationship between the real distribution and the predicted one, as measured with CE, depends from the distance between them but, since CE is asymmetrical, it depends as well on the specific values involved, thus reflecting only partially the quality of the predictions.

heading: 5 Conclusions
text: In this shared task we proposed a framework for learning from disagreements between multiple annotators across a variety of datasets and subjective tasks. We evaluated the ability to predict the distribution of different interpretations of the annotators so to include different point of views.
We are delighted that this second LeWiDi shared task attracted the attention of a substantial number of research groups, and that the participants engaged so actively with the issues raised by the datasets, coming up with such creative solutions. We believe that proposing the same task on a multiplicity of datasets challenged participants and promoted constructive and fruitful thinking as it forced visualisation of the problem in its entirety rather then focusing on the solution to a specific dataset.
One hope is that the shared task and the datasets we released will stimulate further research in this area, by the participant groups and others. To promote this, the Codalab page will remain open to submissions after the deadline so that researchers can continue test their models on the datasets. We believe for instance that further thinking is still needed on issues such as the most appropriate form of evaluation for tasks in which human subjects frequently disagree, or the usefulness of modelling individual annotators or groups of annotators.
Limitations
The main limitation of this paper is that we couldn’t provide a full analysis of the results due to the partial or lacking descriptions for some of the submitted systems, and in particular the best performing ones. Nonetheless, we believe some interesting conclusions could still be reached both about the datasets used and the results obtained. Another limitation of this campaign is that only textual datasets were used, which didn’t give us the opportunity to consider the applicability of methods across modalities–a shame, given that this is an area where fruitful interchange has taken place between research in nlp and in computer vision. We hope this limitation will be overcome in future editions, while still encouraging widespread participation.

heading: Acknowledgments
text: Elisa Leonardelli was partially supported by the StandByMe European project (REC-RDAP-GBVAG-2020) on “Stop online violence against women and girls by changing attitudes and behaviour of young people through human rights education” (GA 101005641) and by the StandByMe 2.0 project (CERV-2021-DAPHNE) on “Stop gender-based violence by addressing masculinities and changing behaviour of young people through human rights education” (GA 101049386). Alexandra
Uma and Massimo Poesio were partially supported by the DALI project, ERC Advanced Grant 695662. Massimo Poesio was also partially supported by the ARCIDUCA project, EPSRC grant number EP/W001632/1. Gavin Abercrombie and Verena Rieser were supported by the EPSRC projects ‘Gender Bias in Conversational AI’ (EP/T023767/1) and ‘Equally Safe Online’ (EP/W025493/1), and Verena Rieser was also supported by ‘AISEC: AI Secure and Explainable by Construction’ (EP/T026952/1). Valerio Basile was partially supported by the project "Toxic Language Understanding in Online Communication - BREAKhateDOWN" funded by Compagnia di San Paolo (ex-post 2020). Barbara Plank is partially supported by the DIALECT project, ERC Consolidator Grant 101043235.

heading: A F1 results
text: The general leader board according to F1 scores is shown in Table 8. For each team, the best F1 among submitted predictions are shown (each team could submit up to 5 predictions for dataset). Thus they can differ from the F1 shown in Tables 5, 6, 4 and 7, which are the "relative F1", i.e. relative to the systems with the lowest CE.

heading: B Error Analysis
text: We based the error analyses of soft evaluation on the 14 best performing systems because their errors are the most indicative regarding difficulties in solving our task (but also because those teams submitted predictions for all datasets).
All items present two soft labels (“0” and “1”) which represent the agreement level for the each possible label, expressed as the proportion of annotations in favour of one or the other label. By design, for each item, soft labels “0” and “1” are in
the [0-1] range and sum up to 1. The possible specific ratios between agreement/disagreement that a soft label can assume for each item, depends in first
instance on the number of annotations collected for the item: for example, given an odd number of annotations, soft labels cannot be equal [“0”: 0.5, “1”: 0.5]. Hence, since the 4 datasets employed distinct numbers of annotators, they exhibits not homogeneous discrete distributions of agreement levels. To be able to generalize across levels of agreement between annotators and across datasets, we thus split the range of possible combination of judgments into 5 bins, 6 and each item of all datasets has been accordingly reassigned to one of the 5 bins. For example the bin [1-.8] includes all the items for which 80% to 100% of annotators agreed on label “0”. For simplicity, we refer to each bin using only the probability of label “0” since probability of label “1” is the remaining difference.
Considering the same bins of agreement, we calculated the confusion matrix between true and predicted soft labels shown in Figure 2. The matrix shows how the items with more disagreement (bins comprehending level of agreement between 0.8 to 0.2) are classified less precisely, i.e. the diagonal has lower values for these probabilities which often are classified into adjacent level of disagreement. On the contrary, items with higher agreement are more correctly classified, with 84% for label “0” and 45% for label “1”. Furthermore, soft label tends towards label “0”, as the total percentages under the predicted columns show.
Finally, the test of the MD-Agreement dataset has been recently further annotated for the presumable reason of disagreement, following a two-levels taxonomy of disagreement (Sandri et al., 2023) . For this dataset, it has been shown how only in less than 1% of cases, disagreement is due to sloppiness of the annotators, while around 15% is due to insufficient information contained in the tweet, 20% to ambiguity in text and over 65% is due to subjectivity in judgments. This confirms that, at least for carefully collected annotations, what we observe is genuine disagreement and not only poor annotations’ quality. Moreover, we used this second layer of annotations, to understand whether the best systems partecipating in LeWiDi performed particularly good for specific types of disagreement and if some types of disagreements instead where incorrectly classified by all systems. However, at least for the MD-Agreement, from a qualitative analysis we failed to observe a specific pattern of
6([“0”: 1.0-0.8, “1”: 0.0-0.2], [“0”: 0.8-0.6, “1”: 0.2-0.4], [“0”: 0.6-0.4, “1”: 0.4-0.6], ["0": 0.4-0.2, "1": 0.6-0.8], ["0": 0.2-0.0, "1": 0.8-1.0]
errors’ distribution in relationship to the category of disagreements.

heading: Future Work
text: We hope this limitation will be overcome in future editions, while still encouraging widespread participation