MAD-TSC: A Multilingual Aligned News Dataset for Target-dependent Sentiment Classification
Year: 2023

Abstract:
Target-dependent sentiment classification (TSC) enables a fine-grained automatic analysis of sentiments expressed in texts. Sentiment expression varies depending on the domain, and it is necessary to create domain-specific datasets. While socially important, TSC in the news domain remains relatively understudied. We introduce MADTSC, the first multilingual aligned dataset designed for TSC in news. MAD-TSC differs substantially from existing resources. First, it includes aligned examples in eight languages to facilitate a comparison of performance for individual languages, and a direct comparison of human and machine translation. Second, the dataset is sampled from a diversified parallel news corpus, and is diversified in terms of news sources and geographic spread of entities. Finally, MAD-TSC is more challenging than existing datasets because its samples are more complex. We exemplify the use of MAD-TSC with comprehensive monolingual and multilingual experiments. The latter shows that machine translations can successfully replace manual ones, and that performance for all included languages can match that of English by automatically translating test examples.

heading: 1 Introduction
text: Text analysis needs to address both objective aspects, such as topic extraction, and subjective aspects, such as sentiment and opinion classification. In spite of recent progress brought by the introduction of large language models (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019), sentiment classification remains a challenging task. Expression of sentiments varies according to the data sources, languages and domain of the texts. These challenges are particularly important in targetdependent sentiment classification (TSC), which focuses on determining the sentiment expressed toward a given entity in a given context. The bulk of
TSC-related research efforts are focused on major languages. This focus is an effect of the availability of generic and of task-specific resources in these languages (Brauwers and Frasincar, 2022; Nazir et al., 2020). A majority of datasets are monolingual and, when they are multilingual (Balahur and Turchi, 2014; Barriere and Balahur, 2020; Cortis and Davis, 2021; Pontiki et al., 2016; Severyn et al., 2016), the examples are not aligned across languages. Equally, a majority of existing datasets and methods are devised for texts such as tweets, reviews or comments (Nakov et al., 2016; Pontiki et al., 2016; Severyn et al., 2016) in which sentiment is most often expressed explicitly. Somewhat surprisingly, less attention is given to TSC in news, despite the usefulness of an automatic analysis of their content for the understanding of societally impactful processes such as disinformation or polarization (Hamborg and Donnay, 2021).
Our main contribution is the introduction of MAD-TSC, the first large multilingual aligned dataset for TSC in news articles. It includes 5,110 annotated entity mentions from 4,714 unique sentences. Each sentence has professionally-translated and aligned versions in eight languages (English, Spanish, German, Italian, French, Portuguese, Dutch, and Romanian). Sentences originate from 286 news sources published in over 30 countries. These characteristics differentiate the proposed dataset from existing resources, and in particular from NewsMTSC (Hamborg and Donnay, 2021), a monolingual dataset focused on American politics which is the closest to MAD-TSC. We first present the dataset creation process, and provide a qualitative analysis of its content. Then, we propose a thorough evaluation of state-of-the-art TSC methods on MAD-TSC in monolingual and multilingual settings, with particular focus on the usability of machine translation in TSC. The main findings are the following: • The proposed dataset is more challenging since
8286
it includes more complex examples compared to existing resources, as detailed in Subsection 3.5. • Performance for individual languages varies due to the fact that the quality of available pretrained language models is itself variable, with the best scores being obtained for English. • Results with machine translation of training sets from English toward target languages and with manual translations are on par. • The performance level for other languages matches that of English by translating test examples to English in order to take advantage of the pretrained language models available in this language. The same is true when both the training and test sets are automatically translated to English. The last two findings are particularly interesting since they show that if a domain-specific TSC dataset is available, it can be effectively used for multiple languages. Overall, the introduction of MAD-TSC will facilitate progress in multilingual TSC. The dataset and the full evaluation protocol are available online1 to encourage further research, and to ensure reproducibility.

heading: 2 Related work
text: Target-dependent sentiment classification (also named aspect-based sentiment analysis (Nazir et al., 2020) or classification (Brauwers and Frasincar, 2022)) is a complex task due to the numerous factors which influence the way sentiments are expressed in texts (Brauwers and Frasincar, 2022), such as the language, the domain or the personal biases of the author/reader.
TSC is often evaluated on short texts such as tweets (Nakov et al., 2016), reviews (Pontiki et al., 2016) or comments (Severyn et al., 2016). A common characteristic of these resources is that sentiment is often expressed in an explicit way. News are more challenging texts because sentiments are expressed implicitly or indirectly (Hamborg and Donnay, 2021), often include multiple targets in a single sentence (Brauwers and Frasincar, 2022), and both negative and positive arguments about a target entity are combined due to the fact that journalists are supposed to be objective (Balahur et al., 2010; Hamborg et al., 2019; Liu, 2010).
Multilinguality is important in order to be able to analyze texts in different languages. Multilingual datasets are proposed for tweets (Lam-
1https://github.com/EvanDufraisse/MAD_TSC
pert and Lampert, 2021; Vilares et al., 2017), reviews (Jiménez Zafra et al., 2015; Pontiki et al., 2016) and institutional texts (Cortis and Davis, 2021). While interesting, these datasets differ from MAD-TSC because they do not focus on news. Equally important, they are only aligned at a domain level, but not at an example level. This limits their utility in terms of comparative evaluation in monolingual and multilingual settings. Multilingual approaches were also explored for news representation. For instance, bilingual word embeddings were used to compensate data scarcity in under-resourced languages (Akhtar et al., 2018) or to transfer models between source and target languages in zero-shot settings (Jebbara and Cimiano, 2019).
Classical TSC methods rely on engineered features based on lexicons and syntactic analysis (Biber and Finegan, 1989; Baccianella et al., 2010; Jiang et al., 2011; Kiritchenko et al., 2014; Vo and Zhang, 2015). Strong progress in TSC was made possible by the introduction of deep language models, such as BERT (Devlin et al., 2019; Zeng et al., 2019). Improvements are obtained when pretraining includes a larger proportion of news (Hamborg and Donnay, 2021) – this is the case for English RoBERTa (Liu et al., 2019) or XLNET (Yang et al., 2019) – or when including an intermediate tuning on domain-related data (Du et al., 2020). Naturally, further improvements are obtained by introducing TSC-specific architectures (Brauwers and Frasincar, 2022; Nazir et al., 2020; Zhou et al., 2019). We follow this trend and use pretrained language models in our experiments.
To our knowledge, there are only four datasets dedicated to TSC in news. The first one (Balahur et al., 2010) has 1,592 examples, and includes only quotes from political news. Quotes are interesting because they include a lot of sentiment expressions, but they are also easier since sentiment is often expressed explicitly (Hamborg and Donnay, 2021). The second one (Steinberger et al., 2017) has 1,274 examples. The size of these datasets makes them difficult to use with deep-learningbased TSC methods. The third dataset (Hamborg et al., 2019) includes 3,002 examples. However, as noted later by its authors (Hamborg and Donnay, 2021), the dataset is imbalanced and its sentiment expressions are predominantly explicit. The dataset which is closest to ours is NewsMTSC (Hamborg and Donnay, 2021). Their common characteris-
tics include: a focus on political news, an identical definition of the task, and a similar annotation process. Importantly, we follow Hamborg and Donnay (2021) and instruct annotators to think from the author’s perspective in a holistic way. They are instructed to consider the “what” of the sentence (events, facts) but also the “how” (choice of words, author’s attitude). This choice contrasts with previous works (Balahur et al., 2010; Steinberger et al., 2017), which distinguish author- and reader-levels in TSC, and is important in order to minimize the influence of personal biases. The main differences between MAD-TSC and NewsMTSC relate to multilingualism, complexity of examples, political topics and geography of examples, while maintaining the same order of magnitude in the number of samples, with 5,110 for MAD-TSC , and 11,361 for NewsMTSC. These differences are detailed in Subsection 3.5, and they make MAD-TSC appropriate for a thorough evaluation of TSC in multilingual settings.

heading: 3 Dataset Construction
text: We build on previous works devoted to the creation of sentiment classification datasets (Nakov et al., 2016; Pontiki et al., 2016), and particularly of TSC ones (Balahur et al., 2010; Hamborg and Donnay, 2021; Steinberger et al., 2017). We describe the main steps of the dataset creation process and analyze the resulting dataset.

heading: 3.1 Data Sources
text: Our objective is to create a dataset which: (1) is multilingual and aligned at a sentence level across languages to enable a comprehensive evaluation of TSC, (2) includes content from a large number of high-quality journalistic sources, which offer a diversified view of the included topics, (3) covers societally-impactful topics in different countries. Voxeurop2 is a multilingual news website which aims to offer interesting and high-quality news to European audiences. The project inherits from Presseurop, which was active from 2009 to 2013, and whose objective was to make Europe-related news from over 200 sources available. Voxeurop and Presseurop articles are available in up to ten languages. The content is translated by professional translators, thus ensuring high-quality texts in all available languages. The content is published using a Creative Commons BY-NC-ND, an open
2https://voxeurop.eu
license which facilitates its redistribution and reuse for non-commercial purposes, which will be also used to distribute MAD-TSC. We have collected 7,370 news articles which have translations in all eight languages, amounting to a total of 122,263 sentences in English and comparable numbers in other languages. A wide majority of the examples are related to politics, with the others pertaining to business, culture and society. Most of the entities mentioned in articles refer to prominent public figures from different European countries at the time of publication (2009–2013). Well-represented political sub-topics include: the economic crisis which started in 2008, European Union evolution process, election-related news, political crisis at a national level, and corruption scandals in different countries.

heading: 3.2 Sample Selection
text: Named entity detection was performed using the Flair model for English (Akbik et al., 2018), which led to an initial pool of 30,303 sentences with at least one mention of a person. We combine entity linking with Blink (Wu et al., 2020) and coreference resolution with neuralcoref3 to obtain reliable entity counts in articles. Entities which are mentioned a single time in an article are not kept for annotation because they are not considered in focus. This filtering led to 19,223 candidate sentences. The alignment of sentences for the eight languages is inspired by lingtrain4. It is based on sentence embeddings from a multilingual-sentence BERT model (Yang et al., 2020), with English as source and the other languages as targets. Two matching criteria are used: (1) the need for reciprocal best matching (inter-match) between source and target sentences, and (2) a cosine similarity threshold of 0.5. Both criteria need to be met for all language pairs in order to select a sentence. Automatic alignment was manually checked for three languages (EN, FR, RO), with a sample of 1,000 examples. It was correct in 98.1% of cases. The remaining imperfections usually relate to additional context being provided by the translator in one of the languages. This does not affect the sentiment expressed about the target entity, and the obtained alignments of sentences are usable.
Following sentence alignment, it is also necessary to align entity mentions across languages, and
3https://github.com/huggingface/neuralcoref 4https://github.com/averkij/lingtrain-aligner
we used a rule-based approach for this task. NFKD unicode normalization is first applied to examples in all languages. Then we computed a normalized Levenshtein distance between the English mention of the entity and the words from the target sentence. A similarity threshold of 80% between the English and the target mention in any of the other languages was used. To add flexibility to the matching process, we also considered nearly contiguous sequences as valid matches. We have checked this matching and it is correct in over 99% of cases on a subset of 1,000 mentions.
The sentiment classes are not evenly distributed in news, and we followed an initial selection procedure which is inspired by the one introduced in Hamborg and Donnay (2021). It involves an undersampling of potentially neutral mentions as predicted by a simple binary classifier. This led to a pool of 11,000 examples which were selected and proposed to annotators.

heading: 3.3 Sample Annotation
text: Annotations were crowdsourced using a custom web interface. Aware of the challenges of news annotations (Balahur et al., 2010) (i.e. low interannotator agreement and low suitability), Hamborg and Donnay (2021) devised a process in which participants are asked to annotate following the news author’s perspective. While some news articles express sentiments in a manner which is easy to recognize, others convey them in an intricate and/or implicit manner. Equally, the sentiment often depends on text parts distant from the target entity, which are not included in the text presented for annotations. The annotation guide made the annotators aware of the complexity of the task. They were presented with examples of sentences which include intricate and/or implicit sentiments expressions, as well as irony. Annotators also had the possibility to label examples as unknown whenever they could not determine the label of an example.
TSC annotations are usually collected using 3-, 5- or 7-points Likert scales (Balahur et al., 2010; Hamborg and Donnay, 2021; Nakov et al., 2016; Pontiki et al., 2016). Following an initial experiment which involved 50 sentences, we opted for a 5-points scale which offers a good balance between annotation simplicity and expressiveness. Possible labels were: negative, weakly negative, neutral, weakly positive and positive. The annotation was supported by a Web interface which is illustrated in
Appendix E. Examples were provided in English, French and Romanian to facilitate the annotation.
Annotations were provided by a total of 21 volunteer participants, whose demographics are presented in Appendix D. They were recruited via a call for participation which was circulated via group and personal e-mails. Participants provided explicit consent to use their annotations and demographic data at the beginning of the experiment. The choice to work with volunteers is motivated by the fact that crowdsourcing performance is similar for paid and volunteered participation (Mao et al., 2013). Samples were presented randomly in order to avoid any ordering effect, and users were free to stop at any point. Each sample was labeled by three annotators in order to allow annotation consolidation. All users speak at least two of the three languages used in the annotation interface.

heading: 3.4 Label Aggregation
text: Since the annotation task is prone to disagreement (Hamborg and Donnay, 2021; Steinberger et al., 2017), a consolidation of annotations is necessary. We first removed all samples for which there was at least one “unknown” label. Following Hamborg and Donnay (2021), we reduced the five initial labels to three classes (negative, neutral and positive) by aggregating the two possible labels for the negative and positive sentiments. Finally, we kept only samples for which there was an unanimous voting or majority agreement with a third vote in a neighboring class (for instance two positive, one neutral). The inter-rater reliability, measured using Fleiss’ kappa (Fleiss, 1971), reaches KF = 0.58 and KF = 0.67, before and after consolidation, respectively. The two values indicate that the task is challenging, but the final reliability score corresponds to good agreement (Hallgren, 2012). This consolidation strategy leads to a coherent labeling of MAD-TSC, and is used in experiments.

heading: 3.5 MAD-TSC Analysis
text: MAD-TSC includes a total of 5,110 labeled target entity mentions for all eight languages, with 1,839, 2,011, 1,260 of them labeled as negative, neutral and positive, respectively. There can be more than one target entity per example, and there are 4,714 unique examples in MAD-TSC.
Figure 1 shows the total number of unique linkable entities, grouped by source language, from the original corpus of articles used to design MAD-
TSC . These sources are generally correlated to journals from different countries, indicating that the proposed dataset comes from a variety of sources. The total number of unique linkable entities aggregated across all source languages is lower, with 1,007 entities. This is in contrast with NewsMTSC, which is sampled only from American newspapers.
We illustrate the content of the dataset by a number of quantitative and qualitative characteristics. We first present example-related statistics for MADTSC, and compare them to NewsMTSC (Hamborg and Donnay, 2021). These authors use sentence length as a proxy for the complexity of the dataset, and showed that texts in NewsMTSC are longer than those from previous datasets. We follow their approach and report the statistics regarding the number of characters for English examples included in MAD-TSC. The mean is 192.3 characters (72.1 stdev), while the corresponding value for NewsMTSC is 152.2 (109.1 stdev). The number of words per example is a related measure, but more informative from a semantic perspective. MADTSC examples include a mean of 31.1 words (11.6 stdev), while the corresponding values are 25.2 (15.5 stdev) for NewsMTSC. MAD-TSC can thus be considered as more complex than NewsMTSC.
A second analysis focuses on entities which can be linked to English Wikipedia articles using Blink (Wu et al., 2020). MAD-TSC contains 1,007 distinct linkable entities, with a mean of 6.3 mentions per entity (28.9 stdev). NewsMTSC includes 975 linkable entities, 5.5 mentions per entity (43.9 stdev). Donald Trump, Hillary Clinton, and Barack Obama, the top-3 entities from NewsMTSC cover 20.1%, 11.7% and 7.7% of mentions, respectively. In MAD-TSC, Angela Merkel, Silvio Berlusconi, and Nicolas Sarkozy cover 10.2%, 4.9%, and 3.5% of mentions, respectively. We conclude that MAD-
TSC has a distribution of entity mentions which is less skewed.
Third, we examine the geographic distribution of linkable entities in both datasets. The obtained distributions are presented in Figure 2, and they confirm that MAD-TSC is much more diversified than NewsMTSC from a geographical point of view.
Finally, we present clouds of frequent words in the two datasets in Figure 3. This qualitative representation of the two datasets confirms that the main topics are different, with focus on European and on American topics, respectively.

heading: 4 Experiments

heading: 4.1 Compared TSC Methods
text: For English, we compare the performance of four existing TSC methods using English and multilingual pretrained language models. Whenever available, RoBERTa (Liu et al., 2019) backbone is preferred due to its better performance for TSC in news (Hamborg and Donnay, 2021). Otherwise, a BERT (Devlin et al., 2019) backbone is selected. Details about pretrained models used in experiments are provided in Appendix A. A short description of the TSC methods is provided below: • SPC (Song et al., 2019) is based on the classical
Sentence Pair Classification task of Bert models. The input is designed as “[CLS] <sentence> [SEP] <entity> [SEP]”. • TD (Gao et al., 2019) only considers the last hidden states of the entities’ tokens and merges their representation through a max-pooling layer. • PM (Seoh et al., 2021) employs a prompt model for TSC. Our implementation of this method uses the simple prompt "<entity> is [mask]" with ["good", "ok", "bad"] proposed in the original paper as verbalizer. This prompt is translated in
all MAD-TSC languages. • BASE - version of SPC without access to the
entity mention in its input. It can be deployed for any general sentiment classification strategy since annotations of the entity are not used.
We implement the efficient fine-tuning process introduced by Mosbach et al. (2021) to optimize our TSC models. This type of optimization was successfully used in a TSC context (Seoh et al., 2021). We fix the learning rate to 2e-5 and train with earlystopping conditions up to 40 epochs using AdamW optimizer, and batches of size 32. Initial experiments confirmed that this approach is better than the more classical hyperparameter search used in BERT (Devlin et al., 2019). Details about the optimization process are provided in Appendix B. All results are reported by averaging scores of five runs launched with different seeds. Pytorch (Paszke et al., 2019) is used for all implementations.

heading: 4.2 Dataset Splits and Metrics
text: We run experiments with MAD-TSC in monolingual and multilingual settings, and also use NewsMTSC for experiments in English. The training/validation/test subsets are sampled randomly and include 3,810/300/1,000 labeled mentions, respectively. Results for NewsMTSC are reported with the official splits from Hamborg and Donnay (2021). TSC evaluation can be performed with different metrics (Hamborg and Donnay, 2021; Nakov et al., 2016; Pontiki et al., 2016), we use macro F1 (F1m) on all classes as primary metric. Performance with other metrics, such as F1 only on positive and negative classes (F1pn), accuracy (acc), and average recall (rec) follow similar trends. A selection of such results, along with standard deviations are reported in Appendix C.

heading: 4.3 Experiments with MAD-TSC and NewsMTSC
text: We compare the TSC methods from Subsection 4.1 on both the English subset of MAD-TSC and on NewsMTSC. Models are trained and evaluated on each dataset and on their combination. Results with different train and test set combinations are reported in Table 1. Regardless of the specific combination, F1m scores obtained with SPC, TD and PM are similar. The fact that BASE has lower performance (approximately 6 and 3 points for NewsMTSC and MAD-TSC, respectively) confirms that the need to provide the target entity in TSC. The performance of SPC and TD obtained when training and testing on NewsMTSC is 3 to 4 points better than the one originally reported in (Hamborg and Donnay, 2021). This is probably due to a better parametrization of these two methods here. SPC and TD scores are even on a par with those of the more complex GRU method (Hamborg and Donnay, 2021), which needs an external knowledge source that is not available for languages other than English. In addition, results confirm that MAD-
TSC is more challenging than NewsMTSC. The transfer of the models trained on one dataset toward the other test set gives suboptimal results. The combination of the two train sets has a marginal positive effect on each test set. Finally, the language model pretrained specifically for English is clearly better than its multilingual counterpart, probably due to the curse of multilinguality which affects multilingual models (Conneau et al., 2020).

heading: 4.4 Experiments with Individual Languages
text: Results for the eight MAD-TSC languages are presented in Table 2. They are reported using SPC, a commonly used TSC method (Cao et al., 2022; Hamborg and Donnay, 2021; Seoh et al., 2021; Song et al., 2019), and its performance is close to that of PM and TD in Table 1. The best F1m scores are obtained for English and French, and the lowest scores are reported for Dutch and Spanish. When using monolingual pretraining (TG), the difference between the best and worst scores is over 10 points. In contrast, the results obtained with multilingual pretraining (ML) are much more similar across all languages. Performance variability is explained by the quality of pretrained models, and in particular by the size of the datasets and that of the subsets relevant for politics. Interestingly, the multilingual pretraining is much better than the language-specific one for Dutch and Spanish, and is also better for Romanian. Inversely, monolingual models are clearly better for English and French, that are the two languages which have the best monolingual pretraining. The results from Table 2 indicate that strong monolingual pretraining is preferable in TSC, but it can be successfully replaced by multilingual pretraining when the dataset for a particular language is insufficient.

heading: 4.5 Experiments with Machine Translation
text: Machine translation (MT) has strongly progressed in recent years, notably due to the introduction of neural architectures (Stahlberg, 2020). A successful deployment of MT for sentiment classification
would greatly facilitate the task in the multilingual setting because it would reduce, or even remove, the need for specific annotations in each language. Building on previous works which apply MT to TSC (Balahur and Turchi, 2014; Mohammad et al., 2016), we report results with English as pivot language. Test and/or train subsets of the other languages are translated to English. The translation is performed with two methods: (1) M2M100 (Fan et al., 2021), a recent massively multilingual translation model, by using the largest available model (12B parameters); (2) the API of DeepL5, a wellknown commercial machine translation service.
The F1m scores obtained with different MT configurations are reported in Table 3. The results are very interesting, particularly when translating the test set to English with DeepL (row with EN train and ENDL test). F1m scores are globally better than 72.3, the performance of SPC obtained with manual translations for English in Table 1. The maximum gain is 1.6 points for Spanish, and Dutch is the only language for which DeepL translations are slightly worse (-0.2 points). F1m is also interesting with M2M100, albeit lower than that of ENDL. These results surprised us initially, but we reach the same conclusions after running the experiments a second time in an independent manner. When translating the test set, all languages benefit from the strong pretraining available for English, and strong performance can be obtained for them if a good translation model is available from the target language toward English. This condition is met for all languages included in MAD-TSC . Qualitatively, the findings reported here could be
5https://www.deepl.com/en/docs-api
explained by the fact that, while the polarity of sentiments is preserved by both machine and human translators, professional translators are more creative and sometimes add context in sentences for their international public. While useful for human readers, these changes can have a slight negative influence on sentiment classification.
Results are also interesting when the English training set is translated toward the target languages using DeepL and M2M100 (rows with TGDL / TGM2M train and TG test). The associated F1m scores are on par with those obtained with the manual translations. However, the translation of training sets is less effective than that of test sets. This happens because TSC training is done in languages other than English, and is based on weaker pretrained language models.
We also translated both the training and test sets from Dutch and Romanian to English, two of the languages which have low performance in Table 2, using DeepL. The F1m scores for the two languages are 72.3 and 73.6, respectively. We conclude that interesting performance can be obtained by automatically translating TSC datasets from other languages to English.

heading: 4.6 Analysis of Example Complexity
text: We complement the quantitative results by a qualitative analysis of factors which influence TSC performance. The analysis is done for English, and findings are similar for the other languages.
We first test the hypothesis that the complexity of examples is correlated to their length (Hamborg and Donnay, 2021). We split sentences in five subsets depending on the number of words per example and report F1m per subset: 75.0 for up to 20 words per sentence; 72.9 for 21 to 30 words; 70.9 for 31 to 40 words; 70.0 for 41 to 50 words; 70.0 for 50 words and more. These scores confirm that TSC difficulty increases with example length, but differences become smaller above 30 words.
The number of entities detected in each example is an interesting proxy for complexity since the expressed sentiments can vary for multiple entities. We compute F1m separately for examples which include 1, 2, 3 or more detected entities. The scores obtained for the three subsets are 73.6, 70.1, and 67.4, respectively. They confirm that the presence of more entities makes TSC more difficult.

heading: 4.7 Ablation of Training Set
text: The annotation of TSC datasets is a cumbersome task, and it is important to minimize this effort, while preserving the final performance. In Figure 4, we present the results obtained by sampling 1,000, 2,000, 3,000 and 4,000 training examples from NewsMTSC and MAD-TSC, and with the full datasets. F1m scores increase up to 3,000 samples, but the added benefit of more samples is reduced beyond this dataset size. The trend is similar for MAD-TSC , and the results reported in Figure 4 indicate that its size is sufficient to tackle the TSC task effectively.

heading: 5 Conclusion
text: We introduce MAD-TSC, a dataset for multilingual target-dependent sentiment classification. Compared to existing resources, the proposed dataset is aligned across languages, includes examples about geographically diversified entities. Examples are longer and more complex because sentiment is often expressed in an implicit way. Given its aligned character, MAD-TSC dataset enables a comparison of sentiment classification between languages. Performance varies significantly, and this variation is to a large extent explained by the quality of pretrained models available for each language.
Importantly, the MT experiments show that human translations can be replaced by automatic ones. The automatic translation of test sets from target languages to English is particularly interesting since it brings target-dependent sentiment classification in different languages to the same quality level as that of English. This allows TSC to be scaled for the languages included in this study without the need to develop language-specific training sets. The only condition is to have a labeled
domain-related dataset in one language, English or other.
Future work will focus on limitations of the dataset: (1) the handling of example complexity, (2) the coverage of the dataset in terms of domains and entity types, (3) the number of included languages, and (4) the quality of pretrained language models. These limitations are discussed in more details in the dedicated section below.
Limitations
The analysis from Subsection 4.6 shows that the number of entities per example has an important influence on results. For now, the handling of examples with more than one entity includes the detection of the mention, but does not consider other criteria. It would be useful to adapt the TSC methods in order to determine whether the sentiment about all entities is expressed by the same part of the example or not. If sentiment is expressed in different parts of the example, a splitting of the example into parts which express the sentiment about each entity would prove useful.
Despite a more diversified coverage of the political domain compared to NewsMTSC, MADTSC remains focused on politics. It would be interesting to include other major news domains, such as environment, business, culture, technology, sports, etc. Equally important, all targets from MAD-TSC are person names. It would be useful to also include sentiment expressed about other types of named entities (organizations, locations, events, etc.), as well as other polarization-prone concepts in each domain. Such extensions of the dataset would provide a more complete view of TSC performance. Ultimately, they would make the analysis of sentiments expressed in news articles more comprehensive and reliable.
While MAD-TSC is the first multilingual aligned dataset designed for TSC in news, it would benefit from the inclusion of more languages, including non-European ones. This limitation is due to the unavailability of massively multilingual and aligned news datasets which could be used to include more languages. A potential solution to overcome these limitations would be to enrich the dataset with manual translations in other languages. However, this solution is costly and is left for future work.
Finally, the comparison of results between languages is biased because the effectiveness of available pretrained language models is variable. This
is a limitation which is shared by most NLP works which focus on multilingual datasets and reuse pretrained models, themselves trained on whatever datasets available for each language.
Ethics Statement
The work presented here is part of a project which was reviewed and approved by our institution’s ethical committee. This committee provided useful guidance regarding this specific work concerning the selection of news sources, and the annotation process. The recommendations were integrated in the dataset creation process.
One potential issue is that the dataset includes negative sentiments expressed about public figures. This is also applicable to any TSC dataset that focus on politics. Sentiment expressions were collected from publicly available news sources which have a right to freedom of expression in the European Union (EHCR Article 10). News articles were collected from diversified newspapers, which lean toward different parts of the political spectrum, and this reduces the risk of mischaracterizing any of the mentioned entities. Sentiment classification datasets are needed in order to understand how sentiment is expressed in the media, and thus contribute to the characterization of societal debates.

heading: Acknowledgements
text: This work was supported by the European Commission under European Horizon 2020 Programme, grant number 951911 - AI4Media. This work has been funded by the BOOM ANR Project - ANR-20CE23-0024. It was made possible by the use of the FactoryIA supercomputer, financially supported by the Ile-de-France Regional Council.

heading: A Language models
text: All models used in this work are publicly available in the Hugging Face models repository. The main characteristics of the pretrained models used in our experiments are presented in Table 4. Considering the improvements brought by RoBERTa (Liu et al., 2019) pretraining over subsequent fine-tuning tasks
compared to BERT (Devlin et al., 2019), models that replicate RoBERTa in another language are favored when possible. All models are cased versions. Those models are between 100M and 150M of parameters.
The M2M model was downloaded from this repository6.

heading: B Training details
text: Choice of training hyperparameters have been made resulting from the reading of (Mosbach et al., 2021). However, we persist in the use of a validation set but add a loose early-stopping condition of 10 epochs without loss improvement. Results reported are for the model with the best validation loss found during training. Code will be made available upon publication for further implementation details.
The fine-tuning of a pretrained model for TSC takes approximately 3 hours on a Nvidia A100 card. We fine-tuned 52 models with 5 different seeds for each configuration. This leads to a budget of 780
6https://huggingface.co/facebook/ m2m100-12B-last-ckpt
GPU-hours for training. The inference times for the various models used in this paper (Flair NER, Blink, translation models) amount to another 24 GPU-hours. The total budget spent for training and inference is 804 GPU-hours.

heading: C Supplementary results
text: Tables 6, 7, 8 provide the standard deviation values for the experiments reported from Tables 6, 7, 8 of the main text. Notations from the main tables are preserved.

heading: D Annotator Demographics
text: The main demographic characteristics of annotators are: (1) gender - 5 female/16 male; (2) age groups - 14 between 25 and 34, 5 between 35 and 44, 2 over 44, (3) countries of origin: France (16), Romania (4), Ivory Coast (1).

heading: E Annotation interface
text: This section presents the top and bottom parts of the instructions and registration page (Figures 5 and 6, respectively), and an example of annotation page (Figure 7).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: "Limitations" section provided in the template

heading: 3 A2. Did you discuss any potential risks of your work?
text: "Ethics Statement" section provided in the template

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: "Abstract" and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Abstract, Section 1, 2, 3, 4, 5, Limitations, Ethics Statement

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1, 2, 3, 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Subsection 3.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 2, Subsection 3.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Subsection 3.1, Ethics Statement

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, Section 4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A, Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4, Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, Appendix C

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, Section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix E

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 3

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics Statement

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix D

heading: Future Work
text: Future work will focus on limitations of the dataset: (1) the handling of example complexity, (2) the coverage of the dataset in terms of domains and entity types, (3) the number of included languages, and (4) the quality of pretrained language models. These limitations are discussed in more details in the dedicated section below. Limitations
The analysis from Subsection 4. 6 shows that the number of entities per example has an important influence on results. For now, the handling of examples with more than one entity includes the detection of the mention, but does not consider other criteria. It would be useful to adapt the TSC methods in order to determine whether the sentiment about all entities is expressed by the same part of the example or not. If sentiment is expressed in different parts of the example, a splitting of the example into parts which express the sentiment about each entity would prove useful. Despite a more diversified coverage of the political domain compared to NewsMTSC, MADTSC remains focused on politics. It would be interesting to include other major news domains, such as environment, business, culture, technology, sports, etc. Equally important, all targets from MAD-TSC are person names. It would be useful to also include sentiment expressed about other types of named entities (organizations, locations, events, etc. ), as well as other polarization-prone concepts in each domain. Such extensions of the dataset would provide a more complete view of TSC performance. Ultimately, they would make the analysis of sentiments expressed in news articles more comprehensive and reliable. While MAD-TSC is the first multilingual aligned dataset designed for TSC in news, it would benefit from the inclusion of more languages, including non-European ones. This limitation is due to the unavailability of massively multilingual and aligned news datasets which could be used to include more languages. A potential solution to overcome these limitations would be to enrich the dataset with manual translations in other languages. However, this solution is costly and is left for future work. Finally, the comparison of results between languages is biased because the effectiveness of available pretrained language models is variable. This
is a limitation which is shared by most NLP works which focus on multilingual datasets and reuse pretrained models, themselves trained on whatever datasets available for each language