MaChAmp at SemEval-2023 tasks 2, 3, 4, 5, 7, 8, 9, 10, 11, and 12: On the Effectiveness of Intermediate Training on an Uncurated Collection of Datasets
Year: 2023

Abstract:
To improve the ability of language models to handle Natural Language Processing (NLP) tasks and intermediate step of pre-training has recently been introduced. In this setup, one takes a pre-trained language model, trains it on a (set of) NLP dataset(s), and then finetunes it for a target task. It is known that the selection of relevant transfer tasks is important, but recently some work has shown substantial performance gains by doing intermediate training on a very large set of datasets. Most previous work uses generative language models or only focuses on one or a couple of tasks and uses a carefully curated setup. We compare intermediate training with one or many tasks in a setup where the choice of datasets is more arbitrary; we use all SemEval 2023 text-based tasks. We reach performance improvements for most tasks when using intermediate training. Gains are higher when doing intermediate training on single tasks than all tasks if the right transfer task is identified. Dataset smoothing and heterogeneous batching did not lead to robust gains in our setup. 1

heading: 1 Introduction
text: The introduction of word embeddings and later contextualized transformer-based models, i.e. Large Language Models (LLM), have led to performance improvements on many Natural Language Processing tasks. Technically these approaches are multi-task learning approaches (with parameter sharing over time), where we first train a language model on raw data, save the weights, and then retrain the weights for our target task. It has been shown that intermediate steps of training can be beneficial for downstream performance. This intermediate step of training can be done through language modeling to adapt to new domains or languages (Gururangan et al., 2020; Muller et al.,
1Code available at: https://bitbucket.org/ robvanderg/semeval2023/
2021) or on NLP datasets directly (Phang et al., 2018; Aribandi et al., 2022), exploiting the synergies across tasks. Intermediate training on NLP datasets has benefits both for performance and efficiency (retraining on tasks is cheaper compared to training a full language model).
One important desideratum in finding the right language model for a downstream task is the training data and its distance to the target data. For an intermediate training step on NLP datasets this is even more complex, as there are more design decisions to make; e.g.: which NLP tasks are relevant for the target task? Which datasets are closest to the target data? How many (or how much) of them do we use? Which LLM to use as a starting point? Previous work has investigated mostly single dimensions in this choice of intermediate training datasets (in a carefully curated setup), less is known about what to do if one has a more varied set of NLP tasks.
In this paper, we use all text-based tasks of SemEval 2023 (tasks 2-12) as a seemingly arbitrary set of NLP tasks for evaluating the effect of intermediate training. More concretely, we seek to answer:
• Is the selection of the target or the source task more important for successful transfer?
• How does training on a combination of tasks compare to intermediate training on single tasks?
• Are dataset smoothing or heterogeneous batches beneficial for intermediate training?
• Which properties of datasets and/or tasks are good predictors for performance gains for transfer learning?
We test all of these under a setting of highly varied datasets, in different languages, with different tasks, different task types (single labels, sequences, etc.), different training sizes, and predictions over different input lengths (word, sentence, document).
230

heading: 2 Intermediate Training
text: An intermediate step of language model training has shown to be beneficial to adapt a language model to a new domain, like social media or biomedical data (Gururangan et al., 2020; Barbieri et al., 2022) as well as to new languages (Muller et al., 2021; Chau and Smith, 2021). Here, the intuition is to repurpose the knowledge of previously trained models, and specialize them towards the target domain and/or language by doing further finetuning on a language modeling objective.
Phang et al. (2018) showed that an intermediate training step on an NLP tasks can also be beneficial for performance of LLM’s on the GLUE datasets, which are all on the sentence level. Follow-up work attempted to identify how datasets can be selected for the intermediate training step. Wang et al. (2019) find that language modeling is hard to beat as intermediate task, and that multitask pretraining outperforms single-task pre-training. They also included sequence-to-sequence tasks, and conclude that these are too distant to be beneficial for classification tasks. Correlations on performance of dataset pairs are low showing that it is hard to predict which datasets are beneficial. This is confirmed by the findings of Chang and Lu (2021), who conclude that task complexity is not a good predictor for being a good transfer dataset, whereas Pruksachatkun et al. (2020) find the opposite (complex tasks are good to transfer from), although they do conclude that future work is necessary.
Weller et al. (2022) compare intermediate training versus joint training on sentence-level tasks, concluding that for small datasets, joint training is more beneficial, and for larger datasets intermediate training should be used. Poth et al. (2021) evaluate a variety of (supervised) approaches to automatically identify which source datasets are beneficial, and conclude that pre-computable sentence representations are efficient for this task, and confirm that within task-type transfer outperforms cross-task-type transfer which is in line with the findings of Padmakumar et al. (2022) who compare transfer across and within task types.
Instead of selecting tasks, recent work has attempted to train on a wide variety of tasks. This is commonly done in the space of generative language models, where training on a variety of tasks is easier because many NLP tasks can be converted to generation tasks, and can then directly be used to (re-)train an autoregressive language
evaluated on task 12. Models are depicted by , and the target task is shown in brackets. Note that the multi-task model can output all SemEval tasks, and single (11) is included to better visualize retrain-single (12).
model (Aribandi et al., 2022; Sanh et al., 2022; Chung et al., 2022). In this setup it is easier to exploit a large variety of task types and a much higher amount of datasets (~50-1,800 datasets) is used compared to previous work. However, autoregressive language models still lag behind in performance compared to autoencoder language models for many tasks. Work that focused on intermediate training for encoder models and a large variety of task types is mainly done in the biomedical domain (Parmar et al., 2022; Fries et al., 2022).
Compared to previous work on autoencoder models, we have a larger variety of tasks as well as languages. Our selection of datasets is somewhat arbitrary (all SemEval 2023 tasks), leading to a more challenging setup for multi-task learning compared to previous work who usually used a carefully curated set of datasets (with often 1 language and/or task type). The most similar to our setup is van der Goot (2022), as they also use an arbitrary set of tasks (all SemEval 2022 textbased tasks) and also compare joint training with intermediate training on the full collection of the data. However, their results show no clear trend on when one approach outperforms the other. We build on this work by considering also single-task intermediate training, and systematically analyzing the differences in performance based on properties of the datasets. Furthermore, we evaluate the effect of diverse batching and dataset smoothing.

heading: 3 Setup
text: We first find the best strategy within each dataset (Section 4); we refer to these models as
single. Then we use these best settings of each task to train a single multi-task model covering all tasks (multi-task). We re-train this multi-task model on each target task separately (retrain-all), which is the intermediate training setup described in Section 2. We compare this to a setup where we re-train from single task models (retrain-single), where we focus mostly on the best source transfer task. A schematic over-
view of the setup is shown in Figure 1. For the multi-task models, we evaluate the effect of dataset smoothing, where we use multinomial smoothing with a factor of 0.5. We also experiment with task-diverse batches (i.e. heterogeneous batching: multiple tasks/datasets in a single batch), which has previously shown to be beneficial for intermediate training of generative models (Aghajanyan et al., 2021).
We use MaChAmp (van der Goot, 2022) v0.4 with default hyperparameters for all experiments, with the bert-base-multilingual-cased language model for efficiency reasons. For our final test submissions, we compare 7 language models on the best strategy for each task. We used a pre-selection of multilingual language models based on previous empirical results achieved by MaChAmp.2. The list of language models and their results can be found in Appendix C.

heading: 4 Data and Baselines
text: For each task, we describe an overview of the task, the baseline approaches we evaluated, and the results. An annotated example for each task can be found in Table 1, and an overview with dataset statistics in Table 2. We followed the officially recommended metrics for each (sub)task; we used
2https://robvanderg.github.io/blog/ tune_lms.htm
the internal implementation of these metrics in MaChAmp for model selection when available. We use the Scikit-Learn implementation for f1 scores and conlleval.pl for spans for the scores reported in this paper. For tasks without publicly available dev data (3,7,8,9,10, and 12), we use 80% for train and 20% for dev. Task 6 is described in Appendix B, as we did not manage to officially participate.

heading: 4.1 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER 2)
text: Task 2 concerns multilingual named entity recognition. The data is characterized by its large size and the fact that the entity labels are fine-grained; in total 35 labels are used. The data is taken from Wikipedia, questions from the MS-MARCO QnA corpus (Nguyen et al., 2016), and search queries from ORCAS (Craswell et al., 2020), and is labeled using weak supervision (Malmasi et al., 2022).
In the basic setup, we use a simple feedforward layer on top of the encoder for word-level classification of the BIO labels (seq), we also experimented with a CRF layer that enforces valid BIO-sequences (bio). We further experimented with single language models (SL) and multilingual models (ML). In the multilingual setup, we distinguish between a model that shares the decoder across all languages (shared), and a model that trains a separate decoder for each language (sep).
Results (Table 3) show that the CRF layer is beneficial in all settings. Furthermore, sharing as many parameters as possible is beneficial; the multilingual model with separate decoder outperforms the mono-lingual baselines, but the highest scores are obtained when also sharing the decoder.

heading: 4.2 Task3: Detecting the genre, the framing, and the persuasion techniques in online news in a multilingual setup
text: Task 3 includes three classification tasks on news articles (Piskorski et al., 2023). The inputs are relatively long (734-1210 words per article), which poses problems for current language models. The first subtask is genre classification, in which each article is classified as opinion, reporting, or satire. The second subtask is framing classification, which is a multi-label classification problem with 14 labels. The third subtask is framing technique classification, which is also multi-label, and has 23 labels. For the third subtask, there are also instances without any label, whereas for the second task, each instance has at least one label.
For the first subtask, we use a single feedforward layer to obtain predictions and use a cross-entropy loss. We experiment with single- (SL) and multilingual (ML) models, and for the multilingual models evaluate a shared decoder (shared), and separate language decoders (sep). For subtasks 2 and 3, we evaluate the same setups, and attempt to model the task in three different ways: first, we consider the multi-labels as if they are one label by concatenating them (clas). Secondly, we attempt to model them as separate tasks (sep_clas). Third, we use a multi-label setup (multi_clas) in which we use a BCE loss with a Sigmoid layer and manually set the threshold above which probability we output a label. For the classification tasks, only the first 128 subwords are used due to memory restrictions.
Table 13 (Appendix A) shows that the multilingual classification models perform well for subtasks 1 and 2. Results on subtask 3 are less stable, and multi-clas does better here after finding the optimal threshold. Sharing the decoder is beneficial for subtasks 1 and 3.

heading: 4.3 Task 4: ValueEval: Identification of
text: Human Values behind Arguments
Task 4 (Kiesel et al., 2023) is a classification task for arguments; they are to be classified in one or multiple of 20 human values, which are described in Kiesel et al. (2022). The input consists of a conclusion, the premise’s stance towards the conclusion, and the premise itself. We include all three texts with a special SEP token as divider and give the subwords in the stance segment ID’s (Devlin et al., 2019) of 1.
Because this task is a multi-label classification problem, we compare three approaches: 1) con-
sider the combination of classes as one label (clas) 2) train 20 separate (binary) decoders, one for each label (sep_clas) 3) train a multihead classifier, which uses a BCE loss with a Sigmoid layer and then outputs all labels above a certain threshold (multi_clas. We also evaluated whether concatenating the data from the Zhihu dataset (Kiesel et al., 2022) to the training data was beneficial, but saw lower scores across all approaches.
Results in Figure 2 show that training separate classifiers for each label is not beneficial, simply concatenating labels leads to higher performance. However, training a multi-head classifier and tuning the threshold leads to superior performance, where tuning the threshold is crucial.

heading: 4.4 Task 5: Clickbait Spoiling
text: Task 5 concerns the classification (subtask 1) and spoiler detection (subtask 2) for clickbait posts and their corresponding full text. A clickbait post is a short text that is intended to inappropriately entice readers to visit a web page. The data was taken from Facebook, Reddit and Twitter (Fröbe et al., 2023; Hagen et al., 2022). For subtask 1, three types of spoilers are classified; phrase, multi, and passage. These labels refer to the type of extraction that is needed to find the spoilers, phrase means that the spoiler is just a short phrase, passage refers to spoilers consisting of subsequent phrases, and multi to more complex spoilers (combination of phrases throughout the document). Subtask two concerns the extraction of spoilers in the text from the web page, this is done by locating a (series of) span(s).
For subtask 1, we identify which pieces of information from the dataset are useful for the target labels. We include the text of the post, the description, title text, keywords, links of media, and URL of the webpage. We tried each of these inputs in isolation and found that all of them beat the major-
ity baseline, so our final model includes all inputs. For subtask 2, we convert the character offsets of the spans to BIO labels on the word level and evaluate a sequence labeler with and without CRF layer. Note that this does not take the clickbait post into account, and is thus a non-realistic setup.
Results show that the original clickbait text (Post text) is the most predictive source of information in isolation. However, each of the used information categories outperform the majority baseline (19.68). Because of these positive results, we also train a model with all categories, which slightly outperforms using only the post text. After ablating the least informative category (media) from the “all” model, we observe a performance drop and decided to stick with the full combined model. For subtask 2, the CRF layer with BIO constraints is superior once again.

heading: 4.5 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data
text: Task 7 consists of data from clinical trial reports for breast cancer studies (Jullien et al., 2023). The first subtask is to distinguish entailment from contradiction based on a clinical trial report and a statement. The second subtask is to extract the spans of the supporting facts to justify the output of subtask 1. The facts are sentences, and the sentence boundaries are given.
We implement the first task as classification task. We tried to predict based on the statement, and additionally experiment with adding the section and/or the text. The second subtask we also implemented as a classification task, we include pairs of the statement and 1 sentence from the trial report at a time, with a binary label (relevant or not).
For subtask 1 The majority baseline would score 33.33 F1, so by just using the statement we can
already obtain a substantially higher performance (Table 5). For subtask 2, the majority baseline would score 52.21, so the classification per sentence pair is effective.

heading: 4.6 Task 8: Causal medical claim identification and related PICO frame extraction from social media posts
text: Task 8 (Khetan et al., 2023) is based on medical data from Reddit (Wadhwa et al., 2023). The first subtask is to identify spans that represent claims, experiences, experience-based claims, or questions. The second subtask is to find spans that identify the population, intervention, or outcome frame of a claim. It should be noted that the spans of the first task are much longer (average of 23 words) compared to the spans of the second subtask (average of 2 words). The text for task 8 was not released, but a scraping script was provided. We scraped the data on 07-11-2023, and after aligning the annotations we noticed that some posts were deleted or edited. For subtask 1, we removed 767 of the 5695 posts, and for subtask 2, 72 out of 597.
The results (Table 6) show a clear trend; for the longer spans of subtask 1, the sequence label approach with a feedforward layer is ineffective,

heading: 10-1 80.68 78.96

heading: 10-2 52.09 55.57

heading: 10-3 36.28 35.16
text: and much can be gained with a CRF layer. On the contrary, for the second subtask, the difference is negligible, probably due to the short length (and thus easier boundary detection) of the spans.

heading: 4.7 Task 9: Multilingual Tweet Intimacy Analysis
text: The task is to identify intimacy on tweets on a scale of 1-5. More details about the data collection can be found in Pei and Jurgens (2020) and Pei et al. (2022). Besides the six languages available in training (Table 7), Hindi, Arabic, Dutch, and Korean are test-only languages.
We implement the task as a regression task in the MaChAmp toolkit, which uses an MSE loss. We compare mono-lingual models against multilingual models and evaluate the use of a shared decoder and language-specific decoders. We use the average (absolute) distance between the gold and predicted label for model selection (because it is the only regression metric currently available in MaChAMp), and report Pearson r following the official metric.
Results (Table 7) show that each of the model varieties performs best for some of the languages. The highest average is obtained by the multilingual model with a shared decoder, however, it performs bad on French. A manual inspection revealed that this difference is mostly due to bad performance on short sentences.

heading: 4.8 Task 10: Towards Explainable Detection of Online Sexism
text: This task concerns the detection and classification of sexism against women on social media data from Gab and Reddit (Kirk et al., 2023a,b). The task is divided into three subtasks: 1) binary classification (sexist or not), 2) only for sexist posts: in which category does it belong: threats, derogation, animosity, or prejudiced discussions. 3) which subcategory does the post belong to, there are two or three subcategories per main category from subtask 2.
We implement each of these tasks as a classification task and compare it to a model that includes
all three tasks simultaneously. We use macro-f1 for all tasks, following the official metrics (although task 1 is a binary task).
Results (Table 8) show that the multi-task model is only beneficial for the coarse categories classification (subtask 2). This might be due to this task being in between both other tasks, and thus more closely related to the others. It should be noted that across the tasks there is no error propagation, as subtask 2 is only evaluated on sexists posts and subtasks 2 and 3 are implemented as separate tasks.

heading: 4.9 Task 11: Learning with Disagreements (Le-Wi-Di)
text: Task 11 contains a variety of 4 NLP tasks (Leonardellli et al., 2023): misogyny detection on Arabic tweets (ArMIS), abuse detection in dialogues (ConvAbuse), hate speech detection on data concerning the Brexit (HS-Brexit) and offensiveness detection on tweets from 5 different topics (MD-Agreement). All of these tasks contain soft labels, which are the average scores over multiple annotators (i.e. a float between 0.0 and 1.0). The official evaluation metric is cross-entropy, but we use MaChAmp’s default metrics for each task-type for model selection.
We predict the label as a regression task as well as a classification task, as the number of annotators is between 4 and 10, and the number of labels is
relatively small. Furthermore, we evaluate using a multi-dataset model with a shared encoder as well as separate decoders for each task.
Results (Table 9) show that the regression task type performs better for this task. Single-dataset models outperform the multi-dataset setup, probably because the tasks are too diverse.

heading: 4.10 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset
text: Officially there are three sub-tasks, but they all concern the same task, they merely differ in the setup. The task is binary sentiment analysis (Muhammad et al., 2023a); the first subtask is for mono-lingual models, the second for multilingual models and the third is for transferring to new unseen languages (Muhammad et al., 2023b). We consider all of these settings and use the best models from the multilingual setting for the cross-lingual setup.
We compare single language models to multilingual models, where we test with shared decoderheads heads and separate decoder-heads. Table 10 shows that the single-task models perform best on average. Although, for challenging languages (SL < 10.0) multilingual learning is beneficial.

heading: 5 Results
text: The scores on the development data for all our setups (Figure 3) show that the single-task intermediate training is most successful (highest for 12/21). It should be noted, however, that we only plot the best single transfer task, which does need to be found first. The multi-task setup only performs best for one task (8-1), this is because it is the only model that has to share parameters in the model that is used for prediction. For most tasks, the gains obtained with intermediate training are substantial, but there are five cases where the single model performs best; two of these are the soft labels of task 11.
Results on the test data (Table 11) show that al-
though we rank high for 3 tasks (4-2, 8-1, and 8-2), for most tasks, we rank somewhere in the middle. This was expected, as we do almost no tuning per task, no ensembling, use no domain/languagespecific embeddings and do not use any nonSemEval data nor data augmentation.

heading: 6 Analysis
text: We plot the difference of each retraining setup to the single task baseline in Figure 4; positive numbers mean the baseline was outperformed. Results show that multi-task learning and intermediate training leads to performance loss more often than performance gains in our setup, which is probably a result of having large training sets and good performing baselines. In general, some datasets are particularly good at being on the receiving end of transfer, whereas there is no clear dataset to transfer from (rows are more consistently colored than columns). Tasks that particularly benefit from intermediate training are tasks 6-3, 10, 11-1, and 11-2. For task 6-3, we have a very low baseline, and results seem unstable, task 10 gains from almost all
single task transfers, but mostly from itself (note that we just train twice on the same data here).
The performance of the models trained on all datasets is not more robust compared to the singletask source models, although scaling up further might change this effect (Aribandi et al., 2022). Diverse batching as well as dataset smoothing only lead to performance improvements in specific cases in our setup. We do not find improvements similar to Aghajanyan et al. (2021) for diverse batching, which is probably either because they use the same decoder for all tasks (in a generative model), or because of the scale of datasets.
To evaluate which properties are important for dataset selection in multi-task learning we perform a correlation study. We use the single dataset transfer results (differences to baseline on dev, i.e. first 21 columns of Figure 4) as target variable, and evaluate the correlation against a variety of properties of the tasks. Results (Table 12) show that the strongest (negative) correlations are found for the dataset size variables. For source datasets, we hypothesize that this is because the model overfits towards large datasets, which impedes transfer. For target datasets, this is because the model is undertrained, and there is more space for improvement. Perhaps surprisingly, baseline scores are much less informative compared to dataset size, although baseline scores are expected to correlate with baseline performance. However, it should be noted that different metrics are used, making the correlation less reliable. The number of tasks and number of languages are both negatively related to performance, as there is already sharing going on in the baseline setting, adding even more varieties of data is not beneficial. Perhaps surprisingly, task type overlap is also negative, however, this is
tas k2
_1 tas k3 _1 tas k3 _2 tas k3 _3 tas k4 _1 tas k5 _1 tas k5 _2 tas k6 _1 tas k6 _2 tas k6 _3 tas k7 _1 tas k7 _2 tas k8 _1 tas k8 _2 tas k9 _1
tas k1
0_1 23
tas k1
1_0 tas k1 1_1 tas k1 1_2 tas k1 1_3 tas k1 2_1 ret rai n
ret rai
n-d ive
rse
ret rai
n-s mo
oth
ret rai
n-s mo
oth -di
ve rse mu lti
mu lti-
div ers
e
mu lti-
sm oo
th
mu lti-
sm oo
thdiv
ers e
Source task(s)
task2_1 task3_1 task3_2 task3_3 task4_1 task5_1 task5_2 task6_1 task6_2 task6_3 task7_1 task7_2 task8_1 task8_2 task9_1 task10_123
task11_0 task11_1 task11_2 task11_3 task12_1
Ta rg
et ta
sk -0.3 -0.2 -0.1 -76.0 -0.8 -2.2 -0.5 -0.2 -0.7 -1.9 -0.6 -1.1 -0.0 -1.0 -0.7 -0.9 -0.5 -0.4 0.1 -0.1 -0.2 -2.7 -10.7 -2.3 -3.1 -3.6 -16.4 -2.7 -4.3 1.0 -3.7 -6.5 -56.6 -4.1 -6.3 -0.8 -8.3 -4.1 -29.1 -3.0 2.1 -8.9 -6.3 -8.9 -0.6 -2.2 -5.1 -4.2 0.2 -4.2 -7.4 -15.5 -8.9 -3.5 -14.1 -24.8 -24.3 -21.0 -9.0 -20.0 -8.8 -51.8 -11.2 -20.2 -9.1 -14.4 -10.0 -9.4 -11.1 -20.4 -8.1 -14.8 -7.8 -14.2 -13.5 -11.2 -9.9 -10.1 -11.8 -0.8 -5.9 -5.0 -5.2 -2.3 -8.0 -4.2 -4.4 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -42.3 -13.4 -4.5 -6.4 -4.4 -3.1 -2.2 1.2 -31.5 -4.0 -3.0 -2.0 -0.3 0.1 0.9 0.9 -1.9 1.4 -1.0 2.6 -3.1 0.5 2.6 0.4 -1.4 -0.9 -5.8 -13.9 -6.5 -8.4 -1.6 -10.5 -5.6 -4.2 -2.6 -0.3 -1.4 -66.8 -2.0 -0.2 -1.9 -1.2 -1.7 -47.1 -0.5 -2.9 -1.5 -0.7 -0.8 -1.5 0.3 -0.1 -0.5 -1.6 -0.9 -4.9 -13.6 -7.9 -6.9 -30.9 -17.3 -16.2 -8.7 -6.8 -5.5 -3.0 -15.5 2.8 -1.4 4.1 -4.8 -6.0 -15.4 3.9 -7.6 4.3 4.0 -6.7 -7.7 -5.0 -6.7 -7.4 -7.3 -4.6 2.9 0.8 1.9 1.3 -0.6 -15.5 -1.3 2.4 -4.2 -1.3 -0.3 -49.2 -2.1 -12.2 -49.2 2.5 -2.1 -49.2 -1.8 -4.9 -1.1 -0.1 -1.9 -2.3 -0.4 -2.8 -0.5 -3.0 -1.2 2.4 -4.7 -2.0 -1.4 -49.2 -49.2 -1.3 -2.1 -1.3 -0.0 -0.9 -76.5 -1.3 -2.7 -0.4 -0.5 -0.5 -47.8 0.1 -2.0 -0.4 -0.5 -1.1 -1.2 -0.6 -1.1 -0.3 -1.5 -0.8 -3.5 -10.3 -3.6 -5.0 -59.1 -38.6 -59.6 -42.8
-0.4 11.4 4.8 -33.7 1.4 18.3 10.6 9.4 -0.3 10.6 7.2 -0.4 10.7 10.4 1.6 2.6 11.7 10.6 2.7 11.5 2.8 -0.1 16.7 4.5 11.9 -0.5 5.3 6.2 18.2
-1.9 -0.1 -2.0 -61.8 0.0 -3.6 -1.0 -1.6 -1.0 -2.7 0.3 -6.7 0.6 1.8 -1.2 -1.6 -1.8 -0.9 0.5 -1.8 -1.5 -6.2 -5.3 -5.0 -5.0 -4.5 -13.6 -10.4 -26.8
-20.6 -14.9 -16.7 -75.6 -19.0 -14.4 -13.3 -5.1 -9.9 -41.0 -9.5 1.3 -0.0 -19.6 -9.0 -10.8 -17.9 -1.5 -7.8 -1.6 -11.0 -3.2 -15.3 2.3 -19.9 -14.3 -18.1 -6.8 -9.0
0.0 0.1 -0.2 -34.9 -0.3 -1.6 1.0 -1.1 -0.1 -33.2 -0.3 -1.4 -1.3 -0.1 -0.2 -0.3 -0.5 -0.3 -0.4 -0.0 -0.3 -0.7 -4.2 -1.5 -2.1 2.1 -1.8 1.3 3.3
2.1 0.9 1.8 -25.9 -0.1 -8.9 0.9 0.2 1.3 -25.5 1.4 -3.8 0.5 1.0 -1.8 -2.1 0.9 1.7 -0.7 -0.7 0.0 -0.9 -8.9 -1.0 -2.3 -4.6 -10.8 -5.8 -5.5
0.1 -0.0 -0.0 -1.6 -0.0 0.1 0.0 0.0 -0.0 17.5 -0.0 0.0 0.0 -0.0 0.0 -0.0 -0.0 0.0 0.0 0.0 0.0 0.3 0.5 0.2 0.2 0.4 0.7 0.7 0.3
5.3 4.0 5.0 -57.7 -3.4 -0.6 6.6 6.2 5.1 -18.2 -1.0 3.6 2.8 5.8 4.9 7.3 6.6 4.3 5.6 5.7 5.6 -8.6 -8.8 -7.0 -6.2 -7.1 -10.6 -8.3 -6.2
-0.4 -1.8 -0.3 -4.7 -0.8 -2.1 -2.0 -1.9 -1.5 -0.9 -1.9 -1.8 -2.0 -0.5 -0.2 0.5 -1.9 -1.3 -0.6 -0.2 -0.7 -2.6 -2.9 -2.5 -2.1 -3.0 -3.6 -2.7 -1.9
0.1 -2.2 2.3 -7.0 0.1 -2.2 -0.9 1.7 0.7 5.0 -1.1 5.1 1.1 -0.1 1.1 1.8 0.4 -0.4 9.2 -0.4 1.7 -3.0 5.6 4.7 2.2 -0.6 -2.8 -0.1 7.6
-16.0 -9.7 -9.1 -17.8 -15.1 -17.7 -10.7 -11.9 -15.8 -17.8 -1.0 -15.9 -16.8 -16.0 -9.1 -13.9 -9.5 -5.6 -12.2 -14.5 -8.3 12.5 -1.1 -14.8 -17.3 -6.5 12.1 -10.2 -9.8
0.5 -0.3 -0.2 -4.4 -0.2 0.3 0.3 0.3 0.2 -0.9 0.4 0.1 0.2 -0.7 0.0 0.2 0.3 0.4 -0.4 0.1 -0.1 -0.2 -1.1 -0.1 0.5 0.6 -1.3 0.3 0.6
-1.3 1.8 2.1 -26.5 -0.7 1.0 2.1 -0.0 -1.3 -17.5 0.0 0.4 0.8 0.0 -1.9 -0.6 0.6 4.6 0.5 1.0 -0.2 -1.5 -3.2 -2.1 -1.5 -5.0 -8.1 -5.1 -3.7
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Figure 4: Difference in metric with intermediate training and multi-task learning versus baseline. The target datasets are on the y-axis, the source on the x-axis. Note the the intermediate step can be 1 dataset or all. The last 4 columns represent the multi-task models that are not retrained. Note that the color gradient start at 0 to 30, as we focus on gains. Also note that each row has its own metric, and for the task11 subtasks we use (1/cross_entropy) *100.
highly dependent on the choice of datasets, and the task types are imbalanced in our sample.
To gauge the effect of simply upscaling the number of datasets, we also trained models on the combination of the data in Section 4 and the SemEval 2022 datasets used by van der Goot (2022). This leads to approximately twice as large training data, but no robust gains in performance; for some tasks
the combined data increases performance for others it does not (results in Appendix D). Although the scale of data does not match recent studies in the space of generative language models (Chung et al., 2022), our results do suggest that scaling up might not be the most prominent direction for obtaining robust multi-task autoencoder models.

heading: 7 Conclusion
text: We evaluate the effect of multi-task learning and intermediate training on all text-based SemEval 2023 tasks and showed that for a non-curated set of benchmarks it is hard to obtain consistent improvements. We found that specific target tasks are likely to gain from intermediate training, but finding the right source dataset/task is non-trivial. Training on a wide variety of tasks has shown to not be more robust compared to finding the best source task to transfer from. In contrast to previous work, having task-heterogeneous batches did not lead to consistent performance gains in our setup.

heading: Acknowledgements
text: I would like to thank my colleagues at NLP North and the anonymous reviewers for their feedback on this project. Thanks to all task organizers for providing the data. I acknowledge the IT University of Copenhagen HPC resources made available for conducting the research reported in this paper.

heading: A Task 3: Results
text: Due to space constraints, results on task 3 can be found here (Table 13).

heading: B Task 6: LegalEval: Understanding Legal Texts
text: Task 6 has three subtasks, and consists of legal texts. The first subtask is to identify the rhetorical roles in a text, these can be extracted as spans, and we model them as BIO-labels on the word level. It should be noted that the labels are roughly assigned on the sentence level, meaning that the spans with the same label are often subsequent (i.e. a paragraph that is an analysis, can be 5 sentences long, and consist of 5 spans), however, finding the boundaries is also part of the task. We included the None label, because it is also included in the evaluation (and data) as a class. The second subtask is named entity recognition, containing 13 different types of entities. For this subtask, two different types of texts are used: preambles and judgements. The third task concerns predicting the outcome of a legal judgement document (binary: accept or reject), and providing the relevant text spans in the document that explain this outcome.
For subtask 1, we compare the use of a standard sequence labeling to an approach with a CRF layer. For the second subtask, we compare the same types of models, but also use different strategies to handle the multi-dataset setting. We train models on the separate datasets, and attempt to combine them within single decoder, as well as a setup where each dataset has its own decoder. For the third subtask, we could not identify the annotation of the relevant text spans in the provided data, so we only predict outcome as a binary classification.
For subtask 1, it is clear that the CRF layer with BIO constrants is beneficial (Table 14). For subtask 2, the single dataset models perform best and surprisingly, the CRF layer is not beneficial for one of the datasets, leading to a lower average compared to using a simple feedforward layer. For subtask 3, our performance matches the majority vote baseline, this is because the model predicts every case as being rejected. We find two possible reasons for this: 1) the dev data is balanced, but the training data has more rejected cases. 2) the input texts are long (average of 4012 words), and only the first 128 subwords are taken into account in the default settings of MaChAmp (to save memory). We leave a more detailed analysis for future work.

heading: C Language models comparison
text: In table 15 we compare the performance of a variety of language models for the best single task settings. For our final test split submissions, we used the best setting of Figure 4, and retrained it with the best language model for each dataset.

heading: D Training on SemEval 2022 and 2023 data
text: We use the best settings of van der Goot (2022) for the 2022 data, and train the multi-task model on the combination of all 2022 and 2023 datasets. We plot the best setting (smoothing/heterogeneous batching) for each setup in Figure 5.

heading: Future Work
text: We leave a more detailed analysis for future work