Multi-Task Learning Improves Performance in Deep Argument Mining Models (2024)

Abstract: The successful analysis of argumentative techniques in user-generated text is central to many downstream tasks such as political and market analysis. Recent argument mining tools use state-of-the-art deep learning methods to extract and annotate argumentative techniques from various online text corpora, but each task is treated as separate and different bespoke models are fine-tuned for each dataset. We show that different argument mining tasks share common semantic and logical structure by implementing a multi-task approach to argument mining that meets or exceeds performance from existing methods for the same problems. Our model builds a shared representation of the input and exploits similarities between tasks in order to further boost performance via parametersharing. Our results are important for argument mining as they show that different tasks share substantial similarities and suggest a holistic approach to the extraction of argumentative techniques from text.

heading: 1 Introduction
text: Text content generated by online users is a fundamental source of information for understanding the ideas, feelings, and behavior of large populations of interest for social scientists. Within these texts, it is important to be able to recognize ideas and worldviews expressed by individuals on a large scale. To this end, argument mining (AM) has emerged in recent years as a sub-field of natural language processing (NLP) focusing on creating language models capable of detecting and classifying argumentative strategies in online texts.
Within AM, several different sub-tasks have been proposed. For example, Misra and Walker (2013) focus on identifying agreement and disagreement in online texts, Oraby et al. (2017) propose a method to distinguish factual from emotional argumentation techniques, Lawrence et al.
(2017) detect the presence of certain rhetorical figures in arguments, and Wachsmuth et al. (2017a,b) produce measures of argument quality. These are only some examples of the many distinct classification tasks that have been identified in AM, not to mention a wide range of work on span identification (e.g. Morio et al., 2022). In this paper, we suggest that all these AM sub-tasks share substantial similarity and use this idea to formulate a model that achieves high accuracy in several of these problems.
More specifically, existing work in AM treats many of the classification tasks within the field as separate problems and focuses on fine-tuning bespoke models for each task (e.g. Abbott et al., 2011; Stab and Gurevych, 2014, 2017; Sheng et al., 2020). While this approach has been demonstrated to work in many settings, it fails to take advantage of the substantial similarities between AM tasks.
In this paper we propose to take advantage of the similarities across AM tasks by constructing a multi-task model (Caruana, 1997; Zhang et al., 2014; Zhao et al., 2018; Liu et al., 2015), that is trained on all tasks at once and builds a shared latent representation of the inputs for each task, and uses this representation to make more accurate predictions for each individual task. Our models also provide evidence that AM sub-tasks do indeed share substantial conceptual overlap (Schulz et al., 2018); the latent representations of different tasks output by our model depicted in Figure 1 clearly depict clusters of individual tasks as well as substantial overlap between these clusters in representation space, indicating that the same latent features are informative for multiple tasks.
The model we propose achieves performance similar to or greater than existing models on all tasks for which we had information on previous metrics, and it also surpasses individual-task models fine-tuned on similar architectures for these tasks. In addition, our models allow for substantial
46
computational gains over individual-task models as they permit training inference for many outputs at once, instead of training and evaluating an individual model for each desired task.
Overall, our results have the important implication for AM as a field that further research and model-building should not only focus on taking advantage of the structure of the specific task of interest (e.g. Jin et al., 2022), but also on incorporating information from similar tasks into the model for better performance.

heading: 2 Related Work
text: We build on an active research agenda in argument mining (AM)—the automated extraction of argumentative structure, reasoning, and features from text (Habernal and Gurevych, 2017). Cabrio and Villata (2018) identify two stages in AM: identifying arguments within documents and classifying those arguments on their characteristics, such as supporting, attacking, or background information. Our work is situated in the second stage, involving the identification of features or typologies of arguments.
Much computational work in AM has investigated argumentation in online interactions (Abbott et al., 2011; Rosenthal and McKeown, 2015; Swanson et al., 2015), due in part to the vast amounts of available data and the ease of collecting it. But some scholars have used news articles to construct corpora of propaganda and fact-checking (e.g., Da San Martino et al., 2019; Rashkin et al., 2017). Still others have leveraged monologues like persuasive essays or legal decisions (Stab and Gurevych, 2014; Walker et al., 2019). We incorporate all three
types of data into our models to further show that tasks with different data-generating processes and textual characteristics nevertheless exhibit common semantic structure.
There is evidence that many natural language tasks share a common core (Radford et al., 2019), and models trained on one task tend to also perform well on others. Halder et al. (2020) demonstrate that multi-task approaches benefit model performance in several natural language tasks such as topic detection and sentiment analysis. Multi-task approaches have been more rare within AM, but two existing works suggest the framework may offer benefits to these unique tasks.
Schulz et al. (2018) apply multi-task learning to token-level tagging for AM tasks. They consider six datasets each with different token annotations and train a recurrent model to learn all of them at once, providing initial evidence that gains in model performance can be attained via multi-task learning in AM. Similar results are echoed in Schiller et al., who focus on stance detection instead. Morio et al. (2022) present an end-to-end multi-task architecture for identifying argument components in unstructured text. Our task differs from theirs in two ways: First, Morio et al.’s model is especially focused on span identification and relation classification, wherein the model links, for example, premise to claim. Our primary objective in this paper would more accurately be described as component classification. Second, the component classification part of their model is focused on more traditional AM tasks like classifying texts as claims for or against. We focus instead on argument characteristics that are more complex or subjective and, in some cases, can be understood as being nested within overarching concepts.
Cheng et al. (2020) propose a model trained simultaneously on two tasks: argument identification within texts and argument-rebuttal pair matching across texts. Again, our approach is focused on argument classification, not span identification, and we aim to classify argument types. Moreover, our use of multi-task learning differs slightly. Whereas Cheng et al. train a model to perform two complementary but quite distinct tasks, we show that a single model can perform multiple argument classification tasks simultaneously. Accordingly, our proposed architecture differs from both Morio et al. and Cheng et al.
A prevalent model architecture for multi-task learning within computer vision involves segre-
gating the network into shared and task-specific components. This conventional structure, termed a “shared trunk” (Crawshaw, 2020), typically comprises a universal feature extractor, constructed of convolutional layers that are employed by all tasks, and a distinct output branch for each task (Zhang et al., 2014; Dai et al., 2016; Zhao et al., 2018; Liu et al., 2019). Further enhancements on this shared trunk template have been made by (Zhao et al., 2018) and (Liu et al., 2019), who incorporated taskspecific modules into the original framework.
This template is not confined to computer vision but is also prevalent in multi-task learning models in NLP. Traditional feed-forward architectures, using the shared trunk template in combination with task-specific modules, have been utilized for multitask NLP by a variety of researchers (Collobert and Weston, 2008; Collobert et al., 2011; Liu et al., 2015, 2016). These architectures bear a structural resemblance to their counterparts in computer vision, featuring a shared, global feature extractor followed by task-specific output branches. However, in the context of NLP, the features in question are text representations.

heading: 3 Data
text: We draw on three benchmark corpora to create a dataset with a diverse number of argument characteristics. We take eight tasks from the Internet Argument Corpus (IAC), a collection of posts extracted from several online debate and discussion fora (Abbott et al., 2016; Walker et al., 2012). Each post is annotated on a variety of characteristics, such as whether the post expresses disagreement or uses an emotion- or fact-based argument, with a value in [−5, 5] on each characteristic. Some researchers have dichotomized these data by removing observations around the midpoint Oraby et al. (2015). This practice is not appropriate in the multi-task setting, however, as it would remove too much information that the model could use to build shared representations across tasks. Instead, we dichotomize the data by simply cutting on the scale midpoint.
A wide array of studies have used the IAC to construct unique tasks (Galitsky et al., 2018; Hartmann et al., 2019; Misra et al., 2016) and train single-task models (Lukin et al., 2017; Misra and Walker, 2013; Oraby et al., 2016). Three tasks have received notable attention: the classification of disagreement Abbott et al. (2011); Wang and Cardie
(2014), emotional or factual arguments Oraby et al. (2015), and nasty or nice tone Lukin and Walker (2013).
The second benchmark corpus we draw on is IBM-Rank-30k, a corpus of crowd-sourced arguments Gretz et al. (2020). Two quality scoring functions then translated binary annotations into a continuous value of argument quality in [0, 1]. We use scores produced by the authors’ weighted average scoring function because it accounts for coder reliability, leading to less noisy annotations. As with the IAC labels, we dichotomize the data by cutting on the scale midpoint.
The final corpus is introduced by Da San Martino et al. (2019), who collect articles from both propagandistic and non-propagandistic news sources and annotate sentences within each article that contain one or more of eighteen different propaganda techniques, such as loaded language or causal oversimplification. We extract all sentences from each article, including those that are annotated as containing no propaganda techniques. Data from all three corpora are combined to create our final dataset. We use 80% of the data for training and set aside 10% each for validation and test sets.
Finally, to help guard against overfitting, we conduct four types of data augmentation on the training set (Shorten et al., 2021). In back-translation, we translate the text into a different language, then translate it back to the original language. We choose German as the target language for its high lexical similarity to English. In contextual word embedding, we randomly choose thirty percent of tokens, feed the surrounding words to BERT (Devlin et al., 2018), and substitute the predicted word in for the original. In synonym augmentation, we randomly choose thirty percent of tokens and substitute the most similar word from the WordNet lexical database (Fellbaum, 1998). Finally, in random cropping, we randomly delete thirty percent of tokens. Table 1 shows the total number of observations in the training set as well as the class balance for each task.

heading: 4 Methodology
text: Our methodology is based on a multi-task learning approach which leverages the shared information across tasks corresponding to different sources of data, leading to improved performance on each task. The model architecture and the loss function are the two key components of our methodology.
Additionally, we make use of several standard training and optimization techniques, described in this section, in order to improve performance.

heading: 4.1 Model Architecture
text: Our model architecture shares a key similarity to network templates comprising a shared trunk feeding task-specific modules, common to multi-task learning architectures proposed in previous works (e.g. Zhao et al., 2018; Liu et al., 2015). Morio et al. (2022) also use an architecture with corpus-specific branches, but this portion of their architecture is only used in pre-training; they then fine-tune the model on each corpus individually. Our model is fully multi-task, thus showing the benefit of a multi-branch architecture that needs no fine-tuning.
This architecture aims to utilize shared information across tasks through the shared trunk while learning distinct task features through the taskspecific modules. Following the same principle, we use a network with double-branching in layers following the shared trunk, in order to make use of commonalities across different types of tasks as well as more fine-grained information about each individual task.
We therefore use a feed-forward neural network with four sequential sets of layers: a base text embedding model shared across all tasks, followed by a shared encoder, which is followed by a double branching structure feeding two sets of taskspecific modules. The main results we report use small BERT as the base embedding model (Devlin et al., 2018), but any base model can be used, and we report results across five such models below.
The base embedding model is followed by three dense layers, each followed by dropout. These layers help in learning features that are shared across tasks. The architecture then branches out to learn task-type and task-specific features. In particular, the architecture consists of four sets of layers, de-
scribed below, and visualized in Figure 2. Each dense layer in the network uses a ReLU activation, with the exception of the final activation layer, which is a sigmoid for binary classification.
• Shared embedding layers: We use a BERT model (Devlin et al., 2018) to obtain an embedding of the text input. In order to keep the model size small and training practical, we use small BERT (Turc et al., 2019), which outputs a 128-dimensional embedding. The embedding model, shared across all tasks, is fine-tuned through our training.
• Shared encoding layers: In addition to the base embedding model, all tasks share an encoder, consisting of two sequential dense layers each followed by a dropout layer. This helps learn a shared representation, used by all tasks, while allowing for sparsity and reducing the problem to learning our target features.
• Task-type Layers: The first branching in the network architecture follows the shared layers aiming to learn coarse-grained task-specific features which are expected to share logical structures across tasks within each type. This is particularly suitable for multi-task learning on data consisting of a mixture of datasets, where the number of labels exceeds the number of sources in the mixture. Given such input data, in the first step towards learning the shared representation, the task-type layers learn dataset-specific features, while still utilizing commonalities between individual tasks sharing a dataset. For each task-type branch, we use two sequential dense layers each followed by dropout. Since we have three sets of target labels each corresponding to their own dataset, we use three main branches.
• Task-specific Layers: Each main branch further branches out into individual task layers. These layers aim to learn more fine-grained features from the representations produced through the main branches, and output a vector representation for each task. Each taskspecific branch contains two sequential dense and dropout layers, which feed a sigmoid activation layer for predicting labels. The number of these sub-branches equals the number of individual features in the combined dataset. In the branch corresponding to propaganda
techniques, we additionally use a maximum pooling layer to reduce the eighteen individual propaganda technique labels to a single binary propaganda classification, predicting whether a propaganda technique is used.
The full architecture is illustrated in Figure 2. Using this architecture, we obtain a vector representation of the size of the fine-grained features described in the dataset. Note that this need not be the same as the size of the target output. It is not in this case, as we apply max-pooling to eighteen entries of the output corresponding to propaganda techniques in order to obtain a binary label. The network outputs a real-valued 10-dimensional vector which is then mapped to a binary vector of size 10 using individual thresholds for each label. For the results produced in the main text of this paper, we use a model with 32536 trainable parameters in addition to the parameters in small BERT.1

heading: 4.2 Loss Function
text: The loss function plays a crucial role in our multitask learning approach, which relies on a mixed corpora corresponding to different task types. The custom loss function is designed to handle the data size imbalance across task types, in addition to class imbalance. This helps the model capture the contribution of each prediction to the overall loss, while task types are randomly shuffled in the input data. Considering this, given predicted labels ŷ and true labels y, the total loss L used in our gradient descent optimization is:
L (ŷ|y) = ∑
k
νkL(ŷ|y,Dk),
1Including bias terms, there are 64 × 32 + 32 learnable parameters in the shared layer, 32 × 32 + 32 between the shared layer and each of the task-type branches, 32× 16+ 16 in each task-type branch, 16×16+16 between each task-type branch and each task-specific branch, and 16× 8 + 8 in each task-specific branch.
where Dk denotes the set of data point indices corresponding to task-type k, and νk ∼ 1/|Dk| are the task-type weights. The loss for each task type k, which accounts for the class imbalance across output labels, is:
L(ŷ|y,Dk) ∼ 1 |Tk| ∑
j∈Dk
∑
t∈Tk
∑ c∈Ct wct l(ŷj |yj = c),
where l(.) is the loss function, Tk denotes the set of tasks within task type k, and Ct is the corresponding set of classes. The class weights wct , which are proportional to the inverse of the enrichment of class c in task t within dataset k, counter the impact of class imbalance. We use the binary crossentropy loss for the loss function l throughout our computations. In the implementation, the loss computation is vectorized using masked matrices to filter entries by task.

heading: 4.3 Model Training
text: For training the parameters in our model, we take advantage of an array of optimization and training enhancement techniques. We use an AdamW optimizer (Loshchilov and Hutter, 2017) for the stochastic gradient descent with an initial learning rate of 0.0003. To help avoid overfitting, we employ a weight decay rate of 0.01 and 40% dropout. We use 5% of data for warmup, a batch size of 256, and stop training after two epochs without a decrease in loss. We also incorporate threshold tuning, maximizing true positive rate while minimizing false positive rate, for optimal mapping of the sigmoid layer’s output to binary labels. All training hyperparameters are tuned through a standard grid search over 72 sets of hyperparameters and selected based on validation F1 score.

heading: 5 Empirical Results
text: We evaluate our multi-task model’s performance in terms of prediction metrics, computational efficiency, and comparison against existing metrics on the target labels. We also offer evidence that the tasks we combine do indeed share important similarities by presenting text embeddings and intermediate layer representations, in Figure 1 and Figure 3. We show that our model performs favorably in comparison to previously published models (Table 2), while being substantially more computationally efficient than single-task counterparts (Figure 4).

heading: 5.1 Commonalities Across Tasks
text: Our model was trained on three different corpora, described in section 3, which we argue possess important semantic similarities. To provide evidence of our ten tasks existing within a common representation space, we present t-SNE projections (Van der Maaten and Hinton, 2008) of the input text embeddings corresponding to each label at three different locations within the neural network. Figure 1, discussed in section 1, shows the t-SNE projection from the output of the BERT model we use as our base encoder. Points are color-coded according to their task. If our text data carried mutually exclusive information applicable only to the particular task for which it was labeled, we would see distinct clusters of representations in Figure 1.
There is some minor evidence of clustering, particularly with respect to the propaganda and argument quality tasks, but even those tasks have observations spanning the entire representation space, and they clearly mix with other task representations. This suggests the fine-tuned BERT model is learning representations that reflect similar semantic and logical structures across tasks. We also highlight that the clustering behavior within tasks observable in the figure shows that our model’s embeddings are not completely discarding task-specific structure. Rather, our model learns task-specific representations, and those representations exist within a common space with other task-specific representations, thus further lending evidence to the theory behind our approach.
This pattern is preserved throughout the layers of our model. Figure 3 presents similar t-SNE projections of two other intermediate layers: a shared layer (before any model branching occurs) and the final task-specific layer before the sigmoid activation (after the double-branching). Following the BERT model, each successive layer in the neural network gradually becomes more task-specific, and encodes information that is more relevant to distinguishing among tasks and among labels within tasks. It is notable, then, that we observe similar levels of clustering in the t-SNE projections regardless of model layer. Propaganda and argument quality tasks appear to inhabit more discernible regions of the representation space, but their clusters are neither well-defined nor tightly constrained.
We take this consistent pattern as evidence that AM tasks share a common semantic space. Enabling a model to learn these fine-grained similari-
ties and differences between tasks and across task types is therefore likely to improve performance relative to models that rely solely on shared features or no sharing at all. We test this conjecture in the next section.

heading: 5.2 Performance Evaluation
text: We evaluate the performance of our model primarily in terms of weighted F1 scores, which account for the class imbalances noted in Table 1. Our proposed method represents a significant departure from common approaches to argument classification, so we want to be sure our model is performing favorably relative to other models.In comparison with previous metrics (Table 2), our model shows superior performance in predicting all of the tasks for which we had previous information available. This indicates that effectively leveraging shared features improves performance across multiple tasks.
Table 3 shows a comparison of the predictive performance (as measured by the class-weighted F1-score) between baselines, single-task, and multitask versions of the same model. The baseline metrics represent random guessing and the unigram metrics are produced by a naive Bayes classifier. As may be expected, baselines underperform all
deep-network-based approaches. Morio et al.’s (2022) multi-task model outperforms their single-task benchmarks about 80% of the time, and we see similar results here. Our multitask model outperforms single-task models based on the same encoder architecture in six of our nine tasks. Ablating some layers brings that number up to seven. Again, we take this as evidence that our multi-task model is capable of exploiting the common structure between tasks in order to improve predictions. In Table 6 in the Appendix, we show that this performance gain is not merely due to adding additional trainable parameters; multi-task models of various sizes perform comparably.
We further investigated the impact of changing the base encoding model from small BERT to small ELECTRA (Clark et al., 2020) and base ALBERT (Lan et al., 2019), as well as freezing all BERT layers to prevent the pre-trained weights from being fine-tuned. In addition, we examined the effect of removing the base encoder entirely and using embeddings from two decoder-only architectures— Llama 2 (Touvron et al., 2023) and GPT-3 (Brown et al., 2020)—as input to the model. Table 4 shows a comparison of performance across these different variants of our multi-task model. All models have the architecture described in Section 4, however, the base encoder differs each time. Generally, multi-task models trained on different encoders or embeddings seem to display similar performance, indicating that the gain in performance due to the adoption of our framework is not necessarily due to the specific architecture of the encoder chosen. This is further demonstrated by the comparison of performance for each model variant across individual tasks, which is offered in Table 5 of the Appendix.

heading: 5.3 Ablation Study
text: We executed an ablation study to dissect the contributions of each component of our proposed multitask architecture to its performance. To this end, we omit each of the shared, task-type, and task-specific
layers to obtain the ablated neural networks. The results of this ablation analysis are detailed in Table 3, which compares the performances of the full multi-task model (‘Multi-Task’) against the counterparts with the shared (‘Multi-Task-s’), task-type (‘Multi-Task-p’), and task-specific (‘Multi-Task-t’) layers removed.
Removing any part of the model leads to a decline in task performance for a majority of tasks. Ablating the task-specific layers (‘Multi-Task-t’) causes the most extreme performance drops, with decreases of up to 39.99 F1 points compared to the full model. This points to the significance of the task-specific branches for learning fine-grained representations. No ablated model surpasses the complete multi-task architecture on more than 2 tasks, suggesting that each element of the model structure enables gains in generalization. The results in Table 3 further show that ablation of the task-type and task-specific layers could lead to marginal improvement on two tasks at the cost of significant decline in performance on a few other tasks. This suggests that while the ablated model could better fit a minority of the tasks, different components of the full model facilitate simultaneous learning of all tasks toward consistently strong performance. Moreover, the model with shared layers ablated does not surpass the full model on any task, reinforcing the importance of the shared representations contained in those layers. Overall, these findings affirm the hierarchical design of our multi-task learning framework, where each layer contributes uniquely to the model’s overall success.

heading: 5.4 Computational Efficiency
text: A key consideration, particularly when adding more trainable parameters as our model does, is whether the performance gain comes at the cost of more costly computation. We evaluate the peak GPU RAM usage and time to train our multi-task model and compare them to the same metrics from training the full set of single-task models. We conduct this evaluation by randomly sampling 5%,
10%, 20%, and 40% of the training data to assess how computational load increases with data size. All models for this analysis were trained on one NVIDIA A100 GPU for one epoch. Figure 4 displays the results.
Our multi-task model achieves better performance using substantially lower computational resources overall, proving the branched task-specific modules in our model architecture to be an effective, yet practical, strategy for learning finegrained features for label prediction. Comparing our model’s performance with single-task classification on individual tasks (Table 3), we observe that it achieves comparable performance while decreasing the computation time by at least 31%.
Put together, these observations indicate that this
multi-task learning approach simultaneously has a performance and computational efficiency advantage over single-task models. Computational efficiency plots for different multi-task model sizes are included in the Appendix for comparison.

heading: 6 Conclusion
text: Natural language tasks share substantial semantic and structural similarities, and deep learning models have been shown to be able to take advantage of these similarities in order to achieve better performance (Radford et al., 2019). In this paper, we further extend this result to the field of argument mining. We show that AM tasks do indeed share a substantial amount of features, and that these shared features can be used to boost model performance across previously unrelated tasks. We combine three data sources and propose models that outperform existing models on several of these tasks. Our models are also more computationally efficient and have better overall predictive accuracy than single-task models with comparable architectures. Aside from the practical usefulness of our models, our results are important for argument mining as a field, as they suggest that further research and model building should focus on exploiting commonalities between different tasks to boost performance.
In future work, we propose to extend our analysis to several other AM tasks that share commonalities with those studied here (e.g. Jin et al., 2022), as well as other language tasks such as topic modeling. We also propose devising improved model architectures for our multi-task setting. In particular, we propose to take advantage of frameworks such as contrastive learning (e.g. Chen et al., 2020) to encode known similarities between tasks within the representations learned by the model.

heading: 7 Limitations
text: As with all proposed models, ours carries important limitations. Although we show in the Appendix that the choice of base encoder does not have a drastic effect on performance, we suspect that the performance of our models is largely dependent on the ability to fine-tune a base encoder. Indeed, baseline models using unigram features performed quite poorly. Fine-tuning large base encoders—not to mention training one from scratch—can be computationally expensive. However, transfer learning may be able to help. Common semantic and logical structures across tasks point to opportunities for using transfer learning or pre-trained models from warm start to re-train on new tasks.
Multi-task models also depend on data quality and sufficient semantic overlap across tasks. This is especially challenging in AM, as argument annotation is often highly subjective (e.g. Walker et al., 2012), which can lead to noisy training data. Combining one low-quality dataset with other higher-quality ones may have a detrimental effect on model performance, as the model is unable to learn a shared representation space from noisy annotations, thus degrading performance on all tasks.

heading: A Additional Results
text: In this appendix, we compare the performance of our multi-task model with alternative designs and configurations for multi-task learning, in terms of model architecture, network size, and the base encoder.
A.1 Model Architecture Table 5 compares the performance of our multitask model—which incorporates branched tasktype and task-specific modules—with a standard “shared-trunk” alternative, which consists of only a small BERT encoder followed by a sigmoid activation layer. This comparison shows the utility of our model architecture. Our multi-task model outperforms the shared-trunk model on all but two tasks, where the F1 metric is within 1 percentage point of that of the shared-trunk model. This performance gain comes at a negligible memory cost and a small increase in computation time (Figure 5).
A.2 Network Size We also compare the performance of the small multi-task model we presented in the main text with alternative networks that preserve the same architectural design but increase the sizes of the layers, from 17024 to 272384 and 438784 trainable parameters, following the base encoder. This comparison shows that the superiority in performance, due to the task-type and task-specific modules, is consistent across various network sizes and is not
simply due to adding more trainable parameters on top of the shared trunk (Table 5). Moreover, Figure 5 further confirms that the layers following the BERT encoder are responsible only for a negligible increase in usage of computational resources, as multiplying the combined size of those layers by 16 (Multi-Task, Medium) and 32 (Multi-Task, Large) does not result in a substantial increase in elapsed time for training or peak GPU memory usage.
A.3 Alternative Embedding Models In addition to comparing our model with other multi-task models, we also compare it to other base encoders. In particular, we deploy base ALBERT (Lan et al., 2019) and small ELECTRA (Clark et al., 2020), replacing the small BERT encoder with each of these other base encoders in our multi-task model. Although small BERT achieves the best average performance across different tasks, as the results in Table 6 suggest, using ELECTRA yields an average F1 score within 2 percentage points of that of small BERT, while ALBERT shows more variability across tasks with a lower average F1 score.

heading: Future Work
text: In future work, we propose to extend our analysis to several other AM tasks that share commonalities with those studied here (e. g. Jin et al. , 2022), as well as other language tasks such as topic modeling. We also propose devising improved model architectures for our multi-task setting. In particular, we propose to take advantage of frameworks such as contrastive learning (e. g. Chen et al. , 2020) to encode known similarities between tasks within the representations learned by the model