Multi-Label Classification for Implicit Discourse Relation Recognition (2024)

Abstract: Discourse relations play a pivotal role in establishing coherence within textual content, uniting sentences and clauses into a cohesive narrative. The Penn Discourse Treebank (PDTB) stands as one of the most extensively utilized datasets in this domain. In PDTB-3 (Webber et al., 2019), the annotators can assign multiple labels to an example, when they believe that multiple relations are present. Prior research in discourse relation recognition has treated these instances as separate examples during training, and only one example needs to have its label predicted correctly for the instance to be judged as correct. However, this approach is inadequate, as it fails to account for the interdependence of labels in real-world contexts and to distinguish between cases where only one sense relation holds and cases where multiple relations hold simultaneously. In our work, we address this challenge by exploring various multi-label classification frameworks to handle implicit discourse relation recognition. We show that multi-label classification methods don’t depress performance for single-label prediction. Additionally, we give comprehensive analysis of results and data. Our work contributes to advancing the understanding and application of discourse relations and provide a foundation for the future study.

heading: 1 Introduction
text: Discourse relations serve as a fundamental framework for creating coherence among sentences and clauses within a text. The automated identification of the discourse relations, which connect sentences and clauses, holds significant importance for a wide range of downstream Natural Language Processing (NLP) tasks including text summarization (Huang and Kurohashi, 2021), question answering (Pyatkin et al., 2020), and event relation extraction (Tang et al., 2021). Penn Discourse Treebank (PDTB) (Webber et al., 2019; Prasad et al., 2008) is one of the most widely used datasets for this task. It
includes over 2000 documents from the Wall Street Journal published in 1989, which have been manually annotated with discourse relations.
With regard to PDTB annotation, annotators can assign multiple sense labels to an example when they believe that all hold simultaneously. Consider the following example from PDTB-3.
(1) [In the past decade, Japanese manufacturers concentrated on domestic production for export]. [In the 1990s, spurred by rising labor costs and the strong yen, these companies will increasingly turn themselves into multinationals with plants around the world]. (wsj_0043). Labels: Asynchronous and Concession
This example demonstrates the simultaneous presence of two distinct discourse relations. The annotator wanted to capture the sense that the first sentence described an earlier time period and the second, a subsequent time period. But the annotator also wanted to capture the sense that DESPITE Japan’s concentration on domestic production for export in the earlier time period, the future will be different – i.e. conceding what was true in the 1980s, and asserting that the future will be different. Therefore, the pair of arguments in the example is linked by two discourse relations that capture different dimensions of the relationship between them. Understanding these complex relations is crucial for the interpretation of discourse.
However, for those instances with two annotated labels, all previous work on discourse relation recognition treat them as separate and different examples during training, and at test time, a prediction matching one of the gold types is taken as the correct answer.
Nevertheless, by treating instances with multiple labels as separate examples, the model may not effectively capture the inherent complexity of discourse relations. Real-world texts often contain multiple layers of meaning, and forcing the model
8437
to treat them as distinct instances may oversimplify the problem. This approach may not reflect the true nature of discourse relations, where multiple relations can coexist between two arguments. Moreover, when treating instances with multiple labels as separate examples, the model loses valuable contextual information about how these relations interact and influence each other. Furthermore, treating multi-label instances as separate examples might introduce training ambiguity and lead to conflicting patterns, impairing the model’s ability to generalize to new examples. Besides, when implementing discourse relations recognition in downstream tasks, the inability to recognize multi-labels could potentially lead to adverse effects. For Example 1, if the model fails to identify both Concession and Asynchronous relation simultaneously, it may struggle to respond to questions concerning temporal order and contingency relation concurrently.
To address these negative impacts, we explored multi-label classification as a more effective way to capture the complexity of discourse relations. To date, implicit discourse relation recognition has not been approached as a multi-label classification task, despite advancements facilitated by pretrained language models (Zhang et al., 2021b; Long and Webber, 2022; Wu et al., 2023). The current study compares three multi-label classification methods for IDRR, alongside conducting an in-depth analysis of the results and data. The main contributions of our work are as follows:
• This is the first study to treat implicit discourse relation recognition as a multi-label classification issue, marking a novel approach in addressing the complexity of discourse relations more effectively compared to traditional single-label prediction methods. • We explore different multi-label classification methods for IDRR, and we show that a multilabel classifier can even demonstrate better performance than a classifier trained on examples in which multiple labels were split into two distinct and unrelated examples. • We provide a fine-grained analysis of results and conduct some methodological exploration, which can give valuable insights and pave the way for the future study.

heading: 2 Related Work
text: Multi-label Classification Multi-label classification has been widely adopted across different NLP
applications, including intent detection (Ouyang et al., 2022), emotion classification (Yu et al., 2018; Alhuzali and Ananiadou, 2021), and document classification (Xiao et al., 2019; Liu et al., 2022; Lin et al., 2023). Despite the potential for richer text understanding, multi-label classification has been unexplored in discourse relation recognition. In this work, we expand the use of multi-label classification to implicit discourse relation recognition to enhance the accuracy and depth of NLP applications from richer text understanding. Multi-label Study on Discourse Relation Recognition Two notable studies on multiple label examples for discourse relation recognition are: 1) Marchal et al. (2022), which explores inter-coder agreement in multiple label examples, revealing instances where annotators assign more than one relation to a single example from various genres. 2) Scholman et al. (2022a) introduces a crowdsourced corpus containing 6,505 implicit discourse relations across three genres. Annotators in this corpus assign a single discourse sense based on their perspective, resulting in varying interpretations. Nevertheless, the identification of implicit discourse relations has not been addressed in the context of a multi-label classification framework.

heading: 3 Dataset and Evaluations
text: We employed PDTB-3 (Webber et al., 2019) for our evaluation. PDTB-3 represents an advancement over PDTB-2 (Prasad et al., 2008), offering a more extensive collection of annotated multi-label examples. In our study, we concentrated exclusively on implicit discourse relations, disregarding those with explicit connectives. And we consider both intra-sentential and inter-sentential implicits.
About 5% of PDTB-3 implicit discourse relations receive multiple labels, which corresponds to instances with two annotated labels. We treated such instances as single examples with multi-labels during training, and during testing, predictions were considered correct only if they match the specific label.
While previous studies (Ji and Eisenstein, 2015; Bai and Zhao, 2018; Xiang et al., 2022) typically allocate Sections 2-20 of PDTB for training, Sections 0-1 for validation, and Sections 21-22 for testing, the limited size of the test set poses challenges, particularly for rare label and label pairs within the dataset. Acknowledging the concerns raised by Shi and Demberg regarding label sparsity, we
addressed this issue by employing cross-validation for Level-2 classification. In line with the methodology proposed by Kim et al., we adopted a crossvalidation approach at the section level. We divided PDTB-3 into 12 folds, with each fold partitioned into 21 sections for training, two for development, and two for testing. By splitting the data at the section level, we can preserve the inherent paragraph and document structures, ensuring that data from the same sections are grouped together in the same pool.
Following the work in multi-label classification for other tasks (Tsai and yi Lee, 2019; Zhang et al., 2021a), we adopt F1 scores (Manning et al., 2008) as our main evaluation metric. Precision, Recall and Hamming loss (Schapire and Singer, 1998) are reported in the Appendix. We present the macroaveraged results of F1 scores, Precision and Recall.

heading: 4 Performance Comparison: Evaluating the Effectiveness of Different Methods

heading: 4.1 Methods
text: Our work has explored three different multi-label classification techniques, two encoder-only methods and one encoder-decoder method.

heading: 4.1.1 Method 1
text: The output vector corresponding to the [CLS] token aggregates input features and is used for classification. We employ RoBERTa for text representation learning, and add a classification head Wc ∈ RH×|C| on top of the [CLS] token to do classiciation. H is the dimension size of [CLS] representation and C represents the number of classes. We use y ∈ R|C| to denote the ground-truth label for an example, where y ∈ {0, 1}|C|.
The model is trained using sigmoid binary crossentropy loss. If the predicted probability of a label surpasses 0.5, it is regarded as a predicted label.

heading: 4.1.2 Method 2
text: This method resembles Method 1, with several key distinctions. Rather than employing a single classification head to handle all labels, we utilize multiple classification heads Wci ∈ RH×2, each ci dedicated to the i-th specific label and treating them as individual binary classification tasks. In contrast to Method 1, which utilizes sigmoid binary crossentropy loss, we employ softmax cross-entropy for loss calculation here. The loss for each label is computed independently, and subsequently, the
mean of these individual losses is used to update the model. If the predicted probability of a label is greater than 0.5, it is considered a predicted label.

heading: 4.1.3 Method 3
text: In this approach, we use a sequence-generating model that processes input text token by token, predicting labels sequentially while considering previously predicted labels. Our method is similar to the one described in Yarullin and Serdyukov (2021). We utilize RoBERTa’s last transformer block to generate word vectors and use RoBERTa’s [CLS] token embedding as the initial hidden state for our decoder, which is a Gated Recurrent Unit (GRU) in our case. Our model also incorporates a dot-product attention mechanism between encoder and decoder. We train the final model to minimize the cross-entropy objective loss for a given x and ground-truth labels {t∗1, t∗2, . . . , t∗k} ∈ L:
LCE(θ) = − k∑
i=1
logPθ(t ∗ i |x, t∗1:i−1)
During inference, we conduct a beam search to identify candidate sequences with the lowest objective scores among the paths that conclude with the <eos> token. The beam size is set to 4.

heading: 4.1.4 Implementation Details
text: For all experiments across all methods, we use RoBERTa as the pretrained language model. The max sequence length is set to 512. For all experiments, we adopt Adam (Kingma and Ba, 2017) with the learning rate of 1e−5 and the batch size of 64 to update the model. The maximum training epoch is set to 20 and the wait patience for early stopping is set to 10. All experiments are performed with 1× 80GB NVIDIA A100 GPU.

heading: 4.2 Results

heading: 4.2.1 Performance for Each label
text: Table 1 displays F1 scores for each level-2 label across three methods at section-level crossvalidation. Method 2 outperforms the others. Labels like “Cause”, “Condition”, “Purpose”, and “Conjunction” consistently perform well across all methods. However, “Cause+Belief”, “Synchronous” and “Equivalence” consistently receive lower F1 score, indicating they are more challenging. Method 3 lags on “Manner” and “Substitution”, often missing “Manner” in multi-label instances and falsely predicting “Substitution”.

heading: 4.2.2 Count of Predicted Labels
text: Table 2 displays the distribution of the number of the predicted labels for examples with one or two gold labels across the three methods. We did not impose a limit on the number of predicted labels. However, none of the examples received more than two labels for any method, likely due to the data not containing examples with more than two labels. Analyzing the table, we find that distinguishing one or two labels is challenging, as over half of multi-label examples receive only one label, while more than 5% of single-label examples get two labels. Method 2 tends to assign two labels more often. Method 3 consistently assigns at least one label to all examples, aligning with PDTB’s typical one or two annotated labels per example.

heading: 4.2.3 Predictions for multi-label examples
text: Table 3 evaluates the performance of the three methods on multi-label examples, namely those examples with two labels. The methods can get: both labels correct, one label correct, both labels incorrect, or make no prediction. Method 1 slightly lags in “both labels correct”, but excels in minimizing “both labels incorrect” cases. Method 2 performs
best in “both labels correct” but has more “both labels incorrect” instances, indicating more entirely incorrect predictions. Method 3 has a higher “both labels incorrect” count, but it can consistently predict labels for all instances.

heading: 4.3 Multi-Label vs. Single-Label Prediction:
text: A Comparative Performance Analysis
To compare multi-label and single-label prediction methods, we evaluated Method 2 under singlelabel criteria. In this evaluation, the highest probability label is chosen, and for multi-label examples, we consider it correct if the predicted label matches one of the gold labels. We did not evaluate the single-label prediction method in terms of multi-label criteria since it is not feasible when the method inherently provides only a single prediction. If we use the top two largest probabilities for the single prediction methods, it would be assumed that all instances has two labels.
We utilize RoBERTa to obtain the [CLS] representation for each example for both single label prediction method and Method 2, but Method 2 uses separate classification heads for binary classification per class, while single-label classification
employs a multi-class mapping layer, with a size of Rh×|C|. Specifically, num_class takes the value of 14 here as the number of the labels is 14. Training loss for both methods is softmax cross-entropy. We use both RoBERTabase and RoBERTalarge for comparisons.
The results in Table 4 indicate that, while the evaluation method for both single classification methods and multi-label classification methods is the same, based on the single-label evaluation criteria, the multi-label prediction method outperforms the single-label prediction method for both RoBERTabase and RoBERTalarge. This suggests that multi-label prediction does not compromise the performance of single-label prediction.
It should also be noted that multi-label classification methods do not necessarily increase computational complexity, since they reduce model redundancy. That is, instead of creating a separate training example for each sense of a multi-label token, a multi-label approach relies on a single model that can predict multiple labels simultaneously. Moreover, multi-label methods are not necessarily more complicated, using (in our experiments) no more computational resources than the single-label prediction method.

heading: 5 Fine-grained Analysis of the Experimental results

heading: 5.1 The Multi-label Classification Method Can Capture the Label Correlations
text: In Figure 1, two co-occurrence matrices visualize the joint distribution of label pairs in the dataset and in the prediction. The upper one is the cooccurrence of label pairs in the dataset and the lower one is the co-occurrence of the predicted label pairs. The gold label pairs are the multi-label data in the dataset.
For the predicted label pairs, all the predictions which have two labels on the test sets over 12- fold are used. Darker shades represent a higher frequency or probability of the label pairs cooccurring. In Method 2, for each label, the model independently predicts whether the label is the gold label by doing a binary classification. Despite that, the two matrices look very similar, which indicates that the model seems to have implicitly captured the correlations between the labels from the data.
However, in Table 3, we have knew that method 2 predict two labels correctly for only 42% of multilabel examples. This implies that the model’s capacity to capture label correlations does not inherently guarantee very high accuracy in predicting
multi-labels. For example, although the model can capture the correlations between “Purpose” and “Manner”, it cannot distinguish which cases have both labels from those cases which only have “Purpose” or “Manner”.

heading: 5.2 When Multi-label Examples Are Predicted as Single Label
text: Figure 2 presents a heatmap, which depicts instances where only one label is predicted for multilabel examples(with two annotated labels). Vertical axes correspond to gold labels, while horizontal ones correspond to predicted labels. Numbers beside the vertical axis show how many examples had two labels but received only one predicted label and the number of such label pairs in the dataset.
Challenge in “Purpose&Manner” vs. “Purpose” Figure 2 reveals around a quarter of instances that should have both “Purpose” and “Manner” labels are only labeled as“Purpose”. Moreover, approximately one-third of instances labeled as “Purpose” are predicted as both “Purpose” and “Manner”.
The following are two examples for “Purpose&Manner” vs. “Purpose” :
(1) [they exercise]1 [to lose weight.]2. Labels:
Purpose and Manner
In Example (1), the purpose of exercising is to “lose weight”, while the manner in which weight loss is to be achieved is through exercising. Thus, both “Purpose” and “Manner” are appropriate sense labels.
(2) [Mr. Achenbaum will work with clients]1 [to determine the mix of promotion, merchandising, publicity and other marketing outlets, and to integrate those services]2. Label: Purpose
In contrast, in Example (2), while the purpose of working with clients is to determine their service needs, the annotators appear to have decided that working with clients is not the manner by which their service needs are determined. As such, only “Purpose” was annotated as a sense label.
While one might disagree with the annotators’ labelling decisions here, the value of multi-sense prediction is to highlight potentially questionable cases that might well justify further review (Klie et al., 2023).
The two examples indicate the challenges the model faces in distinguishing between “Purpose and Manner” and “Purpose” in certain cases, despite similar linguistic structures. More work is needed to determine when an example demonstrates both ‘Purpose and Manner”versus just “Purpose”.
Challenge in rare label combinations The model encounters difficulties with infrequent label combinations, often predicting just one label or even assigning an incorrect label unrelated to the combination. PDTB exhibits imbalances not only in single-label data but also in multi-label data. Some label combinations are more prevalent than others, potentially hindering the model’s ability to generalize effectively to those less common combinations.
Challenge in identifying both elaborative and argumentative discourse relations Scholman and Demberg (2017) find that annotators often disagree on whether the relations are elaborative(illustrate / specify a situation)or argumentative(serve as an argument for a claim), and they note that the relation “Example” and the relation “Specifications” in PDTB-2 can embody both. It appears to be the case that annotators do not appear to have annotated similar relations consistently in the PDTB-3. That is, Figure 1 shows that there are some instances of cooccurrence, but it is only a small fraction of cases
where multiple labels might be justified. In the PDTB annotation, the annotators are not required to annotate both elaborative and argumentative relations simultaneously, which could be causing confusion for models in classification tasks. Therefore, it would probably be worth adding this multi-label annotation to the corpus.

heading: 6 Other Investigations

heading: 6.1 Replacing Cross Entropy Loss with Focal Loss
text: Focal Loss, an adaptation of Cross-Entropy Loss (CE), addresses class imbalance by emphasizing challenging examples. Initially designed for object detection in computer vision by (Lin et al., 2017), it has been applied in recent NLP studies (Tan et al., 2022; Wang et al., 2022). Our dataset, PDTB-3, exhibits imbalances in single-label and multi-label data. While single-label tasks often use standard cross-entropy loss for IDRR, we explored focal loss for the multi-label classification in IDRR.
Focal Loss reduces weights for well-classified instances and accentuates challenging ones, modulating the loss for confidently predicted instances. For positive samples (y = 1), it is defined as::
The focal loss function is defined as follows: for
positive samples (y = 1):
Lfl(p) = (1− p)γ log(p) (1)
For negative samples (y = 0):
Lfl(p) = p γ log(1− p) (2)
Here, γ serves as the focusing parameter, controlling the rate at which easy instances are downweighted. We set it to 1 for positive samples, and 4 for negative.
In our implementation, we replaced the crossentropy loss in Method 2 with the aforementioned focal loss function without further modifications. Table 5 presents the outcomes of employing both loss functions, demonstrating that the adoption of focal loss enhances the overall performance when applied to Method 2 in the context of IDRR.

heading: 6.2 Replacing Section-level Cross Validation with Example-Level
text: Splitting the data at section-level may not be optimal for multi-label classification, since the multilabel examples are limited in number and are not evenly distributed across sections. Therefore, alternative strategies can be considered.
We tried an example-level method to offer a better mix of examples to train more robust models, especially when dealing with sparse labels. We first separated the multi-label data from the single-label data. Then, we divided the multi-label data into 12 portions, and the single-label data was also divided into 12 portions, with each portion having the same proportion of the number of each label. Next, we combined one of the 12 portions of multi-label data with one of the 12 portions of single-label data to obtain a merged set of 12 data portions. Finally, we randomly selected one portion as the test set, another portion as the validation set, and the remaining 10 portions as the training set, thus creating 12 folds of cross-validation data.
Table 6 shows the results for three methods where the cross-validation is done at example-level. We can compare Table 6 with Table 1. Method 2 stands out with the highest total F1 score in both cross-validation approaches. Method 2 also shows less variability for example-level. However, we leave it to future work to decide which crossvalidation strategy should be followed.

heading: 7 Discussions: Key Insights and Future Study
text: Embracing Multi-Label Classification in Discourse Relation Recognition This study not only identifies the limitations of single-label prediction but also explores various multi-label classification methods. The results demonstrate that multi-label classification methods can effectively predict mul-
tiple labels and capture label correlations. Interestingly, the straightforward multi-label classification approach outperforms single-label prediction under single-label criteria. We advocate for the broader use and further development of multi-label prediction methods in this domain, supported by our motivational insights and the practical results we have obtained. Navigating the Challenges of Multi-Label Scenarios Compared to traditional single-label predictions, treating discourse relation recognition as a multi-label prediction task introduces more challenges to this task.
Firstly, the model must accurately determine whether an example should be treated as a singlelabel or multi-label instance; for instance, distinguishing between cases like “Purpose and Manner” and “Purpose” alone. Furthermore, the model needs to more effectively capture and exploit the inter-label relationships to enhance multi-label classification performance. For example, it could be better that the model can identify both elaborative and argumentative relations for certain examples, rather than disregarding one in favor of the other. Thirdly, the issue of data imbalance persists not only in single-label data but also in multi-label data, making it more challenging to address the problem of imbalanced data. Diversifying and Expanding Multi-Label Discourse Datasets Based on our experimental analysis, we have observed that the model often struggles to accurately predict certain rare combinations of labels. The availability of annotated multi-label examples in PDTB-3 is limited, with some label pairs being exceptionally infrequent in this dataset.
Moreover, PDTB primarily consists of news discourse, while it has been noted that the text genre significantly impacts the distribution of discourse relations(Scholman et al., 2022a). By confining the dataset to news texts, there is a missed opportunity to comprehensively understand and model discourse relations and label associations across a wider range of text types.
Additionally, Rohde et al. (2018) pointed out that while other works on discourse coherence acknowledge the possibility of multiple relations between discourse segments, they typically lack annotation for more than one discourse relation. For example, Rhetorical Structure Theory (RST) allows for multiple possible analyses but tends to select one that aligns with the writer’s goals (Mann et al., 1989).
Therefore, the need for more diverse and abundant multi-label data is crucial to enable systems to learn more effectively and comprehensively.
Exploring Advanced Modeling Techniques In this study, we applied three foundational multilabel classification methods to IDRR, and we have found that the focal loss is a superior choice for the multi-label classification method in this task. Nevertheless, addressing the complexities of discourse relations, data imbalance, and the limited availability of annotated multi-discourse relations,etc. requires more advanced customized modeling techniques.
Extend the Multi-label Classification Methods to Other Types of Discourse Relations In addition to annotating multiple sense relations holding in the PDTB when no discourse connective was present, annotators could also record a distinct implicit relation holding in the context of an explicit connective that doe not itself convey that relation ((Webber et al., 2019) Section 3.4). This approach allowed them to represent relations inferred not just from explicit connectives but also from the underlying arguments themselves. While previous work on discourse relation recognition has ignored such cases, it would be valuable to test our multi-label classification methods on these cases as well.
In summary, the application of multi-label methods to implicit discourse relation recognition is valuable and feasible, yet there is great room for further improvement both in terms of data and methodology.

heading: 8 Conclusions
text: We conducted a comprehensive exploration of implicit discourse relation recognition (IDRR) as a multi-label classification problem, addressing realworld complexity and label interdependence. We found that multi-label approaches can have better performance than traditional single-label metrics when evaluated on the criteria for the single-label prediction.
Our exploration of multi-label frameworks, loss function optimization, and cross-validation strategies sets the stage for future research advancements. Additionally, our detailed analysis can provide valuable insights for further investigations in this field.

heading: Acknowledgments
text: This work was supported in part by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1), the University of Edinburgh. The authors also gratefully acknowledge the University of Edinburgh and Huawei Joint Laboratory for their support.
Limitations
In our work, we only used PDTB-3 as our evaluation dataset, despite the scarcity of datasets annotating multiple discourse relations. However, some other datasets such as GUM (Zeldes, 2017) and DiscoGem (Scholman et al., 2022b) probably can be considered for our evaluation and analysis. DiscoGem (Scholman et al., 2022b) can be
considered to see whether it either agree with or contradict multi-sense PDTB-3 sense annotation, although DiscoGeM is not inherently a multi-label dataset but rather a collection with diverse annotations stemming from varying perspectives.

heading: A Dataset Statistics
text: Table 7 and Table 8 give the statistics of the single-label data and multi-label data. Both distributions are very imbalanced. For Table 8, the counts for Level-2 label pairs vary widely, ranging from as low as 4 instances to as high as 378 instances. Some Level-2 label pairs, such as “Purpose/Manner” and “Cause/Level-of-detail”, have relatively high counts, suggesting a significant presence of these combinations in the dataset. Others, like “Asynchronous/Contrast” and “Concession/Asynchronous”, have lower counts, indicating less frequent occurrences of these specific pairs. In summary, the distribution of discourse relations and the label pairs in the PDTB 3.0 dataset varies, with some categories and pairs being more prevalent than others.

heading: B More Results for Three Methods
text: Table 9 to Table 11 are the results for three methods in Macro-F1 scores, Precision, Recall and Hamming loss.
In terms of Hamming loss, method 2 is the best, with a lowest Hamming loss of 5.70%. For all methods, the label “Purpose” consistently shows the highest F1 scores, with precision and recall rates exceeding 92%, indicating exceptional accuracy and consistency in identifying purpose relations. This uniformity suggests that all three methods are particularly adept at recognizing purpose-related discourse. For Method 1, most labels have a stan-
dard deviation in F1 score, recall, and precision under 10.
However, some labels like “Manner” and “Substitution” show higher variability, indicating inconsistency in model performance for these relations. With regard to Method 2, “Manner” with a relatively high standard deviation in both precision and recall suggests variability in model performance across different folds. Method 3 shows lower precision of 15.06 ± 6.09, contrasting with a much higher recall of 55.07 ± 12.14. This stark difference implies challenges in accurately pinpointing “Manner” discourse relations. The “Equivalence” label also poses a challenge across methods, with notably poor performance in Method 1, reflecting difficulties in reliably identifying equivalence relations. Furthermore, labels like “Purpose”, “Condition”, and “Asynchronous” demonstrate lower standard deviations, indicating a more stable and consistent performance in these areas.

heading: C Performance Variation Across Different Folds for Method 2
text: Table 9 to Table 11 indicates a high degree of variability across folds in Precision, Recall or F1 scores. We can have a closer examination of individual
labels for the Table 10. For instance, “Manner” exhibits high Recall (67.77 ± 9.88) but lower Precision (53.36 ± 15.81) and F1 score (58.60 ± 13.47), suggesting it identifies “Manner” instances well but has over-generalized Manner.“Synchronous” consistently scores low across all metrics, indicating difficulty in identifying synchronous relationships. Labels like “Equivalence”, “Condition”, “Substitutions”, and “Manner” have substantial standard deviations in Precision, with ±25.37, ±9.78 , ±10.45 and ±15.81 respectively, highlighting varying model performance across different test sets.
We thoroughly reviewed data and predictions for certain labels in each fold of our model’s performance evaluation. For instance, in Fold 1, “Equivalence” achieved a perfect Precision of 100% as the model correctly identified two true “Equivalence” instances. However, in other folds, Precision for
“Equivalence” ranges from 10% to 30%, indicating significant variability in predictive accuracy.

heading: D When Two labels Are Given to the Single-label Examples
text: Figure 3 illustrates predictions of two labels for examples manually labeled with a single label. Vertical axes correspond to gold labels, while horizontal ones correspond to predicted labels. Numbers beside the vertical axis indicate the number of examples with that label and how many were predicted to have additional labels.
The figure reveals that the majority of labels predicted as two for single-labeled instances are relatively low for most labels. However,about 1/3 examples which only have the label "Purpose" are given the label “Purpose” and the label “Manner” simultaneously by the model. In addition, among
the 218 examples whose label is “Manner”, the model give the additional label “Cause” or “Purpose” for 30 examples. Besides, when the model give additional labels to those examples which are only annotated with “Cause”, the labels are often under the Expansion category.
These observations echo our earlier analysis, suggesting that distinguishing between single and dual labels poses challenges for models, particularly concerning “Purpose”&“Manner’ and “Purpose”. Additionally, models occasionally predict both elaborative and argumentative relations simultaneously although only one relation (elaborative or argumentative) is annotated, underscoring the potential value of incorporating multi-label annota-
tion into the corpus.

heading: E More Results on Using Focal Loss
text: Table 12 demonstrates the results of cross-entropy and focal loss methods using RoBERTab. The results also show that focal loss could be better than cross-entropy for this task due to the data imbalance.
Upon careful examination of Table 5 alongside this table, it becomes evident that focal loss exhibits advantages, particularly for labels with less annotated data such as “Condition”, “Equivalence” and “Synchronous”. Nevertheless, it’s essential to acknowledge that while focal loss demonstrates
usefulness for certain labels, its cannot improve performance across all labels . Consequently, future work can explore alternative methodologies to further enhance performance.

heading: Future Work
text: Consequently, future work can explore alternative methodologies to further enhance performance