Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation
Year: 2023

Abstract:
Implicit discourse relation classification is a challenging task due to the absence of discourse connectives. To overcome this issue, we design an end-to-end neural model to explicitly generate discourse connectives for the task, inspired by the annotation process of PDTB. Specifically, our model jointly learns to generate discourse connectives between arguments and predict discourse relations based on the arguments and the generated connectives. To prevent our relation classifier from being misled by poor connectives generated at the early stage of training while alleviating the discrepancy between training and inference, we adopt Scheduled Sampling to the joint learning. We evaluate our method on three benchmarks, PDTB 2.0, PDTB 3.0, and PCC. Results show that our joint model significantly outperforms various baselines on three datasets, demonstrating its superiority for the task.

heading: 1 Introduction
text: Discourse relations, such as Cause and Contrast, describe the logical relation between two text spans (Pitler et al., 2009). Recognizing discourse relations is beneficial for various NLP tasks, including coherence modeling (Lin et al., 2011), reading comprehension (Mihaylov and Frank, 2019), argumentation mining (Habernal and Gurevych, 2017; Hewett et al., 2019), and machine translation (Meyer, 2015; Longyue, 2019).
Discourse connectives (e.g., but, as a result) are words or phrases that signal the presence of a discourse relation (Pitler and Nenkova, 2009). They can be explicit, as in (1), or implicit, as in (2):
(1) [I refused to pay the cobbler the full $95]Arg1 because [he did poor work.]Arg2
(2) [They put the treasury secretary back on the board.]Arg1 (Implicit=However) [There is doubt that the change would accomplish much.]Arg2
When discourse connectives are explicitly present between arguments, classifying the sense of a discourse relation is straightforward. For example, Pitler and Nenkova (2009) proved that using only connectives in a text as features, the accuracy of 4-way explicit discourse relation classification on PDTB 2.0 can reach 85.8%. However, for implicit cases, there are no connectives to explicitly mark discourse relations, which makes implicit discourse relation classification challenging (Zhou et al., 2010; Shi et al., 2017). Existing work attempts to perform implicit discourse relation classification directly from arguments. They range from designing linguistically informed features from arguments (Lin et al., 2009; Pitler et al., 2009) to modeling interaction between arguments using neural networks (Lei et al., 2017; Guo et al., 2018). Despite their impressive performance, the absence of explicit discourse connectives makes the prediction extremely hard and hinders further improvement (Lin et al., 2014; Qin et al., 2017).
The huge performance gap between explicit and implicit classification (85.8% vs. 57.6%) (Liu and Li, 2016) motivates recent studies to utilize implicit connectives for the training process of implicit relation classifiers. For instance, Qin et al. (2017) developed an adversarial model to transfer knowledge from the model supplied with implicit connectives to the model without such information, while Kishimoto et al. (2020) proposed a multi-task learning framework to incorporate implicit connectives prediction as another training objective. However, we argue that these methods are suboptimal since connectives are still not explicitly present in input texts. This is demonstrated by Kishimoto et al. (2020), concluding that adding implicit connective prediction as a training objective provides only negligible gain for implicit relation classification on PDTB 2.0 (we empirically found the conclusion also held on the adversarial model).
In this paper, we design a novel end-to-end
15696
model to leverage discourse connectives for the task of implicit discourse relation classification. The key inspiration is derived from the annotation process of implicit discourse relations in PDTB, which consists of inserting a connective that best conveys the inferred relation, and annotating the relation label based on both the inserted implicit connectives and contextual semantics (Prasad et al., 2008). We imitate this process by explicitly generating discourse connectives for the implicit relation classifier. Specifically, our model jointly learns to generate discourse connectives between arguments and predict discourse relations based on the arguments and the generated connectives. A potential drawback of this joint model is that the poorly generated connectives at the early stage of joint training may mislead the relation classifier. One possible solution is always feeding true connectives to the implicit relation classifier for training. But it leads to severe discrepancies between training and inference (Sporleder and Lascarides, 2008), since manually-annotated connectives are unavailable during evaluation (Prasad et al., 2008). To address this issue, we adopt Scheduled Sampling (Bengio et al., 2015) into our method. To be more specific, our relation classifier is first trained with hand-annotated implicit connectives and then gradually shifts to use generated connectives.
We evaluate our model1 on two English corpora, PDTB 2.0 (Prasad et al., 2008), PDTB 3.0 (Webber et al., 2019), and a German corpus, PCC (Bourgonje and Stede, 2020), and compare it with other connective-enhanced approaches and existing stateof-the-art works. Results show that our method significantly outperforms those connective-enhanced baselines on three datasets while offering comparable performance to existing sota models.
In addition, we perform the first systematic analysis of different connective-enhanced models to investigate why our method works better. Our studies show that: (1) models learn to use connectives more effectively when putting connectives in the input rather than using them as training objectives; (2) end-to-end training can improve models’ robustness to incorrectly-predicted connectives; (3) our method shows a better balance between arguments and connectives for relation prediction than other baselines. Finally, we show that connectives can effectively improve the predictive performance on frequent relations while failing on those with
1https://github.com/liuwei1206/ConnRel
limited training instances.

heading: 2 Related Work
text: Implicit discourse relation classification, as a challenging part of shallow discourse parsing, has drawn much attention since the release of PDTB 2.0 (Prasad et al., 2008). Most of the work focused on predicting implicit relations directly from input arguments. For example, early statistical methods have put much effort into designing linguistically informed features from arguments (Pitler et al., 2009; Pitler and Nenkova, 2009; Lin et al., 2009; Rutherford and Xue, 2014). More recently, neural networks (Zhang et al., 2015; Kishimoto et al., 2018; Liu et al., 2020; Wu et al., 2022; Long and Webber, 2022) have been applied to learning useful semantic and syntactic information from arguments due to their strength in representation learning. Despite achieving impressive results, the absence of connectives makes their performance still lag far behind explicit discourse parsing.
The question of how to leverage discourse connectives for implicit discourse relation classification has received continued research attention. Zhou et al. (2010) proposed a pipeline method to investigate the benefits of connectives recovered from an n-gram language model for implicit relation recognition. Their results show that using recovered connectives as features can achieve comparable performance to a strong baseline. This pipeline-based method is further improved by following efforts, including integrating pre-trained models (Kurfalı and Östling, 2021; Jiang et al., 2021) and using prompt strategies (Xiang et al., 2022; Zhou et al., 2022). However, some works (Qin et al., 2017; Xiang and Wang, 2023) pointed out that pipeline methods suffer cascading errors. Recent studies have shifted to using end-to-end neural networks. Qin et al. (2017) proposed a feature imitation framework in which an implicit relation network is driven to learn from another neural network with access to connectives. Shi and Demberg (2019) designed an encoder-decoder model that generates implicit connectives from texts and learns a relation classifier using the representation of the encoder. Kishimoto et al. (2020) investigated a multi-task learning approach to predict connectives and discourse relations simultaneously. Our method is in line with those recent approaches exploiting connectives with an end-to-end neural network. The main difference is that those models
focus on using implicit connectives in a non-input manner (i.e. they do not input implicit connectives as features but utilize them as another training signal), whereas our method explicitly generates connectives and inputs both arguments and the generated connectives into the relation classifier.
Our method can be viewed as a joint learning framework. Such a framework has been used to learn information exchange and reduce error propagation between related tasks (Zhang, 2018). Collobert et al. (2011) designed a unified neural model to perform tagging, chunking, and NER jointly. Søgaard and Goldberg (2016) refined this unified framework by putting low-level tasks supervised at lower layers. Miwa and Bansal (2016) presented an LSTM-based model to extract entities and the relations between them. Strubell et al. (2018) proposed a joint model for semantic role labeling (SRL), in which dependency parsing results were used to guide the attention module in the SRL task. Compared with these works, our joint learning framework is different in both motivation and design. For example, instead of simply sharing an encoder between tasks, we input the results of connective generation into the relation classifier.

heading: 3 Method
text: Inspired by the annotation process of PDTB, we explicitly generate discourse connectives for implicit relation classification. Following previous work (Lin et al., 2009), we use the gold standard arguments and focus on relation prediction. Figure 1 shows the overall architecture of our proposed model. It consists of two components: (1) generat-
ing a discourse connective between arguments; (2) predicting discourse relation based on arguments and the generated connective. In this section, we describe these two components in detail and show the challenges during training and our solutions.
Formally, let X1 = {x1, ..., xn} and X2 = {xn+1, ..., xn+m} be the two input arguments (Arg1 and Arg2) of implicit relation classification, where xi denotes the i-th word in Arg1 and xn+j denotes the j-th word in Arg2. We denote the relation between those two arguments as y. Similar to the setup in existing connective enhanced methods, each training sample (X1, X2, c, y) also includes an annotated implicit connective c that best expresses the relation. During the evaluation, only arguments (X1, X2) are available to the model.

heading: 3.1 Connective Generation
text: Connective generation aims to generate a discourse connective between two arguments (shown in the left part of Figure 1). We achieve this by using bidirectional masked language models (Devlin et al., 2019), such as RoBERTa. Specifically, we insert a [MASK] token between two arguments and generate a connective on the masked position.
Given a pair of arguments Arg1 and Arg2, we first concatenate a [CLS] token, argument Arg1, a [MASK] token, argument Arg2, and a [SEP] token into X̃ = {[CLS] X1 [MASK] X2 [SEP]}. For each token x̃i in X̃ , we convert it into the vector space by adding token, segment, and position embeddings, thus yielding input embeddings E ∈ R(n+m+3)×d, where d is the hidden size. Then we input E into L stacked Transformer
blocks, and each Transformer layer acts as follows:
G = LN(H l−1 +MHAttn(H l−1))
H l = LN(G+ FFN(G)) (1)
where H l denotes the output of the l-th layer and H0 = E; LN is layer normalization; MHAttn is the multi-head attention mechanism; FFN is a twolayer feed-forward network with ReLU as hidden activation function. To generate a connective on the masked position, we feed the hidden state of the [MASK] token after L Transformer layers into a language model head (LMHead):
pc = LMHead(hL[MASK]) (2)
where pc denotes the probabilities over the whole connective vocabulary. However, a normal LMHead can only generate one word without the capacity to generate multi-word connectives, such as "for instance". To overcome this shortcoming, we create several special tokens in LMHead’s vocabulary to represent those multi-word connectives, and initialize their embedding with the average embedding of the contained single words. Taking "for instance" as an example, we create a token [for_instance] and set its embedding as Average(embed("for"), embed("instance")).
We choose cross-entropy as loss function for the connective generation module:
LConn = − N∑
i=0
CN∑
j=0
Cij log(P c ij) (3)
where Ci is the annotated implicit connective of the i-th sample with a one-hot scheme, CN is the total number of connectives.

heading: 3.2 Relation Classification
text: The goal of relation classification is to predict the implicit relation between arguments. Typically, it is solved using only arguments as input (Zhang et al., 2015; Kishimoto et al., 2018). In this work, we propose to predict implicit relations based on both input arguments and the generated connectives (shown in the right part of Figure 1).
First, we need to obtain a connective from the connective generation module. A straightforward way to do so is to apply the arg max operation on the probabilities output by LMHead, i.e. Conn = arg max(pc). However, it is a non-differentiable process, which means the training signal of relation classification can not be propagated back to adjust the parameters of the connective generation
module. Hence, we adopt the Gumbel-Softmax technique (Jang et al., 2017) for the task. The Gumbel-Softmax technique has been shown to be an effective approximation to the discrete variable (Shi et al., 2021). Therefore, we use
g = − log(− log(ξ)), ξ ∼ U(0, 1)
ci = exp((log(pci ) + gi)/τ)∑ j exp((log(p c j) + gj)/τ)
(4)
as the approximation of the one-hot vector of the generated connective on the masked position (denoted as Conn in Figure 1), where g is the Gumbel distribution, U is the uniform distribution, pci is the probability of i-th connective output by the LMHead, τ ∈ (0,∞) is a temperature parameter.
After we have obtained the generated connective "Conn", we concatenate it with arguments and construct a new input as X̄ = {[CLS] X1 Conn X2 [SEP]}. This new form of input is precisely the same as the input in explicit discourse relation classification. We argue that the key to fully using connectives is to insert them into the input texts instead of treating them simply as a training objective. Like the connective generation module, we feed X̄ into an Embedding layer and L stacked Transformer blocks. Note that we share the Embedding Layer and Transformers between connective generation and relation classification modules. Doing so can not only reduce the total memory for training the model but also prompt the interaction between two tasks. Finally, we feed the outputs of the L-th Transformer at [CLS] position to a relation classification layer:
pr = softmax(Wrh L [CLS] + br) (5)
where Wr and br are learnable parameters. Similarly, we use cross-entropy for training, and the loss is formulated as:
LRel = − N∑
i=0
RN∑
j=0
Yij log(P r ij) (6)
where Yi is the ground truth relation of the i-th sample with a one-hot scheme, RN is the total number of relations.

heading: 3.3 Training and Evaluation
text: To jointly train those two modules, we use a multitask loss:
L = LConn + LRel (7)
A potential issue of this training is that poorly generated connectives at an early stage of joint training
Algorithm 1 Scheduled Sampling in Training Input: relation classifier RelCls, arguments
X1,X2, annotated connective true_conn, generated connective gene_conn, training step t, hyperparameter in decay k Output: logits 1: p = random() ▷ [0.0, 1.0) 2: ϵt = k k+exp(t/k)
3: if p < ϵt then 4: logits = RelCls(X1,X2, true_conn) 5: else 6: logits = RelCls(X1,X2, gene_conn) 7: end if
may mislead the relation classifier. One possible solution is always providing manually annotated implicit connectives to the relation classifier, similar to Teacher Forcing (Ranzato et al., 2016). But this might lead to a severe discrepancy between training and inference since manually annotated connectives are not available during inference. We solve those issues by introducing Scheduled Sampling (Bengio et al., 2015) into our method. Scheduled Sampling is designed to sample tokens between gold references and model predictions with a scheduled probability in seq2seq models. We adopt it into our training by sampling between manuallyannotated and the generated connectives. Specifically, we use the inverse sigmoid decay (Bengio et al., 2015), in which probability of sampling manually annotated connectives at the t-th training step is calculated as follows:
ϵt = k
k + exp(t/k) (8)
where k ≥ 1 is a hyper-parameter to control the convergence speed. In the beginning, training is similar to Teacher Forcing due to ϵt ≈ 1. As the training step t increases, the relation classifier gradually uses more generated connectives, and eventually uses only generated ones (identical to the evaluation setting) when ϵt ≈ 0. We show the sampling process during training in Algorithm 1.
During inference, we generate a connective Conn through arg max(pc), feed the generated Conn and arguments into the relation classifier, and choose the relation type that possesses the maximum value in pr.

heading: 4 Experiments
text: We carry out a set of experiments to investigate the effectiveness of our method across different cor-
pora and dataset splittings. In addition, we perform analyses showing that our model learns a better balance between using connectives and arguments than baselines.

heading: 4.1 Experimental Settings
text: Datasets. We evaluate our model on two English corpora, PDTB 2.0 (Prasad et al., 2008), PDTB 3.0 (Webber et al., 2019), and a German corpus, PCC (Bourgonje and Stede, 2020). In PDTB, instances are annotated with senses from a three-level sense hierarchy. We follow previous works (Ji and Eisenstein, 2015; Kim et al., 2020) to use top-level 4-way and second-level 11-way classification for PDTB 2.0, and top-level 4-way and second-level 14-way for PDTB 3.0. As for the dataset split, we adopt two different settings for both PDTB 2.0 and PDTB 3.0. The first one is proposed by Ji and Eisenstein (2015), where sections 2-20, sections 0-1, and sections 21-22 are used as training, development, and test set. The second one is called section-level cross-validation (Kim et al., 2020), in which 25 sections are divided into 12 folds with 2 validation, 2 test, and 21 training sections. There are over one hundred connectives in PDTB (e.g., 102 in PDTB 2.0), but some rarely occur (e.g., only 7 for "next" in PDTB 2.0). To reduce the complexity of connective generation and ensure each connective has sufficient training data, we only consider connectives with a frequency of at least 100 in the experiments. PCC is a German corpus following the annotation guidelines of PDTB. For this corpus, we only use the second-level 8-way classification since the distribution of top-level relations is highly uneven (Bourgonje, 2021). A more detailed description and statistics of the datasets are given in Appendix A. Implementation Details. We implement our model using the Pytorch library. The bidirectional masked language model used in our work is RoBERTabase, which is initialized with the pretrained checkpoint from Huggingface. For hyperparameter configurations, we mainly follow the settings in RoBERTa (Liu et al., 2019). We use the AdamW optimizer with an initial learning rate of 1e-5, a batch size of 16, and a maximum epoch number of 10 for training. Considering the training variability in PDTB, we report the mean performance of 5 random restarts for the "Ji" splits and that of the section-level cross-validation (Xval) like Kim et al. (2020). For PCC, we conduct a 5-fold
cross-validation (Xval) on this corpus due to its limited number of data. We use standard accuracy (Acc, %) and F1-macro (F1, %) as evaluation metrics. We show more detailed settings and hyperparameters in Appendix B. Baselines. To demonstrate the effectiveness of our model, we compare it with state-of-the-art connective-enhanced methods and several variants of our model:
• RoBERTa. Finetune RoBERTa for implicit relation classification. Only arguments (Arg1, Arg2) are input for training without using any implicit discourse connective information.
• RoBERTaConn. A variant of the RoBERTa baseline. During training, we feed both arguments and annotated connectives, i.e., (Arg1, Arg2, true_conn), to RoBERTa. During inference, only arguments (Arg1, Arg2) are input to the model.
• Adversarial. An adversarial-based connective enhanced method (Qin et al., 2017), in which an implicit relation network is driven to learn from another neural network with access to connectives. We replace its encoder with RoBERTabase for a fair comparison.
• Multi-Task. A multi-task framework for implicit relation classification (Kishimoto et al., 2020), in which connective prediction is introduced as another training task. We equip it with the same RoBERTabase as our method.
• Pipeline. A pipeline variant of our method, in which we first train a connective generation model, then learn a relation classifier with arguments and the generated connectives. Note that these two modules are trained separately.
Further, we compare our method against previous state-of-the-art models on each corpus.

heading: 4.2 Overall Results
text: PDTB 2.0. Table 1 shows the experimental results on PDTB 2.0. RoBERTaConn shows a much worse performance than the RoBERTa baseline on this corpus, indicating that simply feeding annotated connectives to the model causes a severe discrepancy between training and evaluation. This is also somewhat in accord with Sporleder and Lascarides (2008), which shows that models trained on explicitly-marked examples generalize poorly to implicit relation identification. Discourse connective-enhanced models, including Adversarial, Multi-Task, Pipeline and Our Method, achieve better performance than the RoBERTa baseline. This demonstrates that utilizing the annotated connectives information for training is beneficial for implicit relation classification. The improvement of Adversarial and Multi-task over the RoBERTa baseline is limited and unstable. We argue this is because they do not exploit connectives in the way of input features but treat them as training objectives, thus limiting connectives’ contributions to implicit relation classification. Pipeline also shows limited performance gain over the baseline. We speculate that this is due to its pipeline setting (i.e. connective generation → relation classification), which propagates errors in connective generation to relation classification (Qin et al., 2017). Compared to the above connectiveenhanced models, our method’s improvement over the RoBERTa baseline is bigger, which suggests that our approach is more efficient in utilizing connectives. To further show the efficiency of our method, we compare it against previous state-of-
the-art models on PDTB 2.0 (Liu et al., 2020; Kim et al., 2020; Wu et al., 2022; Zhou et al., 2022; Long and Webber, 2022). The first block of Table 1 shows the results of those models, from which we observe that our model outperforms most of them, especially on accuracy, achieving the best results on this corpus. The only exception is that the F1-score of our method lags behind Long and Webber (2022), particularly on level2 classification. This is because our method cannot predict several fine-grained relations (see Section 4.4), such as Comparison.Concession, which leads to the low averaged F1 at the label-level. PDTB 3.0 / PCC. Results on PDTB 3.0 and PCC are shown in Table 2. Similar to the results on the PDTB 2.0 corpus, simply feeding connectives for training (RoBERTaConn) hurts the performance, especially on the Level2 classification of PDTB 3.0. Adversarial and Multi-Task perform better than the RoBERTa baseline, although their improvement is limited. Despite suffering cascading errors, Pipeline shows comparative and even better results than Adversarial and Multi-Task on the two corpora. This indicates the advantage of utilizing connectives as input features rather than a training objective, particularly on PCC. Consistent with the results on PDTB 2.0, our method outperforms Adversarial, Multi-task, and Pipeline on both datasets, demonstrating the superiority of inputting connectives to the relation classifier in an end-toend manner and also showing that it works well on different languages. We further compare our method with three existing sota models on PDTB 3.0, Kim et al. (2020), Xiang et al. (2022), and Long and Webber (2022). Results in Table 2 show that our approach performs better than these three models.

heading: 4.3 Performance Analysis
text: To figure out why our model works well, we first perform analyses on its behavior answering two
questions: (1) whether it really benefits from discourse connectives; (2) whether it can also make correct predictions when connectives are missing. We then investigate the relation classifier’s performance in the different models when connectives are correctly and incorrectly generated (or predicted).
We perform the first analysis by replacing the generated connectives in our model with manuallyannotated ones2, and compare its performance before and after this setup. Intuitively, if our model benefits from discourse connectives, accuracy and F1-macro should increase after the change. For comparison, we apply the same setup to other connective-enhanced models. We conduct experiments3 on the Level1 classification of PDTB 2.0 (Ji split), and show the accuracy results in Figure 2. As expected, our model’s performance shows a substantial improvement, demonstrating that it does learn to use discourse connectives for implicit relation classification. Other connective-enhanced models also perform better in such a setup but with
2In PDTB 2.0 and PDTB 3.0, each instance contains annotated implicit connectives, making this analysis possible.
3We show more detailed results and also case studies in Appendix C.
a different degree of gain. Specifically, models that use connectives as input features during training (RoBERTaConn, Pipeline, and Our Method) show more increase and have higher upper bounds than models that use connectives as training objectives (Adversarial and Multi-Task). This aligns with our assumption that putting connectives in the input is more efficient for a model learning to use discourse connectives for implicit relation classification than treating them as training objectives. However, inputting connectives for training can lead to another severe issue, i.e., the model relies too much on connectives for prediction. For instance, the RoBERTaConn’s performance will drop from 96.69% to 55.34% when manually-annotated connectives are not available.
To probe whether our model suffers such an issue, we perform the second analysis by removing the generated connectives in our model and observing changes in its performance. The same setting is applied to Pipeline for comparison. Figure 3 shows the Level1 classification results3 on PDTB 2.0 (Ji split). Both models see a performance drop but still outperform RoBERTaConn. This is because these two models’ relation classifiers input the generated connectives rather than the annotated ones for training, alleviating their reliance on connectives. The decrease of Our Method (74.59% → 72.27%) is much smaller than that of Pipeline (71.01% → 58.15%). We speculate that the end-to-end training enables our model to learn a good balance between arguments and discourse connectives for relation classification. By contrast, Pipeline fails to do so due to the separate training of connectives generation and relation classification.
Finally, we show in Table 3 the results of relation classifiers in Multi-Task, Pipeline, and Our
method4 on PDTB 2.0 when connectives are correctly and incorrectly generated or predicted. Note that these three models’ results are not directly comparable in the correct and incorrect groups since their predictions on connectives are different3 (not overlap). To solve this, we calculate the performance gain of each model over the RoBERTa baseline and compare them from the gain perspective. When connectives are correctly generated, Pipeline and Our Model outperform the RoBERTa baseline by more than 10% in accuracy, while Multitask’s improvement is only 6.9%. This suggests that Pipeline and Our Model utilize connectives more efficiently than Multi-Task. On the other hand, when the connectives’ prediction is incorrect, Pipeline’s performance is worse than the RoBERTa baseline by 1.64%. Compared to it, Multi-task and Our Method achieve comparable performance to RoBERTa, showing good robustness when exposed to incorrect connectives. Despite achieving better results than baselines in both groups, our model performs significantly worse in the incorrect connective group than in the correct one. This indicates that its major performance bottleneck originates from the incorrectly generated connectives. A possible improvement is first pre-training our model on a large explicit connectives corpus, like Sileo et al. (2019). By doing so, the connective generation module may generate more correct connectives, thus improving classification performance, which we leave for future work.

heading: 4.4 Relation Analysis
text: We investigate which relations benefit from the joint training of connective generation and relation classification and compare it with other baselines. Table 4 shows different models’ F1-score for each second-level sense of PDTB 2.0 (Ji split). Generally, connectives benefit the prediction of most
4This analysis is not performed on other models (e.g., Adversarial) because they don’t generate or predict connectives.
relation types, especially in Multi-Task, Pipeline, and Our Method. For example, these three models outperform the RoBERTa baseline by more than 4% in the F1-score on the Contingency.Cause relation. On some relations, such as Expansion.Instantiation, connective-enhanced models show different tendencies, with some experiencing improvement while others drop. Surprisingly, all models fail to predict Temporal.Synchrony, Contingency.Pragmatic cause, and Comparison.Concession despite using manually-annotated connectives during training. We speculate this is caused by their limited number of training instances, making models tend to predict other frequent labels. One feasible solution to this issue is Contrastive Learning (Chen et al., 2020), which has been shown to improve the predictive performance of these three relations (Long and Webber, 2022). We leave integrating Contrastive Learning with our method to future work.

heading: 4.5 Ablation Study
text: We conduct ablation studies to evaluate the effectiveness of Scheduled Sampling (SS) and the Connective generation loss LConn. To this end, we test the performance of our method by first removing SS and then removing LConn. Note that removing LConn means that our whole model is trained with only gradients from LRel.
Table 5 shows the Level1 classification results on PDTB 2.0 and PDTB 3.0 (Ji split). We can ob-
serve from the table that eliminating any of them would hurt the performance, showing their essential to achieve good performance. Surprisingly, our model training with only LRel performs much better than the RoBERTa baseline. This indicates that the performance gain of our full model comes not only from the training signals provided by manually-annotated connectives but also from its well-designed structure inspired by PDTB’s annotation (i.e. the connective generation module and relation prediction module).

heading: 5 Conclusion
text: In this paper, we propose a novel connectiveenhanced method for implicit relation classification, inspired by the annotation of PDTB. We introduce several key techniques to efficiently train our model in an end-to-end manner. Experiments on three benchmarks demonstrate that our method consistently outperforms various baseline models. Analyses of the models’ behavior show that our approach can learn a good balance between using arguments and connectives for implicit discourse relation prediction.

heading: 6 Limitations
text: Despite achieving good performance, there are some limitations in our study. The first is how to handle ambiguous instances in the corpus. 3.45% of the implicit data in PDTB 2.0 and 5% in PDTB 3.0 contains more than one label. Currently, we follow previous work and simply use the first label for training. But there might be a better solution to handle those cases. Another is the required time for training. To mimic the annotation process of PDTB, our model needs to pass through the embedding layer and transformers twice, so it takes more
time to train than the RoBERTa baseline. However, our training time is shorter than Pipeline and Adversarial due to those two models’ pipeline setup and adversarial training strategy. Also, note that our method has a similar number of parameters to the RoBERTa baseline since we share embedding layers and transformers between the connection generation and relation classification modules in our approach. Therefore, the memory required to train our model is not much different from that required to train the RoBERTa baseline.

heading: Acknowledgements
text: The authors would like to thank the three anonymous reviewers for their comments. We also thank Xiyan Fu for her valuable feedback on earlier drafts of this paper. This work has been funded by the Klaus Tschira Foundation, Heidelberg, Germany. The first author has been supported by a Heidelberg Institute for Theoretical Studies Ph.D. scholarship.

heading: A Data Description
text: The Penn Discourse TreeBank (PDTB) is the most common corpus for the task of implicit discourse relation classification. The annotation of this corpus follows a specific strategy, which consists of inserting a connective that best conveys the inferred relation, and annotating the relation label based on both the inserted implicit connectives and contextual semantics. Prasad et al. (2008) claimed that this annotation strategy significantly improves the inter-annotator agreement. PDTB has two widely used versions, PDTB 2.0 (Prasad et al., 2008) and PDTB 3.0 (Webber et al., 2019). In both versions, instances are annotated with senses5 from a threelevel sense hierarchy. We follow previous work (Ji and Eisenstein, 2015; Kim et al., 2020) to use top-level 4-way and second-level 11-way classification for PDTB 2.0, and top-level 4-way and second-level 14-way for PDTB 3.0, and show these relations in Table 6. We show the statistics information of Ji and Eisenstein (2015) and Kim et al. (2020) in Tables 7 and 8, respectively.
The Potsdam Commentary Corpus (PCC) is a German corpus constructed following the annotation guideline of PDTB (Bourgonje and Stede, 2020). In this dataset, relations are also organized
5Some instances in PDTB have more than one label. We follow previous work to use the first label for training. While evaluating, a prediction is regarded as correct if it matches one of the annotated labels (Xue et al., 2016).
in a three-level hierarchy structure. However, this corpus is relatively small, containing only 905 implicit data, and the distribution of its relations is highly uneven, especially the top-level relations. For example, the "Expansion" (540) and "Contingency" (246) account for more than 86% of the data among all top-level relations. Bourgonje (2021) concluded that two of four relations were never predicted in his classifier due to the highly uneven distribution of the top-level relation data. Therefore, we only use the second-level relations in our experiments. Furthermore, we use a similar setup to PDTBs for PCC, considering only relations whose frequency is not too low (over 10 in our setting). The final PCC used in our experiments contains 891 data covering 8 relations (shown in Table 9). As for connectives, here, we only consider connectives with a frequency of at least 5 due to the limited size of this corpus.
B Implementation Details
Table 10 shows the hyperparameter values for our model, most of which follow the default settings of RoBERTa (Liu et al., 2019). The value of temperature τ adopts from the default setting in Gumbel-Softmax. The k in inverse sigmoid decay is set to 100 for PDTB 2.0, 200 for PDTB 3.0, and 10 for PCC. We use different k for the three datasets because of their different sizes, and bigger datasets are assigned larger values. For a fair comparison, we equip baseline models with the same
RoBERTabase 67 as our method and apply the same experimental settings (e.g. GPU, optimizer, learning rate, batch size, etc.) to them. For baselines that contain model-specific hyperparameters, such as the adversarial model (Qin et al., 2017), we follow their default setting described in the paper.
Considering the variability of training on PDTB, we report the mean performance of 5 random restarts for the "Ji" split (Ji and Eisenstein, 2015) and that of section-level cross-validation (Xval) like Kim et al. (2020). For PCC, we perform a 5-fold cross-validation on this corpus due to its limited number of data and report the mean results. We conduct all experiments on a single Tesla P40 GPU with 24GB memory. It takes about 110 minutes to train our model on every fold of PDTB 2.0, 150 minutes on every fold of PDTB 3.0, and 5 minutes on every fold of PCC.
For evaluation, we follow previous work (Ji and Eisenstein, 2015) to use accuracy (Acc, %) and F1-macro (F1, %) as metrics in our experiments.
6English version of RoBERTa-Base: https:// huggingface.co/roberta-base
7German version: https://huggingface.co/ benjamin/roberta-base-wechsel-german

heading: C Performance Analysis
text: Table 11 shows the Level1 classification results on PDTB 2.0 (Ji split) when manually-annotated connectives are fed to connective-enhanced models. Note that for models that do not use generated connectives, we insert the true connectives into their input in this setup. We also show the F1 results in Figure 4.
Table 12 shows the Level1 classification results on PDTB 2.0 when the generated connectives are removed from the inputs of relation classifiers in Pipeline and Our Method. This setting is not applied to other baselines, such as Multi-Task, because they either don’t generate connectives or don’t input the generated connectives into the relation classifiers. We also show the F1 results in Figure 5.
We investigate relation classifiers’ performance of Multi-Task, Pipeline, and Our Model when connectives are correctly and incorrectly generated
(or predicted). Other baselines, such as Adversarial, are not included in this analysis because they don’t predict or generate connectives. We mentioned in Section 4.3 that Multi-task, Pipeline, and Our Model’s prediction on connective are different. Specifically, their predictions do not overlap and show different performances, with a mean accuracy of 31.30%, 33.21%, and 32.83% for Multi-Task, Pipeline, and Our Model, on PDTB 2.0, respectively.
Here, we show both good and bad cases of all models from correct and incorrect connective prediction groups in Figure 6. For comparison, we also show results from the RoBERTa and Adversarial baselines. In the first example, connective enhanced models, including Adversarial, MultiTask, Pipeline, and Our Model, make the correct prediction on implicit relation with the help of connective information, while the RoBERTa baseline gives the wrong prediction. In the second example, Multi-Task, Pipeline, and Our Model all make the correct prediction on connectives. However, only the latter two correctly predict the implicit relations. We speculate this is because treating connectives as training objectives can not make full use of connectives. In the third example, all three models incorrectly predict the connective as "However". As a result, Pipeline incorrectly predicts the relation as "Comparison" due to the connective "However". Compared to it, both Multi-Task and Our
Model correctly predict the relation "Expansion", showing better robustness. In the fourth example, all three models predict the connective as "Specifically", which is wrong but semantically similar to the manually-annotated connective "In particular". Consequently, those models all correctly predict the relation as "Expansion". In the final example, Multi-Task, Pipeline, and Our Model wrongly predict the connective as "In fact", "For example", and "And", respectively. And all three models are misled by the incorrect connectives, predicting the relation as "Expansion".
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In Section 6.
7 A2. Did you discuss any potential risks of your work? Our paper is an entirely technical work. We don’t think it has any risk of bias or otherwise.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In Abstract section and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: In Section 4.1.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 4.1.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets and tools we use are allowed for research purposes. For example, we are the member of LDC, so we can use the PDTB dataset for research.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Because the artifacts we used are produced for research purpose.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Those corpora are extracted from news domain, and have been widely used in the field for a long time. We don’t think it contains any offensive content.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Appendix A
C 3 Did you run computational experiments?

heading: In Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In Section 4.1 and Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In Appendix B.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section 4.2, 4.3, 4.4.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave integrating Contrastive Learning with our method to future work