Implicit Sense-labeled Connective Recognition as Text Generation
Authors: Yui Oka
Tsutomu Hirao
Venue: EMNLP 2023 Findings (2023)
Keywords: Implicit Discourse Relation Recognition
Implicit Sense-labeled Connective Recognition
Encoder-Decoder
PDTB-3.0

Abstract: Implicit Discourse Relation Recognition (IDRR) involves identifying the sense label of an implicit connective between adjacent text spans. 
This has traditionally been approached as a classification task. 
However, some downstream tasks require more than just a sense label as well as the specific connective used.
This paper presents Implicit Sense-labeled Connective Recognition (ISCR), which identifies the implicit connectives and their sense labels between adjacent text spans. 
ISCR can be treated as a classification task, but a large number of potential categories, sense labels, and uneven distribution of instances among them make this difficult. 
Instead, this paper handles the task as a text-generation task, using an encoder-decoder model to generate both connectives and their sense labels. 
Here, we explore a classification method and three kinds of text-generation methods.
From our evaluation results on PDTB-3.0, we found that our method outperforms the conventional classification-based method.

Implicit Sense-labeled Connective Recognition as Text Generation
Yui Oka and Tsutomu Hirao
NTT Communication Science Laboratories, NTT Corporation [yai.oka, tsutomu. (sutomu.hirao)eee hirttaeettttooom
Abstract
Comparison. Concession
Implicit Discourse Relation Recognition (IDRR) involves identifying the sense label of an implicit connective between adjacent text spans. This has traditionally been approached classification task. However, some downstream tasks require more than just sense label and the specific connective used. This paper presents Implicit Sense-labeled Connective Recognition (ISCR), which identifies the implicit connectives well as their sense labels between adjacent text spans. ISCR be treated as classification task, but it's actually difficult due to the large number potential categories, the use of labels, and the uneven distribution of instances among them. Accordingly, this paper instead handles ISCR as eext-eeneration task, using encoder-decoder model generate both connectives and their sense labels. Here, we explore classification method and three types of lext-generation methods. From our evaluation results on PDTB-3.0, we found that our classification method outperforms the conventional classification- based method.
American Brands "ist differett (however)
aRRtoach savz. Connective (Implicit)
Theik spumarh didn'
Argument-1
Argument2
Figure I: Example of IDRR. The discourse relation between Arg and Arg2 is Compar momparison... con.Concsssion The corresponding connective provided by the annotator however.
ies focused on identifying implicit discourse rela- tions rather than implicit connectives. However, for downstream tasks such as machine translation, may be necessary to handle both discourse rela- tions and connectives (Meyer and Popescu-Bed 2012; Li et al., 2014; Yang and Cardie, 2014). Here, connectives and discourse relations do not neces- sarily involve a one-to-one correspondence. When only discourse relation is given, we must consider multiple candidate connectives corresponding to it. Furthermore, the candidate connective may already correspond to another discourse relation. There- fore, when exploiting IDRR in machine translation, incorrect translation may result from the failure to perform disambiguation of the connectives. This problem can also occur when only a single connec- tive is provided. Figure 2 shows examples of such issues. This limitation has motivated us to identify both the connectives and their discourse relations. or sense-labeled connectives.
1 Introduction
Discourse relations are often given adjacent text spans, such as clauses and sentences, without ex- plicit connectives. Recognizing the implicit dis- course relations (I.e., Implicit Discourse Relation Recognition (IDRR)) has been addressed as the main task of shallow discourse parsing. Figure shows an example of IDDR. PDTB-3.0 (Bon- nie et al., 2019), a standard benchmark dataset for IDRR, 4 coarse- -rarse-grained and 22 fine-grained discourse relations are defined between two text spans, referred to Argl and Arg2. Recent stud- ies have exploited pre-trained language models as vector representations for Argl and Arg2. In those cases, a model predicts label of discourse relation by classifying the obtained vectors into classifi- cation layers. other words, the previous stud-
This paper addresses the task of predicting not only implicit discourse relation between text spans but also the connective, and we refer to this as Implicit Sense-labeled Connective Recognition (ISCR). ISCR is similar to IDRR in that it is also regarded as a classification task. However, ISCR is particularly challenging because it involves many classes in combining discourse relations and con- nectives. Moreover, the uneven distribution of in- stances belonging to classes also makes this task difficult. Therefore, inspired by Li et al. (2018), we regard our classification task as a text generation

---

This game console has been developed first for the Chinese market.
-Expansion Leveooo--ee----eetee w
Cesmpporll.Aynchroome 22
This game console has been developed specifically for the Chinese market
Exxpansion.Lavello--doiii
-x Exppansion_LLvvllofdeeaa
Domestic sales have plunged 19% since the Big Three ended many of their programs Sept. 30.
*Contingency.Cauee TTempprallAsynchrooou
Figure 2: Example of translations with ambiguous discourse relation and connective. A discourse relation, Expansion.Levee Level-of--etaail corresponds to specificallyy and first, and then first also sponds to Temporal.. Asynchronous.. A connective since corresponds to Contingency.Com Cause and to Temporal Ceypprra.Asycchroooou Asynchronous.
task achieved by encoder-decoder models.
(Shi and Demberg, 2019; Liu et al., 2020b; Xiang et al., 2022; Long and Webber, 2022). However, large number of classes can lead to data sparse- ness, which is not ideal for classification-basse approaches.
We explore three types of text generation pre- diet the implicit sense-labeled connectives: (1) Generating connectives with discourse relations only, (2) Generating connectives with discourse relations between Argl and Arg2. (3) Generating connectives between Argl and Arg2 and then pre- dicting discourse relations based on a classifier us- ing the connectives. The experimental results ob- tained from PDTB-3.0 demonstrate that generation- based approaches outperform simple classifica- tion approach in terms of Accuracy, Macro-F1, and Weighted-F1.
IDRR via Connectives Prediction: Generating connectives between Arg1 and Arg2 is casy for the current pre-trained encoder-decoder model. There- fore, predicting discourse relations with a classifier after generating connectives feasible. Zhou al. (2022) proposed methods that use prompt-tuning to generate connectives and then exploit classifiers to predict discourse relations from the generated connectives. Chan et al. (2023) viewed IDRR problem of predicting the hierarchical paths of connective and discourse relation labels, they also proposed prompt-tuning method. Jiang al. (2023) proposed methods that learn the hierarchical discourse relation representations through multi- task learning and contrastive lcarning. Although they use connective prediction to identify the dis- course relation between text spans, they do not give attention to the performance of connective identification. Furthermore, discourse relations are determined by using classifier rather than using text generator. Therefore, the performance of such methods would degrade as the number of classes to predicted increases.
2 Related Work
2.1 Implicit Discourse Relation Recognition (IDRR)
Dataset: PDTB-3.0 (Bonnie et al., 2019) recent standard benchmark dataset representing discourse relations between adjacent text spans using hi- crarchical structure three levels. The top level indicates the general category, such as Comparison while the second level specifies the type, such as Concession. The third level provides further de- tails about the relation, such Arg2-as-denie A period is used to distinguish each level, and thus for PDTB labels, sequence of tokens such Comparion. Concession Arg2-as-denis used. The most appropriate connectives to link Argl and Arg2 with a specific discourse relation are given as metadata.
is
2.2 Classification as Text Generation
Simple classifier-based approaches face limitations when many classes with uneven distribution of in- stances are given. To tackle this difficulty, Li et al. (2018), Kwon et al. (2023), and Wu et al. (2021) proposed novel approach based text generation by pre-trained encoder-decoder models. For hier- archical text classification task, Kwon al. (2023) used a generation bassdd classifier to capture the la- bel hierarchy and unseen labels explicitly. Wu et al.
IDRR as Classification: IDRR involves selecting the most appropriate discourse relation between Argl and Arg2 from the pre-defined relations. This process carried out typical classification task. Usually, an end-to-end approach is adopted, where large-scale language model used to obtain vector representations of Argl and Arg2, which are then classified using a classification layer

---

(2021) viewed IDRR problem of predicting only the sequence of discourse labels. Although this method predicts sequences containing a con- nective, the connective plays only an auxiliary role in predicting discourse relations, and this method does not focus on predicting the connective.
classifier with tuple of Argl, connective, Arg2 (Figure 3, right). This is a simplified variant of prompt-based methods.
4 Experiment Setting
Dataset: We used the Penn Discourse TreeBank version 3.0 (PDTB-3.0) (Bonnie et al., 2019) to evaluate the performance of ISCR. This dataset con- tains annotated discourse relations in articles from the Wall Street Journal. While they have relation labels at three levels, this study used labels only up to the second level, following the previous IDRR studics (Ji and Eisenstein, 2015; Xiang et al., 2022; Long and Webber, 2022). addition, connective words provided by annotators were used.
Using a pre-trained encoder-decoder enables us generate class labels that cannot be found, or rare, in the training data. This is possible because the datasets used for pre-training may contain such class labels, which implies a significant benefit of using text generation as a classification technique.
3 Text Generation for ISCR
ISCR, the number of sense-labeled connectives to be predicted is significantly larger than that of IDRR, and the distribution of instances belong- ing to each class is unbalanced. The label of the ISCR task is composed of a discourse relation la- bel and connective words. Considering that it is not difficult generate connective words with the pre-trained encoder-decodee model, we employ a text generation approach following Li et al. (2018) and Kwon et al. (2023).
Here, we also used the standard train/val/test split of the dataset (Ji and Eisenstein, 2015). Train- ing data consisted of Sections 2-20, validation data used Sections 0-1, and testing used Sections 21-22. The PDTB-3.0 dataset contains 99 ISCR classes at the top level and 133 at the second level in test set, while there are only four classes at the top level and 17 at the second level for IDRR. Since ISCR predicts sense-labeled connectives, the number of classes increases, which results in unbalanced data. One possible way of improving the ISCR task is to incorporate contextual information obtained from I and Arg2 in the encoder-decoder model. Therefore, we examined the following three gener- ation methods using encoder-decoder models. Connective-Labbe Generation (CLG): Connec- tives and discourse relation labels, or 'Connective ve (Discourse Relation Label),' are generated using an encoder-decoder model such as Li al. (2018) (Figure 3, left). The reason for using the expression 'Connective (Discourse Relation Label)' is that the training data for the pre-trained encoder-decoder model contain many expressions 'some expression + (its description)." When the de- coder outputs parenthesis after particular word, we expect it to predict a description of the word. CLG plus Preceding and Following text span (CLG+PF)): Discourse relation labels and connec- tives with Argl and Arg2, 'Arg1 Connective vv (Discourse Relation Label) Arg2,' are gen- erated (Figure ??). We believe that the prediction of discourse relation labels and connectives with Argl and Arg2 is easier for encoder-decoder mod- cls.
Parameter Settings: We used BART-base (Lewis et al., 2020) as the encoder-decoder model and RoBERTa-base (Zhuang et al., 2021) as the simple classifier. For optimization, we used RAdam (Liu et al., 2020a). The learning rate for the classifier and texi-generation model was set to 1e-6, with batch size of 4. We conducted training for 30 epochs and implemented early stopping after 3. Evaluation Metrics: In IDRR, Accuracy and Macro-FI are commonly used as metrics for eval- uation. However, in ISCR, the number of sense- labeled connectives is significantly larger than that in IDRR, and the unbalanced data distribution is more pronounced. Therefore, used Weighted- F1, which is used an evaluation metric for unbal- anced data distribution.
Results and Discussion
Results of ISCR: ISCR. The Vanilla Classifier is a simple classifi- cation approach using RoBERTa (Zhuang et al.,
Table 1 shows the results for
'htips://catalog Ide www.uppeeeuuueee edw21CC51 "The original label PDTB-3 is excluded.
Classification after Connective Generation (CCG): Connectives are generated and then dis- course relation labels are predicted using a simple
*For more information on the label distributions of IDRR and ISCR, please see the Appendix.

---

Arg-1
[MASK]
Arg2
Arg-1
[MASK]
Arg2
Arg-1
[MASK]
Arg2
BART
BART
BART
RoBERTa
conn
PDTB label
Arg-1
conn
PDTB label
Arg2
Arg-1
conn
PDTB label
Arg2
PDTB label
CLG
CLG+PF
CCG
Figure Overview proposed method
vanilla classifeer
CLG+
0.5
1
3
4
7 8
10
11
12
13
14
15
16
17
18
19
20
Figure 4: Fl score (%) for each label at top level. 1. also (expansion), 2. and (expansion). 3. as result (contingency). 4. as (contingency), 5. because (contingency), 6. but (comparison), by comparison (comparison), (expansion), 9. however (comparison), 10. in fact (expansion). 11. in order (contingency), 12. in other words (expansion). 13. in particular (expansion). 14. indeed (expansion). 13. instead (expansion). 16. meanwhile (temporal), 17. so (contingency). 18. since (contingency), 19, specifically (expansion), 20. then (temporal).
instance
Model
Acc
M-FI
W-FI
Task
Acc
M-F1
W-FI
Vanilla Classifier
33.24
7.78
32.83
Vanilla Classifier
37.65
12.26
33.47
Top
CLG
38.55
16.55
35.14
CLG+PF 43.99
28.54
52.42
CLG++F
39.75
20.06
48.67
CCG
39.80
20,86
45.65
Table 3: Results for connective prediction
Vanilla Classifier
31,95
5.04
31.15
Second
CLG
36.63
12.17
33.01
CL.++FF
38.80
16.21
47.32
CCG
38.80
15.93
47.31
dicate the effectiveness of generating the connec- tive+label along with the preceding and following text spans. The difference is more enhanced for Macro-Fl and Weighted-F11 while CLG+PF and CCG show almost the same score for accuracy.
Table 1: Results for ISCR
Model
Acc
M-FI
W-FI
(Xiang et al., 2022)
64.04
56.63
Top
(Long and Webber, 2022)
75.31
70.05
To discuss the effectiveness of CLG+PF in more detail, show F1 scores of the Vanilla Classifier and CLG+PF for each label in Figure 4. Note that only labels that occur more than 20 times the test set are listed. While the Vanilla Classifier does not predict many labels, CLG+PF can predict labels most cases.
Vanilla Classifier
71.52
66.81
71.43
CLG+PF
71.66
65.74
71.61
Second
(Long and Webher, 2022)
64.68
57.62
Vanilla Classifier
61,66
50.39
60.41
CLG+PF
60.68
50.33
59.47
Table 2: Results for IDRR
Results of IDRR: Table 2 shows evaluation results for IDRR. We found that the Vanilla Classifier per- forms better than that of Xiang et al. (2022), while is inferior to the method of Long and Webber (2022). This is because Xiang et al. (2022) used BERT (Devlin et al., 2019), which seems inferior to RoBERTa. CLG+PF is also inferior to Long and Webber (2022), but it achieves comparable per- formance to the Vanilla Classifier. Furthermore,
2021) that combines individual connectives and discourse relations prediction. First, when com- paring methods with the Vanilla Classifier, our methods obtained better results for all evaluation metrics. These results suggest the effectiveness of classification as a text-generation approach. Sec- ond, when comparing our methods on Weighted- F1, CLG+PF is better than CLG. The results in-

---

CLG+PF outperformed the method of Xiang et al. (2022).
References
Webber Bonnie, Prasad Rashmi, Lee Alan, and Joshi Aravind. 2019. The penn discourse treebank 3.0 annotation manual.
Results of Connective Prediction The results of connective prediction by CLG+PF and the Vanilla Classifier are given in Table 3. CLG+PF outper- formed the Vanilla Classifier for all evaluation met- rics with large differences. These results are unsur- prising, since connective prediction is more com- plicated than IDRR due the larger number of classes.
Chunkit Chan, Xin Liu, Jiayang Cheng. Zihan Li. Yangqiu Song, Ginny Y. Wong, and Simon See. 2023. Discoprompt: Path prediction prompt tuning for im- plicit discourse relation recognition.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies. Volume (Long and Short Papers), pages Minneapolis, Minnesota. Association for Computational Linguistics.
6 Conclusion
This paper investigated ISCR as a way to predict im- plicit sense-labeled connectives between adjacent text spans. Here, we leveraged a text-generation approach, using encoder-decoder models, rather than a simple classification approach. This strategy was motivated by the large number of sense-labeled connectives and their unbalanced distribution. Ex- perimental results obtained from PDTB-3.0 show that our method is superior to the simple classifica- tion method. When generating connectives and cor- responding relation labels together with ArgI and Arg2, the proposed method achieved the best per- formance on Weighted-F1. This method obtained comparable performance in the IDRR setting and better performance for connective prediction than simple classification approach.
Yangfeng Ji and Jacob Eisenstein, 2015. One vector is not enough: Entity- augmented distributed semantics for discourse relations. Transactions of the Associa- tion for Computational Linguistics, 3:329-344.
Yuxin Jiang, Linhan Zhang, and Wei Wang. 2023. Global and local hierarchy-aware contrastive frame- work for implicit discourse relation recognition. In Findings of the Association for Computational Lin- guisties: ACL 2023, pages 8048 8064. Toronto, Canada. Association for Computational Linguistics.
Jingun Kwon, Hidetaka Kamigaito, Young-In Song, and Manabu Okumura. 2023. Hierarchical label genera- tion for text classification. In Findings of the Asso- ciation for Computational Linguistics: EACL 2023, pages 625-632, Dubrovnik, Croatia, Association for Computational Linguistics.
Limitations
Our approach is suitable only for English ISCR because it necessitates pre-trained encoder-decoder models and sufficiently large training datasets for fine-tuning the models. To apply our approach to other languages, we would need datasets in those languages that describe the correspondences be- tween connectives and their discourse relations. We assume that connectives are placed between A and Arg2, and actually there are only a few cases in English where this assumption is not sup- ported. It remains unclear whether this assumption is necessarily valid for other languages.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sqqeene for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages
pre-training
Online. Association for Computa- tional I. Linguistics.
Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014 4. Assessing the discourse factors that influence the quality of machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguisties (Volume 2: Short Papers), pages 283-288, Baltimore, Maryland. Association for Com- putational Linguistics.
Newly introduced discourse relations such as +speech and +belief in PDTB-3.0 are difficult to predict using only features obtained from text. In order to predict such discourse relations, need to obtain the implied intent and confidence level of speakers. How to acquire such information remains a significant challenge not only for shallow discourse parsing but also for NLP.
Maggic Yundi Li, Stanley Kok, and Liling Tan. 2018. Don't classify, translate: Multi-level e- commerce product categorization via machine trans- lation. CoRR, abs/18 182.04777
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020a. On the variance of the adaptive learning rate

---

and beyond. In International Conference on Learn-
3848-3858 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
ing Representations.
Xin Liu, Jiefu Ou, Yanggiu Song, and Xin Jiang. 2020b. On the importance of word and sentence represen- tation learning in implicit discourse relation clas- sification. In Proceedings of the Twenty-Ninth In- lernational Joint Conference on Artificial Intelli- gence, TJCAI-20, pages 3830-3836. International Joint Conferences on Artificial Intelligence Organi- zation. Main track.
I Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. Proceedings the 20th Chinese National Conference on Computational Linguistics, pages 1218-122 122-,, Huhhot, China. Chinese Informa- tion Processing Society of China.
Appendix
Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 10704- 10716 Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.
level
label
sample
top
comparison
154
contingency
529
expansion
643
temporal
148
second
comparison cow.ssssooonncsioonn
98
comparison.contran
54
Thomas Meyer and Andrei Popescu-Belis Belis. 2012. Us- ing sense-labeled discourse connectives for statistical machine translation. In Proceedings of the Joint Workshop on Exploiting Synergies between Informa- tion Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches Machine Translation (Hy- Tra), pages 129-138, Avignon, France. Association for Computational Linguistics.
comparison.similariyy
comparison similarity
2
contingency.causs
406
contingency.cause-tue
15
contingency.ccm ccssccceccecccccccc
4
contingency.co conniiiton
15
contingency purppps
89
conjunction
236
expansion.disjunction
2
expansion.com equivalence
apaanion.equivalenee
30
expansion.instaniattio instantiation
124
Wei Shi and Vera Demberg. 2019, Next sentence pre- diction helps implicit discourse relation classifica- tion within and across domains. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-IJCNIIP) pages 5790-5796, Hong Kong, China. Association for Computational Linguistics.
expansion.levall www.leell---dtaiii
208
expansion.manee manner
17
expansion.subsiie substitution
26
cumporal.asynchronce
105
43
Table 4: Number of samples for each label in PDTB-3.0
Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2021. A label dependence- aware sequence generation model for multi-level im- plicit discourse relation recognition. AAAI Confer- ence on Artificial Intelligence.
Wei Xiang, Bang Wang. Lu Dai, and Yijun Mo. 2022. Encoding and fusing semantic connection and lin- guistic evidence for implicit discourse relation recog- nition. In Findings of the Association for Compu- tational Linguistics: ACL 2022, pages 3247-3257, Dublin, Ireland. Association for Computational Lin- guistics.
Bishan Yang and Claire Cardie. 2014. Context-aware learning for sentence-level sentiment analysis with posterior regularization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 325-335, Baltimore, Maryland. Association for Computational Linguistics.
Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective pre- diction method for fine-graines implicit discourse relation recognition. Findings of the Association for Computational Linguistics: EMNLP 2022, pages

---

sample
because (contingency), and (expansion)
for instance (expansion)
109
specihically (expansion)
100
so (contingency)
87
in order (contingency)
86
then (temporal)
80
as result (contingency)
71
in fact (expansion)
45
but (comparison)
41
however (comparison)
40
as (contingency), in other words (expansion)
33
in particular (expansion)
32
also (expansion)
23
indeed (expansion), instead (expansion)
22
by comparison (comparison), meanwhile (temporal)
21
since (contingency), thus (contingency)
20
while (temporal)
18
although (comparison). consequently (contingency)
14
while (expansion), therefore (contingency)
13
by (expansion), meanwhile (expansion)
moreover (expansion)
accordingly (contingency), for example (expansion), in short (expansion)
as result of being (contingency), in addition (expansion), whereas (comparison) in contrast (comparison), further (expansion), subsequently (temporal)
8
7
that is (expansion), but (expansion), hy contrast (comparison). furthermore (expansion), when (contingency), previously (temporal)
6
rather (expansion), in (expansion). the whole (expansion)
as result (contingency), because (contingency),
besides (expansion), even though (comparison), nevertheless (comparison) given (contingency). similarly (expansion). thereby (expansion). when (temporal)
eventually (temporal), at that time (temporal), if it is (contingency).
if they were (contingency). ), in short (contingency). ), more specifically (expansion). before (lemporal), first (expansion), nonetheless (comparison). however (expansion). or (expansion). while (comparison), with (expansion), with the purpose of (contingency),
yet (comparison), overall (expansion), similariy (comparison), so as (contingency) as (temporal), as it turns out (expansion). such (contingency).
2
at the same time (temporal), earlier (temporal), hence (contingency), if (contingency). if one is (contingency). if they are (contingency). if you are (contingency). if you were (contingency), in return (expansion), in sum (expansion), in that (expansion), in the end (temporal), it is because (contingency), luter (temporal). likewise (expansion), next (temporal), second (expansion), separately (expansion), so (expansion). third (expansion). though (comparison). ultimately (temporal)
Table S: Each label the top level of ISCR and the number of samples for each label


Reviews:
[{"review_id": "sUuNU3cqiq", "reviewer": "Reviewer_mtYh", "summary": "", "questions": "Questions: \n- 1. I don't understand your motivation clearly, especially in lines 52-54. Why is the ambiguity of connective related to your new ISCR task? In my opinion, it should be more related to the word disambiguation task. In this case, each connective can be labeled as mean1, mean2, and so on.\n- 2. Is there any paper that proves the generation model performs better than the classification model when the number of labels increases? You mentioned that the method is inspired by paper[1][2]. But you should know those papers are working in hierarchical classification tasks, in which the correlation information between labels is helpful. They used a generation model to capture such correlation information because the fine-grained label generated at the latter steps can use the coarse label generated at the early steps. But I didn't see your clear motivation to use generation models.\n- 3. The description on lines 129-133 is misleading. These two papers address difficulties in hierarchical classification tasks where a seq2seq model can capture correlations between labels. But in lines 129-133, the author describes that they are designed for uneven distribution of instances.\n- 4. The CCG in Figure 3 first generates Conn and PDTB label then predicts the PDTB label. I am curious why you need to predict the PDTB label again after you already got it in step one.\n- 5. Is the size of 99 and 133 on lines 194 and 195 correct? Because I see there are more than 200 connectives in PDTB 3.0.\n\nReferences:\n- [1] Maggie Yundi Li, Stanley Kok, Liling Tan. Don’t Classify, Translate: Multi-Level E-Commerce Product Categorization Via Machine Translation. In CoRR, abs/1812.05774.\n- [2] Jingun Kwon, Hidetaka Kamigaito, Young-In Song, Manabu Okumura. Hierarchical Label Generation for Text Classification. In Findings of EACL 2023.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and questions, and they are exceedingly helpful to improve our paper.\n\n\n1. Our ISCR proposal is not a word disambiguation task.In a word disambiguation task, the ambiguous word is included in the input sentence, whereas in ISCR, the connective is not given as it is implicit in the input.\n\n\n2. Learning to generate connectives that naturally fall between Arg1 and Arg2 is the motivation for our method. Our goal is to use something other than a hierarchical structure. The reasons for our expectations for the generative models are described in L153-171. Please reread it, as the expected effect is different for each model.\n\n\n3. Thank you for pointing this out. Indeed, these papers approach the hierarchical label classification task using the seq2seq model. On the other hand, the distribution of labels could be more balanced at the lower levels of these hierarchical label-based classification tasks. Even in the PDTB dataset, the data is unbalanced at level 2. Therefore, these papers can be viewed as an approach to unbalanced data.\n*For reference, we describe the distribution of level 2 in the PDTB-3 training set at the end of the comments. \n\n\n4. We proposed this to investigate whether it is better to generate PDTB labels together with connectives or whether it is better to predict PDTB labels from connectives. However, we believe that learning to generate PDTB labels together with connectives will lead to disambiguation of connectives, rather than learning to generate only connectives. Therefore, we trained the generative model to output connectives and PDTB labels.\n\n\n5. Thank you for pointing this out, and we are very sorry for the misunderstanding. You are correct; the connectives listed in PDTB are without the original PDTB-3.0 labels, which were excluded in this study. The second-level results will be presented in the Appendix, and the second-label scores will be compared with the Long et al. scores, so the original PDTB3.0 labels were omitted. However, this description was not included in the main text, so it is added in the footnote.\n\n*The distribution of level 2 in the PDTB-3 training set. We could list the distribution of connectives + PDTB labels, but due to the large number of labels, we will show level-2.\n\n```\n{'expansion.conjunction': 236, 'contingency.cause.result': 210, 'contingency.cause.reason': 196, 'expansion.level-of-detail.arg2-as-detail': 191, 'expansion.instantiation.arg2-as-instance': 124, 'temporal.asynchronous.precedence': 94, 'contingency.purpose.arg2-as-goal': 89, 'comparison.concession.arg2-as-denier': 87, 'comparison.contrast': 54, 'temporal.synchronous': 43, 'expansion.equivalence': 30, 'expansion.substitution.arg2-as-subst': 26, 'expansion.level-of-detail.arg1-as-detail': 17, 'contingency.condition.arg2-as-cond': 15, 'expansion.manner.arg2-as-manner': 13, 'temporal.asynchronous.succession': 11, 'comparison.concession.arg1-as-denier': 11, 'contingency.cause+belief.result+belief': 8, 'contingency.cause+belief.reason+belief': 7, 'expansion.manner.arg1-as-manner': 4, 'contingency.cause+speechact.reason+speechact': 3, 'expansion.disjunction': 2, 'comparison.similarity': 2, 'contingency.cause+speechact.result+speechact': 1}\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The abstract of this paper is very clear.\n- The example in Figure 2 is interesting.\n- The proposed method shows better performance than the baseline.", "reasons_to_reject": "- The motivation is not very clear (see questions).\n- The setting in experiments seems not correct. For example the total number of labels in PDTB 3.0 (see questions).", "questions_to_authors": "Questions: \n- 1. I don't understand your motivation clearly, especially in lines 52-54. Why is the ambiguity of connective related to your new ISCR task? In my opinion, it should be more related to the word disambiguation task. In this case, each connective can be labeled as mean1, mean2, and so on.\n- 2. Is there any paper that proves the generation model performs better than the classification model when the number of labels increases? You mentioned that the method is inspired by paper[1][2]. But you should know those papers are working in hierarchical classification tasks, in which the correlation information between labels is helpful. They used a generation model to capture such correlation information because the fine-grained label generated at the latter steps can use the coarse label generated at the early steps. But I didn't see your clear motivation to use generation models.\n- 3. The description on lines 129-133 is misleading. These two papers address difficulties in hierarchical classification tasks where a seq2seq model can capture correlations between labels. But in lines 129-133, the author describes that they are designed for uneven distribution of instances.\n- 4. The CCG in Figure 3 first generates Conn and PDTB label then predicts the PDTB label. I am curious why you need to predict the PDTB label again after you already got it in step one.\n- 5. Is the size of 99 and 133 on lines 194 and 195 correct? Because I see there are more than 200 connectives in PDTB 3.0.\n\nReferences:\n- [1] Maggie Yundi Li, Stanley Kok, Liling Tan. Don’t Classify, Translate: Multi-Level E-Commerce Product Categorization Via Machine Translation. In CoRR, abs/1812.05774.\n- [2] Jingun Kwon, Hidetaka Kamigaito, Young-In Song, Manabu Okumura. Hierarchical Label Generation for Text Classification. In Findings of EACL 2023.\n", "paper_topic_and_main_contributions": "This paper proposes a new task called Implicit Sense-labeled Connective Recognition (ISCR). And the authors devise a generation model to solve it. My main concerns line in the motivations of this task. Also, I am not convinced the point that generation models are more suitable than classification models for uneven label distribution.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "The writing except the Abstract needs to be improved."}, "extra_text": {}}, {"review_id": "JtHXiNHoHM", "reviewer": "Reviewer_jhFv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and questions, and they are exceedingly helpful to improve our paper.\nWe will respond to your concerns.\n***\n\n- It would be more helpful to present both level1 and level2 performances in Table 2\n\nAns: Thank you for pointing that out. level-2 results are not included due to space limitations. Level-2 results will be listed in the appendix. Level-2 results showed the same trend as level-1.\n|  model |  Acc  | M-F1  | W-F1  |\n| ---- | ---- | ---- | ---- |\n|  vanilla classifier  |  61.66  |  50.39  |  60.41  |\n|  CLG+PF  |  60.68  |  50.33 8|  59.47  |\n\n***\n- Does the training set provide all possible combinations of implicit relations X connectives? if not, results/analyses on unattested relation X connective combinations would be useful\n\nAns: No, it does not.The PDTB dataset comes with implicit PDTB labels and possible connectives terms as metadata. We train on all of this metadata.It is possible to try all possible combinations and to expand the data, but observing the combination of PDTB labels and connectives, we decided it was better not to do so. For example, for 'expansion.conjunction' the following connectives are candidates.\n```\n['and', 'indeed', 'but', 'furthermore',  'first', 'second', 'third', 'meanwhile', 'however', 'moreover', 'in addition', 'besides',  'also',  'overall', 'further',  'in fact', 'as it turns out', 'separately', 'while', 'likewise', 'in return', 'similarly', 'in short', 'in other words',  'that is', 'on the whole']\n```\nIf all these possibilities are learned, they are considered noise. For example, 'first' and 'second' have different meanings. Furthermore, by PDTB label, 'expansion.disjunction' and 'expansion.equivalence' contain only 2-5 conjunctions, and the number of connectives included varies widely by PDTB label. Therefore, we do not learn all possible combinations.\n\n\n***\n- Figure 4 could include test frequencies of the top 20 connective X relation combinations\n\nAns: Thank you for your suggestion, we will add to the text about the frequency. They are as follows respectively.\n```\nalso (expansion)  23, and (expansion) 113, as a result (contingency)  71, as (contingency)  33, because (contingency) 133, but (comparison) 41, by comparison (comparison)  21, for instance (expansion) 109, however (comparison)   40, in fact (expansion) 45, in order (contingency)  86, in other words (expansion) 33, in particular (expansion)  32, indeed (expansion)  22, instead (expansion)  22, meanwhile (temporal)  21, so (contingency) 87, since (contingency)  20, specifically (expansion) 100, then (temporal) 80\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The observation that discourse relations and connectives form a many-to-many relation and the motivation to predict both in a discourse model is forward-looking.\nIn addition to reporting performances on the popular Implicit Discourse Relation Recognition (IDRR) task, this paper presents results in implicit connective prediction as well as ISCR. \nFigure 4 shows great potential for better macro-level predictions on implicit connective and relation when compared to a vanilla classifier.\n\n", "reasons_to_reject": "Even though constructing ISCR as a generation task and including both connective and sense in prediction advocates a more comprehensive discourse model, this paper's performance on the traditional IDRR task does not outperform previous SOTA.\n\n\nSome concerns:\n- It would be more helpful to present both level1 and level2 performances in Table 2\n- Does the training set provide all possible combinations of implicit relations X connectives? if not, results/analyses on unattested relation X connective combinations would be useful\n- Figure 4 could include test frequencies of the top 20 connective X relation combinations\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a generation-based encoder-decoder approach to predict both implicit discourse connectives and relations (i.e. Implicit Sense-labeled Connective Recognition, ISCR).\nThey propose 3 generation-based approaches and show that all of them perform better than a vanilla classifier on ISCR and connective prediction. \nHowever, their combined approach fails to outperform SOTA on the traditional IDRR task. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some relevant literature:\n- Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation: https://arxiv.org/abs/2306.06480\n- Analysis between connective and relation: https://aclanthology.org/2021.codi-main.7.pdf\n- Agreements between gold and predicted connectives: https://aclanthology.org/2021.unimplicit-1.1.pdf\n- Generating Discourse Connectives with Pre-trained Language Models:\nConditioning on Discourse Relations Helps Reconstruct the PDTB https://aclanthology.org/2022.sigdial-1.48.pdf\n"}}, {"review_id": "uMvu9acpsO", "reviewer": "Reviewer_ZGAK", "summary": "", "questions": "1) In my opinion, ISCR is a special case of multi-level implicit discourse relation recognition [1][2]. It would be better to discuss the key differences between them.\n[1] Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition, Findings of ACL 2023 or Earlier Version on arXiv.\n[2] A Label Dependence-Aware Sequence Generation Model for Multi-Level Implicit Discourse Relation Recognition, AAAI 2022.\n\n2) In Figure 3, CCG model is somewhat confusion because only conn is feed into the RoBERTa-based classification model.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and questions, and they are exceedingly helpful to improve our paper.\n\nFirst of all, our proposed ISCR differs from [Kwon et al. (2023)]. In [Kwon et al. (2023)], a single text was input and a generative model predicted the hierarchical labels that applied to that sentence. On the other hand, our task is to predict both the relationship between two texts and the connectives that could be inserted between them. Note that the problem setup and the generative model work differently. In the proposed method, the sequence of Arg1, connectives, PDTB labels, and Arg2 is trained to be a natural. Therefore, it differs from methods that simply generate hierarchical labels.\n\n1. Our focus is not on estimating discourse relation labels, but rather on estimating appropriate unambiguous connectives. Multilevel IDRR, on the other hand, focuses on the estimation of discourse relation labels. This is a major difference.\nIn fact, the connective prediction in [1] is 19.2 (Macro-F1*), while our method is 28.5. In other words, our method has a significant advantage over [1] in connective prediction.\nIn [2], only the PDTB-2 was evaluated, so it cannot be discussed. However, the results seem to indicate that the connective classification scores are low. Similar to [1], [2] focuses on the prediction of discourse relation labels. As for performance, [2] only reports experimental results for PDTB-2, so no comparison can be made. However, because of its focus on discourse relation label prediction, we believe that its performance on connective prediction is lower.\n\n\n2. We proposed to investigate whether it is better to generate PDTB labels with connectives or to predict PDTB labels from connectives.However, we believe that learning to generate PDTB labels along with connectives is better for disambiguating connectives than learning to generate only connectives. Therefore, we trained the generative model to output the connective along with the PDTB label.\n\n\n*PDTB dataset is unbalanced and will be discussed in Macro-F1.\n\n[1] Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition, Findings of ACL 2023 or Earlier Version on arXiv. \n\n[2] A Label Dependence-Aware Sequence Generation Model for Multi-Level Implicit Discourse Relation Recognition, AAAI 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The used generation-based models are effective on ISCR, especially when the label distribution is uneven.", "reasons_to_reject": "1) This paper is the direct application of existing methods on the similar task. The used generation-based models [Kwon et al. (2023)] are proposed for hierarchical text classification, which is the similar task as ISCR.", "questions_to_authors": "1) In my opinion, ISCR is a special case of multi-level implicit discourse relation recognition [1][2]. It would be better to discuss the key differences between them.\n[1] Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition, Findings of ACL 2023 or Earlier Version on arXiv.\n[2] A Label Dependence-Aware Sequence Generation Model for Multi-Level Implicit Discourse Relation Recognition, AAAI 2022.\n\n2) In Figure 3, CCG model is somewhat confusion because only conn is feed into the RoBERTa-based classification model.", "paper_topic_and_main_contributions": "This paper focuses on Implicit Sense-labeled Connective Recognition (ISCR), which identifies both the sense labels and connectives between adjacent text spans. Three text-generation based models are adapted to this task and achieve promising results on PDTB3.0.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the task of Implicit Sense-labeled Connective Recognition (ISCR) as a text-generation task by proposing an encoder-decoder model to generate both the connective and their sense labels while previous work has been addressing the problem as a classification task. The authors examined the relation between connectives and their corresponding sense labels and presented promising results of the proposed generation-based approach to the task. During the rebuttal period, the authors provided further details to address reviewers’ comments and questions regarding the implications of label distribution and hierarchical label classification task. 

The paper would benefit from including results from level 2 labels as well as relevant data insights that can contribute to a better understanding of the results of the task in relation to the proposed approach, as mentioned by the reviewers (jhFv and mtYh).
Decision: Accept-Findings