Enhancing Reasoning Capabilities by Instruction Learning and Chain-of-Thoughts for Implicit Discourse Relation Recognition
Authors: Yuxiang Lu
Yu Hong
Zhipang Wang
Guodong Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Implicit discourse relation recognition
Instruction learning
In-context learning
Chain-of-Thoughts

Abstract: The aim of implicit discourse relation recognition is to comprehend the sense of connection between two arguments. In this work, we present a classification method that is solely based on generative models. Our proposed approach employs a combination of instruction templates and in-context learning to refine the generative model for effectively addressing the implicit discourse relation recognition task. Furthermore, we utilize Chain-of-Thoughts to partition the inference process into a sequence of three successive stages. This strategy enables us to fully utilize the autoregressive generative model's potential for knowledge acquisition and inference, ultimately leading to enhanced performance on this natural language understanding task. The results of our experiments, evaluated on benchmark datasets PDTB 2.0, PDTB 3.0, and the CoNLL16 shared task, demonstrate superior performance compared to previous state-of-the-art models.

Enhancing Reasoning Capabilities by Instruction Learning and Chain-of-Thoughts for Implicit Discourse Relation Recognition
Yuxiang Lu, Yu Hong', Zhipang Wang, Guodong Zhou School of Computer Science and Technology, Soochow University, SuZhou, China (destinylu, tianxianer, zhipangwang)@gmaill. com, gdzhou@suda edu. com
Abstract
ArgI: Solo woodwind players have to be creative
Arg2:
they want to work lot
The aim of implicit discourse relation recog- nition comprehend the sense of connec- tion between two arguments, In this work, we present a classification method that is solely based generative models. Our proposed approach employs a combination of instruc- tion templates and in-context learning refine the generative model for effectively addressing the implicit discourse relation recognition task. Furthermore, we utilize Chain-of- Thoughts to partition the inference process into sequence three successive stages. This strategy en- ables to fully utilize the autoregressive gen- erative model's potential for knowledge acqui- sition and inference, ultimately leading to en- hanced performance this natural language understanding task. The results of our experi- ments, evaluated on benchmark datasets PDTB 2.0. PDTB 3.0, and the CoNLL16 shared task. demonstrate superior performance compared to previous state-of-the-an models.
Implicit Connective:
Relation sense: Contingency
Standard
Label?
Directly output the result
Expension
X
Chain-of-Thought
Rel?
Comn?
Label?
Imp
If
Contingancy
Figure 1: Chain-of-Thoughts.
for IDRR. As an example, generation tasks have been used as an auxiliary task (Jiang et al., 2021) and in a limited form that restricts prompt learn- ing (Zhou et al., 2022; Xiang et al., 2022b). With the emergence of large language models. there is growing interest in utilizing generative models rather than encoder-only models for NLP appli- cations. However, some studies also suggest that generic generative models do not perform as well as fine-tuning relatively small encoder-only models for NLU tasks (Qin et al., 2023). Our experiments also reveal that employing generative models to directly generate relation sense in the context of IDRR is an ineffective approach.
1 Introduction
Discourse relation recognition refers to identifying the sense of the relation between two arguments. This task is categorized into two types: explicit discourse relation recognition (EDRR) and implicit discourse relation recognition (IDRR) depending on whether explicit connectives, such as "because" and "but". are present or absent between the argu- ment pair. Our work investigates the potential of generative models and natural language generation for improving the performance of IDRR.
This work investigates how simple yet effective methods (ICOTT) can unleash the inference capabil- ities of generative models. Figure/ illustrates our approach of utilizing a thinking process to guide the model's output (COT). Specifically, low the model output the relation sense directly; compel the model first identify whether the argument pair pertains to implicit or explicit re- lation data. This approach reduces the unwanted noise generated by explicit data. Next, the model
Recognizing implicit discourse relations in- volves comprehending and examining the seman- tie connections between argument pairs. Previ- ous works have commonly employed semantic en- coding to enhance the model's classification accu- racy (Liu et al., 2020; Dou et al., 2021; Xiang et al., 2022a). Generative models have also been utilized
do not al-
Corresponning author

---

respond the same tasks. To test this approach. we devise instruction fine-tuning templates, as il- lustrated in Figure The templates provide a com- prehensive task definition for the model, enabling a deeper understanding of the task at hand. They uti- lize natural language to guide the model's thought process and restrict the format of the model's out- put to facilitate subsequent evaluation.
Instruction Prompt
Instruction
The determine conjunction connects given text fragments and idantify whether they have remporal, comparative, contingency, or extensional relationship. This analysis should consider both implicit and explicit relation sense. The expected ourput format is: sysscconuncton-rlllloonssp
In-conlext
Example
Input: Text fragmentf: "that prevents the production of pollen" Text fragmen12: "the gene can prevent plant from fertlizing itsaif"
2.2 In-context Learning
Output: muplicttthus-oontngeee
In-context learning enables a language model to grasp a task and produce answers to queries based on given illustrative examples. Essentially, it en- tails training proficient language model for esti- mating conditional probability distribution model, relative specific condition. However, we have discovered our research that providing the model with a specific number of instances during train- ing enhances its adherence to format and facilitates more effective convergence. During our experi- ments, we meticulously prepare example to rep- resent each of the four relation senses, which is visually illustrated in Appendix A.
Example 2-
Inpur Prompt
Imput: fragment:: "cArgfs" Text fragment2: "<Agg2**
Oulput: PRei>-<Commb-<Laaae
Figure 2: The full prompt template of our model.
identifies reasonable conjunction between the ar- gument pairs and bases its final inference on this analysis. To optimize the model's performance, we formulate a prefix prompt for better guidance which is in the form of instructions (I). By fine-tuning the instructions, we enhance the model's 's ability to learn and understand the task definition. Addition- ally, we employ the In-context learning (Min et al., 2022) (I) approach to provide additional examples aid the model's comprehension of the prompt. Our work makes the following contributions. (a) We use generative methods on the IDRR task and explore methods for improving the inference power of generative models. (b) We investigate the impact of instruction lcarning, in- context learning, and Chain-of Chaon-of-Thoughts (COT) on the performance of generative models. Through our exploration, we are able to identify the causes and effects of these learning methods. (c) We achieve state-of-the-art performance on all three datasets, indicating the effectiveness of our approach.
2.3 Chain-of-Thought
While regular training methods require models to tackle complex problems in a single step, people prefer an incremental approach, breaking down problems into smaller components to facilitate com- plex reasoning. This inclination towards incremen- tal thinking enables people to engage in more nu- anced and effective moblem-sslvingg Our approach presents a simple yet effective method of prompt- ing that mimics thinking process the form of natural language prompts, as shown in Figure 2. Rather than providing categorical answer directly, the model first considers whether the implied rela- tionship is explicit or implicit. It then identifies ap- propriate connectives between pairs of arguments before finally providing the answer.
Contributions:
In the autoregressive generative mode, each to- ken output by the model influenced by its prede- cessors, creating natural progression of thought, The reasoning process under standard generation prompt and with COT' is depicted in Figure 1. The latter method requires the model to provide COT before producing response. By incorporating COT into the prompting strategy, the performance
2 Apporach
2.1
Instruction Learning
The primary objective of instruction fine-tuning is to enhance the language models' capacity to re- spond to natural language instructions. The method entails utilizing supervised signals instruct lan- guage models on performing tasks described in instructions. As a result of instruction fine-tuning, language models learn to follow instructions and
'The training set of PDTB 2.0 and PDTB 3.0 lacks explicit data on implicit chapter relations. To enhance the model's 's ahility to generate the intended COT, a limited amount of explicit data was artificially incorporated.

---

PDTB 2.0
PDTB 3.0
CoNLL16
Model
Macro-F1
Acc.
Macro-FI
Ace.
Test Ace.
Blind Acc.
Baseline (roberta-baso
57.65
65.50
63.32
65.50
61.11
62.42
Baseline (roberta-large)
64.03
70.46
67.52
71.44
64.52
68.13
BMGF (Liu al., 2020)
63.39
69.06
65.15
72.21
CVAE (Dou al., 2021)
65.06
70.17
MANF (Xiang et al., 2022a)
56.63
64.04
CPrompt (Xiang al., 2022b)
70.88
75.17
PCP (roherta-base)(hhou al., 2022)
64,95
70.84
68.98
71.31
PCP (roberta-large) (Zhou al., 2022)
67.79
73.80
72.36
74.51
IICOT (flan-t5-base)
65.26
71.13
69.79
73.98
69.84
72.38
IICTT (flan-t5-large)
69.23
76.04
73.06
77.46
73.46
75.84
Table 1: Main experiments on the 4-class classification in Macro-F1 and accuracy. The highest reported results of previous works are denoted by underlines.
Model (PDTB 2.0)
Tem
Com
Con
Exp
Model
Tem
Com
Con
Exp
M-F1
BMGF (Liu et al., 2020)
50.26
59.44
60.98
77.66
Fine-tuning
57.52
61.24
75.98
78.31
67.90
JMCG (He al., 2020)
41.54
57.04
74.76
+Instruction
58.85
63.65
77.23
78.17
69.58
CVAE (Dou al., 2021)
44.01
55.72
63.39
80.34
+ICI.
58.50
62.93
77.83
77.41
69.23
PCP (Zhou et ul., 2022)
56.41
70,38
64.18
80.17
+ICL&Rel
59.36
64.28
78.19
78.03
69.54
IICOT (Ours)
56.99
71.23
68.80
81.24
+CL&Coon
59.66
69.33
78.30
78.62
72.97
Model (PDTB 3.0)
Tem
Com
Con
Exp
Conl&Conn
63.64
67.66
78.15
78.79
72.03
MANF (Xiang et al., 2022a)
42.13
35.83
63.55
70.00
IICOT (Ours)
67.63
70.65
79.78
79.71
73.06
IICOT (Ours)
67.63
70.65
79.78
79.71
Table 3: We conduct ablation experiments on the latest PDTB3.0 dataset of IDRR, evaluating the Fl and Macro- Fl metries in the binary and 4-way scenario.
Table 2: The binary classification performance on PDTB 2.0 and PDTB 3.0 benchmark datasets.
of the model improves. All the specific prompt templates we used are in the Appendix
model specifically fine-tuned for broad range of natural language processing tasks to better suit instruction learning. Remarkably, our approach outperforms current state-of taite-of-thedr tht-hhr-tt methods all datasets. Specifically, we achieve an impressive 4.62% increase in the Contingency category on PDTB 2.0, while realizing significant improve- ments in all of the categories PDTB 3.0.
3 Experiment
3.1 Experiment Settings
This study involves conducting experiments on three benchmark datasets: PDTB 2.0 (Prasad et al., 2008), PDTB 3.0 (Prasad al., 2019), and CoNLL16 (Xue et al., 2016). Notably, the CoNLL16 dataset lacks manually annotated liga- tures in 450 training data instances. To address this, we utilize the gpt-3.5-turbo model to predict the ligatures and establish them as the ground trut. Detailed statistics for each dataset and the settings of hyperparameters can be found in Appendix C. To ensure reproducibility, we will make all of our source code publicly availableh?.
3.3 Ablation Experiments
Table presents the findings of our ablation exper- iments, where Fine-tuning denotes that the input is only the argument pair while the direct output relation sense; Instruction indicates that the task definition directs the model to output labels; ICL refers In-context learning; Conn refers to pre- dicting connected words; Rel denotes predicting explicit or implicit data.
3.2 Main results
Our ablation experiments lead to four key clusions: First, using instruction and ICL improves the model's performance in comparison to directly outputting results. Providing certain amount of example enhances the model's understanding the task. Second, appending Rel or Conn to ICL
Table 1 and 2 present the results of our 4-way and binary classification performance on PDTB 2.0, PDTB 3.0, and CoNLL16 benchmark datasets. We employ the flan-t5 pre-trained model, a b5-based
Chttps://githie http://twww..cc/eeetttt/t//////t

---

representations. This allows us gain insight into the denoising mechanism of the model.
The t-SNE method is frequently preferred for visualizing high high-dimensional data, as it effectively presents local relationships and clustering struc- tures. Specifically, t-SNE is adept at capturing similarities and differences within the data. Fig- ure 3-(b) illustrates the model that incorporates Rel within the COT framework. This contrasts the COT model shown in Figure 3-(a), which does not fea- ture these judgments. The generative model, such as flan-T55 lacks dedicated token [CLS], unlike the BERT model. Consequently, in our study, we employ the encoding vector of the [BOS] token to represent the sentence after applying t-SNE dimen- sionality reduction. This approach enables examine the distribution pattern. The COT judg- ment effectively reduces noise and facilitates the acquisition of semantic knowledge pertaining to explicit data. The results demonstrate that the dif- ferent types of data are well-separated upon the inclusion of COT judgment.
Impice
Impiti
Expicit
(a) Representation distribu- tion with COT
(b) Representation distribu- tion without COT
Figure 3: The noise reduction effect of COT is demon- strated through the distribution.
predict chains before labels further improve the model's reasoning ability. Third, without ICL, adding both Rel and Conn also lead a perfor- mance improvement. This indicates that the COT approach is highly effective. Fourth, the optimal value is reached by combining the approaches men- tioned above, thereby demonstrating their individ- ual validity and mutual reinforcement.
4 Analysis
4.1
Explicit Data
Mitigating overfitting It our contention that the efficacy of COT also lies in its ability to alle- viate model overfitting. Figure presents data on training loss, loss on the development (dev) set dur- ing training, and changes in dev set performance for models trained both with and without COT. The table illustrates that abstaining from COT leads to further drop in training loss, but also shift in dev set loss from low to high, along with trend of decreasing performance. These phenomena sug- gest that the model overfits the training data. COT enhances the informational content of model outputs. Although it increases the complex- ity of the task, this additional information effec- tively guides the model towards the desired direc- tion, thereby mitigating the potential of overfitting.
Our experiment of the CoNLL16 task reveals that incorporating both Explicit and Implicit judgments into the COT further enhances model performance. Accordingly, we propose the inclusion of explicit data in the PDTB dataset to scrutinize whether the performance improves. Our experimental findings (Appendix D) demonstrate that the performance improves upon the inclusion of a limited amount of explicit data. However, with increasing amounts of explicit data, the performance deteriorates substan- tially. This is because the distribution of explicit data differs from implicit data and the introduction of more explicit data results in amplified noise.
Figure 4 and 5 demonstrate that an increase in ex- decrease in the model's accuracy in distinguishing between explicit and im- plicit data. Nonetheless, the model maintains a high accuracy rate of 96.83% and exhibits optimal performance at the 20% threshold of explicit data, These results suggest that the model remains rela- tively unaffected by noise and that successful data augmentation has been achieved.
plicit data corresponds
Conclusion
We aim enhance the reasoning capabilities of generative models in IDRR by employing gener- ation task framework and incorporating Instruction learning, in-context learning, and COT. Through our approach, we achieve a notable improvement over the baseline model, leading to state-of-the-art performance on three benchmark datasets.
4.2 How Chain-of-Thoughts works
Denoising believed that the model achieves denoising ability while generating COT. In order to demonstrate the denoising process, we analyze the output vectors of the model terms of their
In our future research, we plan to further investi- gate the utilization of generative models and even large language models. Specifically, we aim to ex- plore the efficacy of larger models, including the

---

implementation of knowledge distillation to trans- fer knowledge from large models smaller ones.
Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn discourse treebank version 3.0. I.C2011705..
Acknowledgment
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023, Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint 7xXix:3302.06477
The research is supported by National Key R&D Program of China (2020YFB13 1313601) and Na- tional Science Foundation of China (62376182. 62076174).
Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. 2022a. Encoding and fusing semantic connection and lin- guistic evidence for implicit discourse relation recog- nition. In Findings of the Association for Compu- tational Linguistics: ACL 2022, pages 3247-3257, Dublin, Ireland. Association for Computational Lin- guistics.
6 Limitations
The current design of our COT is not yet perfect, leaving ample room for improvement. Furthermore, our experimentation has been limited to the English corpus, without exploring other languages.
Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022b. ConnPrompt: Connective-clozz prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Confer- ence on Computational Linguisties, pages 902 911, Gyeongju, Republic of Korea. International Commit- tec Computational Linguistics.
References
Zujun Dou, Yu Hong, Yu Sun, and Guodong Zhou. 2021. CVAE-based re-anchoring for implicit discourse re- lation classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1275-1283, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Nianwen Xue, Hwee Tou Ng. Sameer Pradhan, Attapol Rutherford, Bonnie Webher, Chuan Wang, and Hong- min Wang. 2016. Conll 2016 shared task on multi- lingual shallow discourse parsing. In Proceedings of the CoNLL- shared task, pages 1-19.
Ruifang He, Jian Wang, Fengyu Guo, and Yugui Han. 2020. Transs-driven joint learning architecture for implicit discourse relation recognition. In Proceed- ings of the 58th Annual Meeting of the Association for Compulational Linguistics, pages 139-148.
Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt- based connective pre- diction method for fine-grained implicit discourse relation recognition. In Findings of the Asvociation for Computational Linguistics: EMNLP 2022, pages 3848 3858, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics
Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not jus classification: Recog- nizing implicit discourse relation on joint modeling of classification and generation.
Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 2418-2431, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2020. On the importance of word and sentence represen- tation learning in implicit discourse relation clas- sification. In Proceedings of the Twenty-Ninth In- ternational Joint Conference on Artificial Intelli- gence, IJCAI-20, pages 3830-3836. International Joint Conferences on Artificial Intelligence Organi- zation. Main track.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in in-context learning work? arXiv preprint 22X2 12837.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind K Joshi, and Bon- nie L Webber. 2008. The penn discourse treebank 2.0. In LREC.

---

PDTТB2.0
PDTB3.0
CoNLL16
Implicit
Tem.
Com.
Con.
Exp.
Tem.
Com.
Con.
Exp.
Tem.
Com.
Con.
Exp.
Train
704
2,104
3,622
7,394
1,515
2,044
6,481
8,660
697
1,952
3,435
7,072
Dev./Blind
54
191
287
651
136
190
579
748
30
185
274
705
Test
146
276
556
148
154
529
643
36
88
122
276
PDTB2.0
PDTB3.0
CoNLL16
Explicit
Tem.
Com.
Con.
Exp.
Tem.
Com.
Con.
Еxр.
Tem.
Com.
Con.
Exp.
Train
2,904
4,674
2,792
5,342
3,435
5.104
3,356
8,634
2,752
4,383
2,579
5,008
Dev./Blind
248
431
277
506
348
422
249
748
318
404
254
503
Test
288
366
181
450
280
488
340
836
169
192
107
212
Table 4: Descriptive statistics of implicit discourse relation instances are reported for the datasets.
A In-context Learning Examples
PDTB 2.0 Performance Variation with Explicit Data Ratio
19.13
Example 1-
02, 09.9
Argument1: there's satisfaction in going against the rules
05.37
Performance
Argument2: he means the rule that player can't an't cut it after certain age
Relation sense: Extension
Example 2-
Argumentl: that prevents the production of pollen
65
Argument2: the gene can prevent a plant from fer- tilizing itself
relation sense: Comparison
Figure 4: Experiments with adding explicit data on PDTB 2.0.
Example 3-
I : he heralded by trumpet fanfare Argument2: the judge marched down the center
PDTB 3.0 Performance Variation with Explicit Data Ratio
aisle in his flowing black robe Relarion sense: Temporal
54
Jadgmani Macro
Example 4-
92 46
$9.85
Argumentl: however, the maximum coupon at
00
85.13
which the notes can be reset is 16 1/4%
Performance
Argument2: the minimum coupon is 13 3/4%
Relarion sense: Comparison
75-
B Chain-of-Thoughts prompt
Only label:
30
40
50
60
Explict data ratio (%)
Instruction: The task is to determine whether they have a temporal, comparative, contingency, or ex- tensional relationship. This analysis should con- sider both implicit and explicit relationships.
Figure 5: Experiments with adding explicit data on PDTB 3.0.
Conn & Label:
Instruction: The task is to determine the conjunc- tion that connects two given text fragments and identify whether they have temporal, compara- tive, contingency, or extensional relationship. This analysis should consider both implicit and explicit relation sense. The expected output format is: conjuunction-relationhijj
Rel & Conn & Label:
Instruction: The task is to determine the conjunc- tion that connects two given text fragments and identify whether they have a temporal, comparative, contingency, or extensional relationship. This anal- ysis should consider both implicit and explicit rela- tion sense. The expected output format is: "type-

---

Joint Model
Joint Model
Dasseline
Baseline
40
0.4
2.0
6.0
22
8003
1000E
2030
4000
4000
80300
13006
3230
2300
4000
X30
TX00
(a) Training
(b) Evaluation loss.
(c) Macro-FI performance (%)
Figure 6: The experimental records encompass training loss, evaluation set loss, and performance metrics.
conjunction-relltionsiip,,,
C Dataset statisties and the hyperparameters
Table 4 presents the statistical information for spe- cific datasets. In our evaluation process, ex- clusively employ implicit data from the PDTB 2.0 dataset and PDTB 3.0 dataset. Conversely, for the CoNLL16 dataset, we utilize all available data in accordance with the official partitioning strategy. During our training process, we employ a batch size of 16 and set the learning rate to 5e-5. We train the AdamW optimizer for 5 epochs, utilizing the default parameter settings. Moreover, we incor- porate both warmup and linear learning rate decay strategies, with warmup ratio of 0.1.
D Add Explicit Data
To examine the impact of explicit data on model performance, we gradually augment the amount of explicit data in the training sets of PDTB 2.0 and PDTB 3.0. The experiment results are detailed in Figure and 5.
E Observation of Overfitting
The presence of overfitting can be readily observed by analyzing the training loss, evaluation set loss, and performance variation in the absence of the COT model. In contrast, the inclusion of the COT model leads to improved performance. is im- portant to highlight that the training steps differ between the two experiments due to the utilization the early stop strategy.


Reviews:
[{"review_id": "jniliGaZt7", "reviewer": "Reviewer_znNK", "summary": "", "questions": "Questions: \n- 1. I don't understand how your work deals with the explicit case during training. Did you input only the arguments of an explicit example to the model? And then also predict the relation type, connective, and label?\n- 2. I doubt your implementation of the RoBERTa baseline is correct. In papers [1][2], they reported a much better performance of the Roberta baseline than your paper. For example, the RoBERTa's performance in paper[2] for PDTB 3.0 level1 is 73.51/67.98 for Acc/F1, which is almost the same as your model's performance.\n- 3. I doubt your statement in line 83. I find paper[3] performs better than your method, especially on the F1 score. I don't think sota is necessary for this work. But your statement about sota bothers me a lot.\n- 4. In Figure 3, I find the representation without COT seems to be better separated than with COT. But your description in lines 240-242 makes me quite confused. Why do you think with COT is well-seperated?\n\nReferences:\n- [1] Yuxin Jiang, Linhan Zhang, and Wei Wang. Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition. In ACL 2023 Findings.\n- [2] Wei Liu and Michael Strube. Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation. In ACL 2023.\n- [3] Wangqiu Long, Bonnie Webber. Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations. In EMNLP 2022.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your time in reviewing our submission.\n\nWe observe that part of the questions is consistent with the comments for rejection. Therefore, we directly respond to the questions.\n\n\n**Q1:** I don't understand how your work deals with the explicit case during training. Did you input only the arguments of an explicit example to the model? And then also predict the relation type, connective, and label?\n\n\n**Answer:** Absolutely. For the explicit data, we merely input the explicitly-related arguments after removing the inner explicit connectives. Therefore, our approach is able to predict the triple of type, connective and label for COT. \n\n\n**Q2:** I doubt your implementation of the RoBERTa baseline is correct. In papers [1][2], they reported a much better performance of the Roberta baseline than your paper. For example, the RoBERTa's performance in paper[2] for PDTB 3.0 level1 is 73.51/67.98 for Acc/F1, which is almost the same as your model's performance.\n\n\n**Answer:** True. In some of the previous work, the substantially better performance of the RoBERTa baseline is reported. We feel sorry to overlook the studies in [1] and [2]. Though, we find that they are published after the deadline of EMNLP 2023. We are willing to include the citations of them in the revised paper.\n\nThere are a variety of factors causing significantly inconsistent performance of the RoBERTa baseline, such as the training environment and hyperparameter configuration. In our case, we mainly carry out experiments using flan-T5. For environmental sustainability reasons, we did not extensively fine-tune the RoBERTa baseline. Therefore, it didn’t show an optimal performance.\n\nOur baseline model is implemented using the Hugging Face trainer, and we will make all training code publicly available for reproducibility and reference.\n\n\n**Q3:** I doubt your statement in line 83. I find paper[3] performs better than your method, especially on the F1 score. I don't think sota is necessary for this work. But your statement about sota bothers me a lot.\n\n**Answer:** Indeed, our performance is lower compared to Long et al's work [3]. However, we observe that Long et al employed data augmentation. This significantly boosted their classification performance for temporal relation labels by nearly 20% compared to SOTA. We believe that the existing approaches may benefit from this attempt, and therefore, take a keen interest in adding it to our study in the extended paper. We promise to pay more attention to compare our enhanced approach with Long et al’s work, and provide sufficient discussion on the achievements.\n\n\n\n\n**Q4:** In Figure 3, I find the representation without COT seems to be better separated than with COT. But your description in lines 240-242 makes me quite confused. Why do you think with COT is well-seperated?\n\n\n**Answer:** We apologize for the oversight. The left side of Figure 3 does not have COT, while the right side does.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An interesting model for IDRR task.\n- Experimental results show the effectiveness of the method.", "reasons_to_reject": "- Some descriptions are not clear (refer to the question).\n- Experimental results of the RoBERTa baseline seems incorrect.\n- They claimed sota but actually not. ", "questions_to_authors": "Questions: \n- 1. I don't understand how your work deals with the explicit case during training. Did you input only the arguments of an explicit example to the model? And then also predict the relation type, connective, and label?\n- 2. I doubt your implementation of the RoBERTa baseline is correct. In papers [1][2], they reported a much better performance of the Roberta baseline than your paper. For example, the RoBERTa's performance in paper[2] for PDTB 3.0 level1 is 73.51/67.98 for Acc/F1, which is almost the same as your model's performance.\n- 3. I doubt your statement in line 83. I find paper[3] performs better than your method, especially on the F1 score. I don't think sota is necessary for this work. But your statement about sota bothers me a lot.\n- 4. In Figure 3, I find the representation without COT seems to be better separated than with COT. But your description in lines 240-242 makes me quite confused. Why do you think with COT is well-seperated?\n\nReferences:\n- [1] Yuxin Jiang, Linhan Zhang, and Wei Wang. Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition. In ACL 2023 Findings.\n- [2] Wei Liu and Michael Strube. Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation. In ACL 2023.\n- [3] Wangqiu Long, Bonnie Webber. Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations. In EMNLP 2022.", "paper_topic_and_main_contributions": "This paper explores a generative model for the IDRR task inspired by recent work about Instruction learning, In-context Learning, and Chain-of-thought. The authors provide extensive experiments to show the effectiveness of the method. While the general idea is intriguing, there is room for improvement in both the statements and experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Wangqiu Long, Bonnie Webber. Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations. In EMNLP 2022.", "Typos_Grammar_Style_and_Presentation_Improvements": "The writing should be improved."}, "extra_text": {}}, {"review_id": "TrSsmzhBuh", "reviewer": "Reviewer_XvkT", "summary": "", "questions": "Suggestions\n\n* More empirical datapoints\n\n1) Comparison between different generative AI models such as Llama and Flan-T5\n2) Comparison between different sizes of models such as 3B, 7B, and 11B\n\n* More analysis of evaluations\n\nWhat does it make still recent language models suffer difficulties in this task? (low performance boosting?)", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments and thoughtful suggestions.\n\n\n**Com1:** Since it is a purely empirical work employing common techniques to this task, the value of this work relies on enough empirical findings and datapoints\n\n\n**Answer:** We realize your concern and regard it as the use of scenario learning and instruction learning. Indeed, they are commonly-used techniques recently. However, we take a keen interest in exploring possibility of leveraging generative models for IDRR, where scenario learning and instruction learning are just supportive techniques. To our best knowledge, there hasn’t yet any previous work dealing with IDRR within a generative framework.\n\nIn addition, the other difference of our approach from the previous work is to bring a new COT into the denoising process. The design of COT is truly grounded on the investigation of data (positive cases and noises), though it is not tailored but on the contrary useful for dealing with similar problems. In the area of utilizing generative LLMs, the design of COT has been a hot topic, because a well-crafted COT can significantly boost the reasoning abilities of LLMs. Our study proposes a simple yet effective COT.\n\n\n**Com2:** It is interesting to see the performance boosting is not remarkable such as 1% accuracy improvement. What makes this low boosting? Is it because of the task setup or these recent language models are not enough to deal with it?\n\n\n**Answer:** It's true that the performance improvement on CoNLL is not pronounced. This attributes to two aspects as below. First, the performance of the baseline model on CoNLL has been at a considerably high level. Second, the generative models like flan-t5-large (with around 800M parameters) are not enough large to treat with the complicated problems in IDRR. In fact, Chunkit Chan's paper [1] provides comprehensive testing of ChatGPT on this task, and they prove that ChatGPT performs quite modestly. In light of this, as you suggested, we will explore larger models in our future work.\n\n**Q1:** More empirical datapoints. Comparison between different generative AI models such as Llama and Flan-T5. Comparison between different sizes of models such as 3B, 7B, and 11B\n\n\n**Answer:** We plan to conduct experiments with larger models such as llama-7b and llama-13b in the future work. However, for a fair comparison with previous work, we are currently validating our approach on the smaller models.\n\n\n**Q2:** More analysis of evaluations\n\n\n**Answer:** We appreciate the suggestion for more in-depth analysis of our evaluations. In our reviewed paper, we plan to delve deeper into the comprehensive performance analysis. This will include investigating specific error cases, identifying patterns in misclassifications, and exploring the impact of various factors on the model's predictions. By conducting a more extensive analysis, we aim to offer valuable insights into the strengths and limitations of our approach, which will contribute to a richer discussion of our results.\n\n\n**Q3:** What does it make still recent language models suffer difficulties in this task? (low performance boosting?)\n\n\n**Answer:** Good question. IDRR is that a challenging task which requires a machine to deeply understand semantics of arguments, and meanwhile sufficiently learn much richer knowledge about discourse structures. In some cases, we nearly observed that some hypothesis needs to be considered first for human to correctly determine the relations, without giving reliable clues in the texts. More importantly, the arguments are long and edited by different persons who have inconsistent writing habits and distinct desires. As a result, any subtle changes made on arguments allow most of the relation types to be explainable, from the perspectives of different readers. By contrast, the existing LLMs are more capable of generating readable texts using fluent languages, though they haven’t yet been proven to possess all-sided capacities as mentioned above, while such capacities are required in IDRR. Transfer learning and knowledge distillation may enhance LLMs, and allow them to work as the tailored versions. However, the available data is so sparse that the comprehensive capacity of IDRR cannot be developed using the data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is an interesting empirical work to guide other studies further. ", "reasons_to_reject": "* Lack of enough empirical datapoints.\n\nSince it is a purely empirical work employing common techniques to this task, the value of this work relies on enough empirical findings and datapoints\n\n* Lack of analysis for evaluations\n\nIt is interesting to see the performance boosting is not remarkable such as 1% accuracy improvement. What makes this low boosting? Is it because of the task setup or these recent language models are not enough to deal with it?", "questions_to_authors": "Suggestions\n\n* More empirical datapoints\n\n1) Comparison between different generative AI models such as Llama and Flan-T5\n2) Comparison between different sizes of models such as 3B, 7B, and 11B\n\n* More analysis of evaluations\n\nWhat does it make still recent language models suffer difficulties in this task? (low performance boosting?)", "paper_topic_and_main_contributions": "This work explores recent generative AI models on the implicit discourse parsing task. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6HiQ7tMAUY", "reviewer": "Reviewer_YJWg", "summary": "", "questions": "Which evaluation metric is depicted in Table 2?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thoughtful comments and valuable suggestions.\n\n**Com1:** The structure and the writing of some parts of the paper can be confusing. The fine-tuning / instruction learning section is not clear enough. Is the Instruction prompt used for fine-tuning the model in multiple epochs/iterations? Or is it provided at de beginning of every prompt without specific parameter fine-tuning? According to Appendix C there is a specific parameter fine-tuning, but this step is not clearly described in the paper (e.g., how is the fine-tuning task modelled. input, output, etc.).\n\n\n**Answer:** We sincerely apologize for the lack of clarity regarding the fine-tuning process in our paper. In fact, we did perform full fine-tuning on the flan-t5-base/large model using the method mentioned in the paper. As you've observed, we provide all the hyperparameter configurations and some training details in the appendix. Furthermore, we will make all training code publicly available to facilitate reproducibility.\n\n\n**Com2:** Regarding the structure, it is a bit confusing that Figures 4 and 5 are analysed in the main body of the paper, but included in the appendix. Furthermore, they are mentioned before Figure 3 in the text. Figure 3 is also slightly confusing since (a) says with COT and (b) without COT, but (a) visualisation is definitely more noisy than (b), and then in the text is stated that: \"Figure 3-(b) illustrates the model that incorporates Rel within the COT framework. This contrasts the COT model shown in Figure 3-(a), which does not feature these judgments.\" Which seems a bit contradictory with the caption of the figure.\n\n\n**Answer:** Due to the page limit, we were unable to include Figure 4 and Figure 5 in the main body of the paper. We feel sorry to bring trouble on reading. In the revised version of this paper, we will rectify this problem by incorporating the necessary figures into the main body and accompanying the discussions with them. \n\n\nFurthermore, I sincerely apologize for the error in the order of Figures 3-(a) and 3-(b). We will correct this mistake in the extended version.\n\n\n**Com3:** Finally, a concerning aspect of this work is that the authors rely on ChatGPT to annotate some missing relations in the CoNLL16 benchmark without any further human validation: \"Notably, the 148 CoNLL16 dataset lacks manually annotated ligatures in 450 training data instances. To address this, 150 we utilize the gpt-3.5-turbo model to predict the 151 ligatures and establish them as the ground truth.\"\n\n\n**Answer:** We fully understand your concern about the use of ChatGPT and human validation. We implemented strict annotation guidelines to ensure ChatGPT's responses were contextually appropriate. Specifically, we provided ChatGPT with argument pairs and relation categories. On this basis, we offered a list of 10 most commonly-used connectives per relation type for prompting ChatGPT, further constraining the responses. Following this process, we manually reviewed around 20 generated results for each relation category. Based on our observations, we deemed ChatGPT's generated connectives to be contextually appropriate.\n\nWe appreciate your question and, accordingly, will conduct a comprehensive human evaluation of these 450 training instances, and provide a direct insight into the results generated by ChatGPT.\n\n\n**Q1:** Which evaluation metric is depicted in Table 2?\n\n\n**Answer:** We sincerely apologize for the lack of clear explanation regarding the evaluation metric for Table 2 in our paper. In our paper, the performance for all binary classification tasks is measured using F1-score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method combines generative models and prompting with an approach based on the chain-of-thought which looks original and interesting. Furthermore, the reported results outperform previous work considering both, Transformer-based approaches, prompting, and original methods.", "reasons_to_reject": "The structure and the writing of some parts of the paper can be confusing. The fine-tuning / instruction learning section is not clear enough. Is the Instruction prompt used for fine-tuning the model in multiple epochs/iterations? Or is it provided at de beginning of every prompt without specific parameter fine-tuning? According to Appendix C there is a specific parameter fine-tuning, but this step is not clearly described in the paper (e.g., how is the fine-tuning task modelled. input, output, etc.).\n\nRegarding the structure, it is a bit confusing that Figures 4 and 5 are analysed in the main body of the paper, but included in the appendix. Furthermore, they are mentioned before Figure 3 in the text. Figure 3 is also slightly confusing since (a) says with COT and (b) without COT, but (a) visualisation is definitely more noisy than (b), and then in the text is stated that:\n\n\"Figure 3-(b) illustrates the model that incorporates Rel within the COT framework. This contrasts the COT model shown in Figure 3-(a), which does not feature these judgments.\"\n\nWhich seems a bit contradictory with the caption of the figure.\n\nFinally, a concerning aspect of this work is that the authors rely on ChatGPT to annotate some missing relations in the CoNLL16 benchmark without any further human validation: \"Notably, the 148 CoNLL16 dataset lacks manually annotated ligatures in 450 training data instances. To address this, 150 we utilize the gpt-3.5-turbo model to predict the 151 ligatures and establish them as the ground truth.\"", "questions_to_authors": "Which evaluation metric is depicted in Table 2?", "paper_topic_and_main_contributions": "The paper titled \"Enhancing Reasoning Capabilities by Instruction Learning and Chain-of-Thoughts for Implicit Discourse Relation Recognition\" proposes a new method to approach the Implicit Discourse Relation Recognition (IDRR) task. The authors combine the use of generative models with a chain-of-thought model to determine the existing relation between arguments in the discourse. They evaluate their models considering three benchmarks: PDTB 2.0, PTDB 3.0, and CoNLL16.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the implicit discourse relation recognition (IDRR) task, proposing a generative model that exploits instruction learning, chain-of-thought, and in-context learning. Overall, the paper does not have serious flaws. While one reviewer found this work exciting, the remaining two reviewers were not particularly excited about it.

Strengths:

The use of instruction learning and chain-of-thought for the IDRR task is new and interesting.

Extensive experiments were conducted.

Weaknesses:

The reviewers agreed that some parts of the paper were confusing (e.g., instruction learning).

There was missing related work (e.g., the Long and Webber 2022 paper).

The empirical contribution of this paper could be strengthened. For instance, there wasn't any empirical analysis of the results (e.g., the seemingly small improvements). In addition, the claims could be strengthened through experiments with different models and different model sizes. The rebuttal did not provide additional empirical results or any analysis of the results. 

The reviewers did not change their scores after reading the rebuttal and believed the paper could benefit from another round of revision.
Decision: Accept-Findings