Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition
Authors: Chenxu Wang
Ping Jian
Mu Huang
Venue: EMNLP 2023 Main (2023)
Keywords: implicit discourse relation recognition
logical semantics enhancement
prompt-based connective prediction
mutual information maximization

Abstract: Implicit Discourse Relation Recognition (IDRR), which infers discourse relations without the help of explicit connectives, is still a crucial and challenging task for discourse parsing. Recent works tend to exploit the hierarchical structure information from the annotated senses, which demonstrate enhanced discourse relation representations can be obtained by integrating sense hierarchy. Nevertheless, the performance and robustness for IDRR are significantly constrained by the availability of annotated data. Fortunately, there is a wealth of unannotated utterances with explicit connectives, that can be utilized to acquire enriched discourse relation features. In light of such motivation, we propose  a $\textbf{P}$rompt-based $\textbf{L}$ogical $\textbf{S}$emantics $\textbf{E}$nhancement (PLSE) method for IDRR. Essentially, our method seamlessly injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction. Furthermore, considering the prompt-based connective prediction exhibits local dependencies due to the deficiency  of masked language model (MLM) in capturing global semantics, we design a novel self-supervised learning objective based on mutual information maximization to derive enhanced representations of logical semantics for IDRR. Experimental results on PDTB 2.0 and CoNLL16 datasets demonstrate that our method achieves outstanding and consistent performance against the current state-of-the-art models.

Prompt-based Logical Semanties Enhancement for Implicit Discourse Relation Recognition
Chenxu Wang' Ping Jia Mu Huang'
Schooo of Computer Science and Technology, Beijing Institute of Technology, Beijing, China Beijing Engincering Research Center of High Volume Language Information Processing and Cloud Computing Applications, Beijing Institute of Technology, Beijing, China (wangchenxu, pjian}ebie edu. cn, suemh200 1eemmh@00@gmail.... @@gmail. com
Abstract
Argumentl: Art helps to get out of
Implicit Discourse Relation Recognition (IDRR), which infers discourse relations without the help of explicit connectives, still a crucial and challenging task discourse parsing. Recent works tend to exploit the hierarchical structure information from the annotated senses, which demonstrate enhanced discourse relation representations can be obtained by integrating sense hierarchy. Nevertheless, the performance and robustness for IDRR are significantly constrained by the availability of annotated data. Fortunately, there is wealth of unannotated utterances with explicit connectives, that can be utilized to acquire enriched discourse relation features. In light of such motivation, we propose Prompt- based Logical Semantics Enhancement (PLSE) method for IDRR. Essentially, our method scamlessly injects knowledge relevant to discourse relation into pre-trained language models through prompt-based connective prediction. Furthermore, considering the prompt-based connective prediction exhibits local dependencies due the deficiency of masked language model (MLM) in cap- turing global semantics, we design novel self-supervised learning objective based mutual information maximization to derive enhanced representations of logical semantics for IDRR. Exxperimental results PDTB 2.0 and CoNLL16 datasets demonstrate that our method achieves outstanding and consistent performance against the current state-of -fate-of-the-art models.'
Explicit Connective: so
Argument2: I don't keep all locked up inside
Argumentl: The fundamentals are pretty strong
Implicit Connective: so
Argument2: I don't see this a bear market at all
Relation Sense: Contingency. Cam.seeeyaauc..euut..u.
An unannotated instance with an explicit con- nective and an IDRR instance from Gigaword (Napoles al., 2012) and PDTB 2.0 (Prasad et al., 2008) respec- tively. For IDRR, the Implicit Connective is absent in the raw text, which are assigned by the annotator based on the semantic relations between Argumentl and Argu- ment2.
Figure 1:
cause). As a fundamental and crucial task in dis- course parsing, IDRR has benefited a multitude of natural language processing (NLP) downstream tasks, such as question answering (Rutherford and Xue, 2015), text summarization (Cohan et al., 2018) and event relation extraction (Tang et al., 2021). Ex- plicit discourse relation recognition (EDRR) has already incontrovertibly demonstrated the remark- able effectiveness by utilizing explicit connectives in discerning discourse relation type (Varia et al., 2019a). However, implicit discourse relation recog- nition remains a challenging and demanding task for researchers due to the absence of connectives.
Early works based on traditional machine learn- ing are dedicated to the extraction of intricate syn- tax features (Lin et al., 2009; Park and Cardie, 2012). With the rapid development of deep learn- ing, pre-trained language models (PLMs), such BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have remarkably improved the perfor- mance of IDRR. Recently, there has been a grow- ing inclination among various work exploit the lelti-level hierarchical information (an instance in- troduced in Appendix A.1) 1) derived from annotated
1 Introduction
Implicit discourse relation recognition (IDRR) fo- cuses identifying semantic relations between arguments (sentences clauses), with the ab- sence of explicit connectives (e.g., however, be-
"Corresponding author. 'Our code will he released at httpps//giithun com/ lalalamdbf atlaaammFF/LLE_... IORS

---

senses (Wu et al., 2022: Jiang et al., 2022; Long and Webber, 2022; Chan al., 2023b). Neverthe- less, such usage exhibits an excessive reliance on the annotated data, thereby restricting the poten- tial of IDRR for further expansion to some degree. Fortunately, there exists plethora of unannotated utterances containing explicit connectives that can be efffectively utilized to enhance discourse relation representations.
semantic representation (detailed in Section 3.2). This learning procedure fosters the connective- related representations to effectively capture global logical semantic information, thereby leading to enhanced representations of logical semantics.
Our work focuses on identifying implicit inter- sentential relations, thus evaluating our model on PDTB 2.0 (Prasad et al., 2008) and CoNLL16 (Xue et al., 2016) datasets. Since the latest PDTB 3.0 (Webber al., 2019) introduces numerous intra- sentential relations, there is a striking difference in the distribution between inter-sentential and intra- sentential relations reported by (Liang et al., 2020). Given this issue. we don't conduct additional ex- periments on PDTB 3.0.
As depicted in Figure 1, even in the absence of annotated sense, the explicit data can proficiently convey Contingency.Causee Resull relation, clu- cidating strong correlation between the connec- tive and the discourse relation. In expectation of knowledge transfer from explicit discourse rela- tions to implicit discourse relations, Kishimoto et al. (2020) has undertaken preliminary investi- gations into leveraging explicit data and connective prediction. This work incorporated auxiliary pre-training task named explicit connective pre- diction, which constructed an explicit connective classifier to output connectives according to the rep- resentation of the |CLSJ. Nonetheless, this method falls short in terms of intuitive connective predic- tion through the [CLS] and fails to fully harness the valuable knowledge gleaned from the masked language model (MLM) task. Besides, Zhou et al. (2022) proposed prompt-based connective pre- diction (PCP) method, but their approach relies on Promix-Prompt, which not inherently intuitive and introduces additional complexity of inference.
The main contributions of this paper are summa- rized follows:
We propose a prompt-based logical semantics en- hancement method for IDRR, which sufficiently exploits unannotated utterances with connectives to learn better discourse relation representations.
Our proposed connective prediction based on Preze-Prompt seamlessly injects knowledge re- lated to discourse relation into PLMs and bridges the gap between the pre-training and the down- stream task.
Our method, aiming at capturing global logical semantics information through MI maximization, effectively results in enhanced discourse relation representations.
In this paper, we propose a novel Prompt-based Logical Semantics Enhancement (PLSE) method for implicit discourse relation recognition. Specifi- cally, we manually design a Cloze-Prompi template for explicit connective prediction (pre-training) and implicit connective prediction (prompt-tuning). which achieves seamless and intuitive integration between the pre-training and the downstream task. On the other hand, previous work (Fu et al., 2022) has revealed that the contextualized representations learned by MLM lack the ability to capture global semantics. In order to facilitate connective pre- diction, it is important to have a comprehensive understanding of the text's global logical seman- ties. Inspired by recent unsupervised representa- tion learning with mutual information (Hjelm et al., 2019; Zhang et al.. 2020), we propose novel self- supervised learning objective that maximizes the mutual information (MI) between the connective- related representation and the global logic-related
2 Related Work
Prompt-based Learning Along with the boom- ing development of lcrge-scale PLMs like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020) and GPT3 (Brown et al.. 2020), prompt-based learning has become new paradigm in the filed of natural language process- ing (Liu al., 2021). In contrast the fine-tuning paradigm, prompt-base based methods reformulate the downstream tasks match the pre-training objec- tive. For example, Schick and Schütze (2021) con- verted a diverse set of classification problems into cloze tasks by constructing appropriate prompts and verbalizers that associate specific filled words with predicted categories.
Representation Learning with MI Methods based on mutual information have a long history in unsupervised representation learning, such as

---

connective-related representatiin
global Ingio-rdated semantic representation
8
8
MI-Max
Exp.
Temp.
but
80
and
also
then
map
Multi-Head
MLM Head
Cross-Attention
but
90
and
also
tbes
Hog
Homsi
Hug
MLM Head
8
8
8
8
8
8
8 ...
8
8
Horon
Transformers Encoder
Transformers Encoder
CLS
T
SEP]
MASKK
SEP]
222
[SEP]
CLS
SEP
[AASK]
SEP
I
SEP
Explicit Data Arg!: Art helps get out me Arg2: don't keep all locked up inside
Implicit Data
Argt: The fundatoentals pretty
Arg don't this bear market all
Connective: so
Sense Label: Contingency
(a) pre-training
(b) prompt-tuning
The overall architecture of our framework. It consists of the pre-training phase (a) and the prompt-tunie tunint phase (b). In the pre-training phase, explicit data is automatically extracted from unannotated corpus, while in the prompt-tume tuning phase, implicit data is randomly selected from the training data.
Figure 2:
the infomax principle (Linsker, 1988; Bell and Se- jnowski, 1995) and ICA algorithms (Hyvärinen and Pajunen, 1999). Alemi et al. 62016) was the first to integrate MI-related optimization into deep learn- ing models. From then on, numerous studies have demonstrated the efficacy of MI-maximization for unsupervised representation learning (Hjelm et al., 2019; Zhang et al., 2020; Kong et al., 2019). In- spired by these superior works, we explore how to derive enhanced discourse relation representations by incorporating global logical semantic informa- tion through Ml-maximization.
capitalize on the potential of data and PLMs. On the other hand, the multi-level hierarchical informa- tion has been explored by (Wu et al., 2022; Jiang et al., 2022: Long and Webber, 2022: Chan et al., 2023b). However, the efficacy of these methods remains significantly constrained by the availability of annotated data. Therefore, our work dedicated to the proficient utilization of unannotated explicit data via crompt-based connective prediction and MI Ml-maximization.
Methodology
Implicit Discourse Relation Recognition For this task, many prior works based deep neu- ral network have proposed various approaches to exploiting better semantic representation of argu- ments, such as shallow CNN (Zhang et al., 2015) and L.STM with multi-level attention (Liu and Li, 2016). Recently, PLMs have substantially im- proved the performance of IDRR by leveraging their robust contextualized representations (Shi and Demberg, 2019; Liu et al., 2020). Furthermore, Kishimoto al. (2020) proposed auxiliary pre- training task through unannotated explicit data and connective prediction. Zhou et (2022) proposed a connective prediction (PCP) method based on Prefix-rrompt. Regrettably, their approaches fail to
Figure 2 shows the overall architecture of framework.2 In the following sections, we will elaborate on our methodologies including a Cloze- Prompt template based on connective prediction, a robust pre-training approach to incorporating ex- plicit data, and an effective prompt-tuning method by leveraging implicit data.
3.1 Prompt Templatize
As Figure illustrates, the input argument pair (Argl, Arg2) is reformulated into Cloze-
'In the pre-training phase. Figure 2 does not depict the Masked Language Mask (MLM) task for universal words, yet it still remains an integral part of our framework to enhance the universal semantics of discourse relations for PLMs.

---

Prompt template by concatenating two arguments and inserting some PLM-specifie tokens such as [MASK]. [CLS] and [SEP]. The [MASK] token mainly used to predict the connective between two arguments, while the [CLS] and [SEP] tokens are inserted to indicate the beginning and ending of the input. In addition, the (SEP) tokens also serve the purpose of delincating the boundaries between two arguments.
the universal semantics of discourse relations for PLMs, we still retain the Masked Language Model (MLM) task for universal words. The cross-entropy loss for MLM is as follows:
1 CMLM N
-log Pus (axs) (2)
Emaasedd
where N and T remain consistent with Equation (I), u denotes the universal token within the argu- ment pair, and P approximates the probability of the answer word
3.2 Pre-training
In this section, will provide comprchensive in- troduction the pre-training data and tasks, includ- ing Connectives Mask (CM), Masked Language Model (MLM), Global Logical Semanties Learn- ing (GLSL).
Global Logical Semantics Learning To address the local dependencies of predicted connectives representations and capture global semantic in- formation with logical coherence, propose novel method based MI maximization learn- ing. Give an argument pair (Argl, and the Cloze-Prompt template T(.), we feed T(x) through a Transformer (Vaswani et al., 2017) en- coder to acquire contextualized token representa- tions H. Specifically, we design a Multi-Head Cross-Attention (MHCA) module to extract the global logic-related semantic representation. As illustrated in the part (a) of Figure 2, we separate H into Harg1, Hemask and Harg2, denoting as the con- textualized representations of argl, connective mask and arg2, respectively. The global logic- related representation Hclogic is calculated through MHCA module as follows:
Pre-training Data Acquiring implicit discourse relational data poses an immensely arduous un- dertaking. Fortunately, explicit discourse relation data is considerably more accessible owing to the presence of connectives. Therefore, we approx- imately collected 0.56 million explicit argument pairs from an unannotated corpus renowned as Gi- gaword (Napoles et al., 2012) by leveraging its vast collection of connectives. The collected data is divided into training and validation sets in 9:1 ratio.
Connectives Mask Based on the strong corre- lation observed between explicit connectives and discourse relations, we propose the "Connectives Mask" task as a pivotal approach to incorporat- ing discourse logical information into PLMs. This task not only involves learning the intricate rep- resentations of connectives but also serves vital bridge between the realms of the pre-training and the downstream task, that facilitates seamless knowledge transfer. For the mask strategy, we re- place the connective with (1) the [MASK] token 90 % of the time (2) the unchanged token 10% of the time. The cross-entropy loss for Connectives Mask (CM) is as follows:
Q = -Heaakk K = V = Harg1 ① Harg2
(3)
HClogic MultiHead(Q,, K, K,
(4)
The learning objective is to maximize the mutual information between the connective-related rep- resentation Hemask and the global logic-related representation HGiogie Due the notorious in- tractability of MI estimation in continuous and high- nigh-dimensional settings, we typically resort to maximizing lower bound estimators of MI. In our method, we adopt a Jensen-Shannon MI estima- tor suggested by (Hjelm et al., 2019; Zhang et al., 2020):
1 LCM N Σ logP(c:|
(1)
T.8D (HGtogie -Hemak)))))))
where N denotes the number of training examples, T embodies a prompt template designed to trans- form the input argument pair represents the connective token between the argument pair, and P estimates the probability of the answer word
Erl- sp( pppp-TT.. (HGlogic Hemask))
(5)
Epx (T. Hemask)]
where Tw discrimination function parameter- ized by neural network with learnable parame- ters W. takes a pair of the global logic-related
Masked Language Model order to enhance

---

representation HGoogi and the connective-related representation Hemask as imput and generates the corresponding score to estimate IJSD H'Ctogic is a negative example randomly sampled from distribu- tion P = IP, and sp()) = log(1 + e²) is the softplus function. The end-goal training objective is maxi- mizing the MI between Hotooic and Hemask and the loss is formulated as follows:
Top-level
Second-level
Connectives
however
Concession
Comparison
although. though but because, so, thus consequently, therefore us, since instead, rather
Contrast
Cause
Contingeney
Pragmatic cause
Altemative
and, also fact, furthermore
Conjunction
LCLSL = IJJDD (Hologio: H Hemask)
Expansion
Instantiation
instance, example finally
List
specifically
Through maximizing TJSD, The connective- related representation Hanask strongly encour- aged maintain a high level of MI with its global logic-related representation. This will foster Hemask to incorporate comprehensive logical se- mantic information, leading to enhanced discourse relation representations.
Restatement
indeed, particular
Asynchronous
then, after, before
Temporal
Synchrony
meanwhile, when
Table The verbalizer mapping connectives between implicit discourse relation labels and connectives on PDTB 2.0 dataset, consisting of four top-level and second-level senses.
The overall training goal is the combination of Connectives Mask loss, MaskedLM loss, and Global Logical Semantics Learning loss:
The cross-entropy loss for the prompt-tuning is as follows:
N N
C = + PMMM + LCLSL
L
log V(c)) (T(x))
(8)
3.3 Prompt-tuning
where N and T remain consistent with Equation
As depicted in the part (b) of Figure 2, the input argument pair x = (Argl, Arg2) and the Cloze- Prompt template T()) exhibit a remarkable consis- tency with the pre-training phase. This noteworthy alignment indicates that the prompt-tuning through connective prediction fosters the model to fully leverage the profound reservoir of discourse rela- tion knowledge acquired during the pre-true vee-rriniin phase. In order streamline the downstream task and avoid introducing additional trainable parame- ters, the MHCA module isn't 't used in the prompt- tuming phase. Besides, we construct a verbalizer mapping connectives to implicit discourse relation labels.
(I), V denotes the verbalizer mapping the connec- tive e to implicit discourse relation label I, and P estimates the probability of the gold sense label
Experiment Settings
4.1 Datasets
The Penn Discourse Treebank 2.0 (PDTB 2.0) PDTB 2.0 is large scale corpus containing 2,312 Wall Street Journal (WSJ) articles (Prasad et al., 2008), that is annotated with information relevant to discourse relation through a lexically-groundae approach. This corpus includes three levels of senses (i.c., classes, types, and sub-types). We follow the predecessors (Ji and Eisenstein. 2015) to split sections 2-20, 0-1, and 21-22 training, validation, and test sets respectively. We evalu- ate our framework on the four top-level implicit discourse classes and the 11 major second-level im- plicit discourse types by following prior works (Liu et al., 2019; Jiang et al., 2022; Long and Webber, 2022). The detailed statistics of the top-level and second-level senses are shown in Appendix A.2.
Verbalizer Construction A discrete answer space, which is subset of the PLM vocabulary, is defined for IDRR. Given that the implicit discourse relational data has already been annotated with ap- propriate connectives in PDTB 2.0 (Prasad et al., 2008) and CoNLL16 (Xue et al., 2016) datasets, we manually select set of high-frequency and low-ambiguity connectives the answer words of the corresponding discourse relations. Therefore, we construct the verbalizer mapping connectives to implicit discourse relation labels for PDTB 2.0 in Table 1, and the verbalizer for CoNLL16 is shown in Appendix A.3.
The CoNLL-2016 Shared Task (CoNLL16) The CONLL 2016 shared task (Xue et al., 2016) provides more abundant annotation for shadow dis- course parsing. This task consists of two test sets,

---

the PDTB section 23 (CoNLL-Test) and Wikinews texts (CoNLL-Blind), both annotated following the PDTB annotation guidelines. On the other hand, CoNLL16 merges several labels to annotate new senses. For instance, Peeeringenc.PPagmmaii cause is merged into Contingency. Cause.Reason to re- move the former type with very few samples. There is a comprehensive list of 14 sense classes to be classified, detailed cross-level senses as shown in Appendix A.3.3
4.3 Baselines
To validate the effectiveness of our method, we compare it with the most advanced baselines cur- rently available. Here we mainly introduce the Baselines that have emerged within the past two years.
CG-T5 (Jiang et al., 2021): joint model that recognizes the relation label and generates the tar- get sentence containing the meaning of relations simultaneously.
4.2 Implementation Details
LDSGM (Wu et al., 2022): a label dependence- aware sequence generation model, which exploits the multi-level dependence between hierarchi- cally structured labels.
In our experiments, we utilize the pre-trained RoBERTa-base (Liu et al., 2019) as our Transform- ers encoder and employ Adam W (Loshchilov and Hutter, 2017) the optimizer. The maximum in-
put sequence length is set to 256. All experiments are implemented in PyTorch framework by Hug- gingFace transformers* (Wolf et al., 2020) on one 48GB NVIDIA A6000 GPU. Next, we will pro- vide details of the pre-training and prompt-tuning settings, respectively.
PCP (Zhou et al.. 2022): connective prediction method by applying Prefix-Promee template.
GOLF (Jiang et al., 2022): contrastive learn- ing framework by incorporating the information from global and local hierarchies.
Pre-training Details During the pre-training phase, we additionally add a MHCA module and a discriminator To to facilitate MI maximization learning. The MHCA module a multi-head atten- tion network, where the number of heads is 6. The discriminator Tw consists of three feedforward lay- ers with ReLU activation, and the detailed architec- ture is depicted in Appendix A.4. We pre-train the model for two epochs, and other syyer--arammee settings include batch size of 64, warmup ratio of 0. and a learning rate of le-5.
DiscoPrompt (Chan al., 2023b): path pre- diction method that leverages the hierarchical structural information and prior knowledge of connectives.
ContrastivelDRR (Long and Webber, 2022): a contrastive learning method by incorporating the sense hierarchy and utilizing it to select the nega- tive examples.
ChatGPT (Chan et al., 2023a): a method based on ChatGPT through the utilization of an in- context learning prompt template.
Prompt-tuning Details The implementation of our code refers to OpenPrompt (Ding et al., 2022). We use Macro-Fl score and Accuracy as evaluation metric. The training is conducted with 10 epochs, which selects the model that yields the best perfor- mance on the validation set. For top-level senses on PDTB 2.0 we employ a batch size of 64 and a learning rate of le-5. For second-level senses on PDTB 2.0 and cross-level senses on CoNLL16, search
Experimental Results
5.1 Main Results
Table shows the primary experimental results of our method and other baselines on PDTB 2.0 and CoNLL16 datasets, terms of Macro-Fi score and Accuracy. Classification performance on PDTB 2.0 in terms of F1 score for the four top-level classes and 11 second-level types is shown in Table 3 and Table 4. Based on these results, we derive the following conclusions:
we conducted meticulous due to the limited number of certain senses. The bath size and learning rate are set to 16 and 1e-6, respectively.
Firstly, our PLSE method has achieved new state-of-the-art performance on PDTB 2.0 and CoNLL16 datasets. Specifically, our method ex- hibits a notable advancement of 1.8% top-level FI, 3.35% top-level accuracy, 0.45% second-level FI,
Some instances PDTB 2.0 and 16 datasets are annotated with multiple senses. Following previous works, we treat them as separate examples during training, At test time. a prediction is considered correct when it aligns with one the ground-trutt truth labels.
"ttpps: / /ttg///huu..o//ggggiiiieee com/hhugiingffcc//rrasffoom

---

PDTB--Tpp
PDTTB-Second
CoNLL-Test
CoNLL-Blind
Model
F1
Acc.
F1
Acc.
F1 Acc.
F1
Acc.
MANN (Lan al., 2017)
47.80
57.39
40.91
40.12
RWP-CNN (Varia et al., 2019b)
50.20
59.13
39.39
39.36
BERT-FT (Kishimoto et al., 2020)
58.48
65.26
54.32
BMGF-RoBERTa (Liu et al., 2020)
63.39
69.06
35.25
58.13
40.68
57.26
28.98
55.19
CG-T5 (Tiang al., 2021)
57.18
65.54
37.76
53.13
LDSGM (Wu et al., 2022)
63.73
71.18
40.49
60.33
PCP (Zhou al., 2022)
64.95
70.84
41.55
60.54
33.27
55.48
26.00
50.99
GOLF (Jiang et al., 2022)
65.76
72.52
41.74
61.16
DiscoPrompt (Chan et al., 2023b)
65.79
71.70
43.68
61.02
45.99
60.84
39.27
57.88
ContrastivelDRR (Long and Webber, 2022)
69.60
72.18
49.66
61.69
ChatGPT (Chan al., 2023a)
36.11
44.18
16.20
24.54
PLSE
71.40
75.43
50.11
64.00
54.69
61.49
39.19
58.35
Table 2: The Macro-FI score (%) and Accuracy (%) are evaluated on PDTB 2.0 and 16 datasets. Italics numbers indicate the reproduced results from (Chan et al., 2023b). Bold numbers correspond to the best results. Underlined numbers correspond to the second best.
Model
Comp.
Cont.
Exp.
Temp.
Second-level Sense
GOLF
DP
Contrast
ours
SMMGF-RoBERIE (Liu 2020)
59.44
60.98
77.66
50.26
CG-T5 (Jinng 2021)
55.40
57.04
74.76
41.54
Comp..oncessson
0.00
9.09
0.00
0.0
GOLF Clang et 2022)
67.71
62.90
79.41
54.55
Comp.Contrast
61.95
59.26
62.63
60.28
DiscoPrumpi (Clun 202%6)
64.45
57.41
Caut
65.35
63.83
65.58
68.36
Contrastive (Tamp Wehher, 2002)
65.84
63.55
79.17
69.86
Cont. Pragmaatc Cause
0.00
0.00
0.00
0.00
PLSE
70.93
67.04 (81.50)
(1.50)
66.13
Exp.Al Alternativ
63.49
72.73
53.85
60.87
Table The performance for top-level classes on PDTB 2.0 in terms of Fl score (%)
Exp..Cnnunntion
60.28
61.08
58.35
59.80
Exp. Instantttnnioo
75.36
69.96
73.04
76.71
Exp.List
27.78
37.50
34.78
40.00
and 2.31 % second-level accuracy over the current ttat-off-the-ar ContrastiveIDRR model (Long and Webber, 2022) on PDTB 2.0. Moreover, with re- gard to CoNLL16, our method also outperforms the current state-of-the-art DiscoPrompt model (Chan al., 2023b) by a margin of 8.70% Fll on CoNLL- Test and 0.47% accuracy on CoNLLL-Blind For the performance of top-level classes on PDTB 2.0, our method significantly improves the performance of senses, particularly in Comp, Cont and Exp. It sur- passes the previous best results by 3.22%, 2.59% and 2.09% F1, respectively, demonstrating sub- stantial efficiency of our method. For the perfor- mance of second-level types PDTB 2.0, also compare our model with the current state-of saf-the-art art models. As illustrated Table 4, our results show a noteworthy enhancement in performance across five categories of second-level senses. Meanwhile, our model still remain exceptional average perfor- mance in most second-level senses.
Restateeett
59.84
60.00
58.45
63.78
63.82
57.69
59.79
64.22
Temp.S Synchrony
0.00
0.0
78.26
57.14
Table The performance for second-level types PDTB 2.0 in terms of F1 score (%). DP and Contrast indicate the DiscoPrompt and ContrastiveIDRR.
ness of these models are excessively reliant on the small-scale annotated data, that may pose a stum- bling block to future research explorations. NLP community, it has become a prevalent trend to leverage large-scale unlabeled data and design superior self-supervised learning methods. There- fore, our work focuses sufficiently exploiting unannotated data containing explicit connectives through prompt-based connective prediction. At the same time, to tackle the local dependencies of predicted connectives representations, we design a MI-maximization strategy, which aids in captur- ing the global logical semantics information perti- nent to the predicted connectives. Overall results demonstrate the great potential of our method. Finally, we compare our method with ChatGPT (Chan al., 2023a), recent large language model, on the IDRR task. The performance of ChatGPT significantly trails our PLSE model by approxi-
the
Secondly, a series of recent works for IDRR (Wu al., 2022: Jiang et al., 2022; Chan al., 2023b; Long and Webber, 2022) frequently utilize the hi- erarchical structure information from the annotated senses, leading to good results. Nevertheless, it's crucial to note that the performance and robust-

---

Model
Top-level
Second-level
72.0
PLSE
F1
Acc.
F1
Aec.
71.5
PLSE
71.40 75.43 50.11
64.00
71.0
w/o LOLSE -MTLes
68.64 73.23 47.53
63.52
70.5
w/o LGLSL -MT IITTIman
69.68 74.87 48.76
63.43
w/o LGLSL
69.17 74.66 47.94
63.71
NScio
70.0
wlo LoM
69.61 73.13 48.71
63.61
69.5
w/o LMEM
70.12 74.57 48.63
63.43
w/o LGLSL, LCM
68.18 73.23 47.20
63.13
68.0
wlo LG1.E. LML.M
68.62 74.95 47.50
63.62
w/o LGLSE, LCM, LMLM 67.12 71.03 46.54
62.17
68.5
68.0
Table 5: Ablation study PDTB 2.0 in terms of Macro-Fl score (%) and Accuracy (%); Lets and Lmeen stand for the multi-task learning method that additionally utilizes the [CLS] representation or the mean representation sequence to predict senses.
Data Scale (x10f)
Figure Effects of the unannotated data scale on the test set in PDTB 2.0.
unannotated data to learn better discourse relation representations by injecting discourse logical se- mantics and universal semantics into PLMs.
mately 35% F1 and 30% accuracy on PDTB 2.0, which highlights ChatGPT's poor capacity to prehend the logical semanties in discourses. There- fore, further research on IDRR remains crucial and imperative for the NLP community.
5.3 Data Scale Analysis
To investigate the impact of unannotated data scale the performance of our model, conduct sup- plementary extension experiments. As show in Figure 3, the performance swiftly accelerates prior to reaching a data scale of 500, 000, yet beyond this threshold, the pace of improvement markedly decelerates. The primary reason is the difference in semantic distributions of explicit and implicit data. For example, the connectives in explicit data play essential role in determining the discourse relations. Conversely, the implicit data does not contain any connectives, while the discourse rela- tions of sentences remain unaffected by implicit connectives in annotations. Therefore, consider- ing both performance and training cost, we select unannotated explicit data at scale of 500, 000.
5.2 Ablation Study
To better investigate the efficacy of individual mod- ules in our framework, we design numerous abla- tions on Global Logical Semantics Learning, Connectives Mask and MaskedLM tasks. Ta- ble 5 indicates that eliminating any of the three tasks would hurt the performance of our model on both top-level and second-level senses. It is worth noting that removing Global Logical Semanlics Learning significantly hurts the performance. Fur- thermore, assess the proficiency of Global Logical Semantics Learning, we compare with an auxiliary classification task leveraging the [CLS) representation or the average representation of sequence, which aims to introduce global infor- mation. The performance's improvement through MTLmman highlights the effectiveness of incor- porating global semanties. Surprisingly. MT Lai detrimentally impucts the model's performance. The reason could that the [CLS) representation encompasses a vast amount of information irrele- vant to global logical semantics, thereby interfer- ing with connective prediction. Our method con- siderably outperforms MT which demon- strates that our MI maximization strategy by inte- grating global logical semantics indeed benefi- cial for connective prediction. Besides, eliminating Connectives Mask or MaskedI.M also dimin- ishes the performance of our model, illustrating that our method can effectively utilize large-scale
5,4 Few-Shot Learning
To evaluate the robustness of our model in fhw-shot learning, we conduct additional experiments using 10%, 20%, and 50% of training data, respectively. The results of few-shot learning are summarized in Figure 4. The orange line illustrates the out- comes achieved by our PLSE method, while the blue line depicts the results obtained without pre- training. The results of the comparison demonstrate that method significantly outperforms the base- line across all metrics. In particular, our method, when trained only 50% of training data, achieves performance comparable the baseline trained on full training data, which further underscores the efficacy of our approach.

---

Zue
20
Macro-Fi (oppleseliin PDTR
Macnn-FI (sccord-oove) PDTB
Maem-FI CoNEi-Tos
Maem-FF CoNL1-Bliod
211
The
(c) Accuracy (op-level)
(D Accuracy (sczandleeee)
(g) Accunacy
(3) Accuncc CoNLLL-Biid
Figure 4: Performance comparison few-shot learning on PDTB 2.0 and CoNLL16 datasets.
6 Conclusion
future, we will explore new methods for predicting multi-token connectives.
In this paper, we propose novel prompt-basce logical semantics enhancement method for IDRR. Our upproach sufficiently exploits unannotated ex- plicit data by seamlessly incorporating discourse relation knowledge based on Eloze-Prompy con- nective prediction and comprehensively learning global logical semantics through MI maximization. Compared with the current -tate-of-the-art meth- ods, model transcends the limitations of overus- ing annotated multi-level hierarchical information, which achieves new state-of-the-an art performance on PDTB 2.0 and CoNLL2016 datasets.
Ethics Statement
This study focuses model evaluation and techni- cal improvements in fundamental research. There- fore, we have refrained from implementing any ad- ditional aggressive filtering techniques on the text data we utilize, beyond those already applied to the original datasets obtained from their respective sources. The text data employed in our research might possess elements of offensiveness, toxicity, faireess, or bias, which we haven't 't specifically ad- dressed since they fall outside the primary scope of this work. Furthermore, we do not see any other potential risks.
Limitations
Despite achieving outstanding performance, there are still some limitations of our work, which could be summarized into the following two aspects. The first is the different semantic distributions between explicit and implicit data, illustrated in Section 5.3, that significantly hampers further improve- ments of our model's performance. promising future research would involve the exploration of innovative approaches to bridging the gap between explicit and implicit data, or the development of strategies to filter explicit data exhibiting similar distribution patterns to implicit data. Additionally, our connective prediction method is limited by its reliance single tokens, thereby transforming two- token connectives into one (e.g.. for example example). Moreover, it discards three-token con- nectives entirely (e.g., as soon as). As result, our approach doesn't it fully exploit the pre-trained models and explicit data to some extent. In the
Acknowledgement
The authors would like to thank the organizers EMNLP2023 and the reviewers for their helpful suggestions. This work is partly supported by the grants from the National Natural Science Founda- tion of China (No. 62 62172044) and the Open Project Program of the National Defense Key Laboratory of Electronic Information Equipment System Re- search (No. 614201001032203).
References
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. 2016. Deep variational information bottleneck. arXiv preprint 1612.0601
Anthony J Bell and Terrence Sejnowski. 1995. An information-
approach to blind separa-

---

tion and blind deconvolution. Neural computation, 7(6):1129-113
maximization. In International Conference on Learn- ing Representations.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Aapo Hyvärinen and Petteri Pajunen. 1999. Nonlin- ear independent component analysis: Existence and uniqueness results, Neural networks, 12(3):42-439.
Yangfeng Ji and Jacob Eisenstein. 2015. One Vector is Not Enough: Augity-Augmented Distributed Se- mantics for Discourse Relations. Transactions of the Association for Computational Linguistics, 3:329- 344.
Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recog- nizing implicit discourse relation on joint modeling of classification and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 2418-2288 2431,,, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chunkit Chan, Jiayang Cheng, Weiqi Wang. Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023a. Chatgpt evaluation on sentence level rela- tions: A focus on temporal. causal, and discourse relations. arXiv preprint wxXiv2234.14822)
Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022. Global and local hierarchy aware contrastive frame- work for implicit discourse relation recognition. arXiv preprint arXh::2111.38773.
Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan Li, Yangqiu Song, Ginny Y Wong, and Simon See. 2023b. Discoprompt: Path prediction prompt tun- ing for implicit discourse relation recognition. arXiv preprint arX0i:2305.000 03973.
Yudai Kishimoto, Yugo Murawaki, and Sadao Kuro- hashi. 2020. Adapting BERT to implicit discourse relation classification with focus on discourse con- nectives. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1152- 1158. Marseille, France. European Language Re- sources Association.
Arman Cohan, Franck Dernnncour, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of NAACL-HLT, pages 615-621.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. 2019. A mutual information maximization perspective of language representation learning. arXiv preprint
Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng- Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 1299-1308.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. Openprompt: An open-source framework for prompt- learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 113.
Li Liang, Zheng Zhao, and Bonnie Webber. 2020. Ex- tending implicit discourse relation recognition to the PDTB-3. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 135- 147, Online. Association for Computational Linguis- tics.
Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Lei Li. 2022. Contextual representation learning beyond masked language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics [Volume I: Long Papers). pages 2701-2714 14, Dublin, Ireland. Association for Computational Linguistics.
Ziheng Lin, Min-Yee Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations the penn discourse treehank. In Proceedings of the 2009 con- ference empirical methods in natural language processing, pages 343-351.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and
Ralph Linsker. 1988. Self-organization in a perceptual network. Computer, 155-

---

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. arXiv e-prints, pages arXiv-2107.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguisticx: Main Volume, pages 255-2699, Online. Association for Computa- tional Linguistics.
Xin Liu, Jiefu Ou, Yanggiu Song, and Xin Jiang. 2020. On the importance of word and sentence represen- tation learning in implicit discourse relation classifi- cation. In Proceedings of the Twenty-Ninth Interna- tional Joint Conference on Artificial Intelligence.
Wei Shi and Vera Demherg. 2019. Learning to explici- tate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics Long Papers, pages 188-199, Gothen- burg. Sweden. Association for Computational Lin- guistics.
Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural net- works with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 224-1233, Austin, Texas. Association for Computational Linguisties.
Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xi- anpei Han, Le Sun, Weijian Xie, and Jin Xu. 2021. From discourse to narrative: Knowledge projection for event relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 732-742.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta:
robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1662.
Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 10704- 10716, Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguisties.
Siddharth Varia, Christopher Hidey, and Tuhin Chakrabarty. 2019a. Discourse relation prediction: Revisiting word pairs with convolutional networks. In Proceedings of the 20th annual SIGdial meeting on discourse and dialogue, pages 442-452.
Siddharth Varia, Christopher Hidey, and Tuhin Chakrabarty. 2019b. Discourse relation prediction: Revisiting word pairs with convolutional networks. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 442-452 Stock- holm, Sweden. Association for Computational Lin- guistics.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
1rXiv:1711.05101
Courtney Napoles, Matthew R Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceed- ings of the joint workshop on automatic knowledge base construction and web-scale knowledge extrac- tion (AKBC-WEKEX, pages 95-100.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Joonsuk Park and Claire Cardie. 2012. Improving im- plicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108-112.
Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treehank 3.0 annota- tion manual. Philadelphia, University of Pennsylva- nia, 35:108.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind K Joshi, and Bon- nie L Webber. 2008 The penn discourse treebank 2.0. In LREC.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu. Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of of aef-the-art art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-535 5551.
Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classi- fying explicit discourse connectives. In HLT-NAACL, pages 799-808.
Changxing Wu, Liuwen Cao, Yubin Ge, Yang Liu, Min Zhang, and Jinsong Su. 2022. A label dependence-

---

aware sequence generation model for multi-level im- plicit discourse relation recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 11486-11494.
A.2 Data Statisties
Top-level Senses
Train
Dev
Test
Comparison (Comp.)
1942
197
152
Nianwen Xue, Hwee Tou Ng. Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hong- min Wang. 2016. CoNLL 2016 shared task on multi- lingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1-19, Berlin, Ger- many, Association for Computational Linguistics.
Contingency (Cont.)
3340
292
279
Expansion (Exp.)
7004
671
574
Temporal (Temp.)
760
64
85
Total
12632
1183
1046
Table 6: Data statistics of four top-level implicit senses in PDTB 2.0.
Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recog- nition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 2230-2235.
Second-level Senses
Train
Dev
Test
Comp.Concessinn Comp.Contrast
184
15
17
1610
171
134
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximiza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 1601-1610, Online. Association for Computational Linguistics.
Cont.Cause
3277
284
272
Cont.Pragmatic cause
64
7
7
Axpernttrn
151
10
9
Conjunnnnnntion
2882
264
209
Int.antttttiitoon
1102
108
122
Exp.List
338
10
12
Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt- based connective pre- diction method for fine theegrained implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Restttstttement
2458
271
216
Temp.Asynchronne
555
50
57
Temp.Synchrony
204
13
28
Total
12406
1165
1039
Table 7: Data statisties second-level implicit
senses in PDTB 2.0.
A Appendix
A.3
Verbalizer for CoNLL16
A.1 Multi-level Sense Hierarchy
Cross-level Senses
Connectives
The hierarchies of both PDTB 2.0 and CONLL16 consist of three levels. According to the PDTB annotation guidelines, all senses are organized into a three-layer hierarchical structure. Figure 5 shows the multi-level sense hierarchy of an IDRR instance in the PDTB 2.0 corpus.
although. however
Conp.Concession
Comp.Contrast
hut
because
so, consequently,
Cont. Cont.Cause. Can.Caus..RRslt
thus, therefore
Cont. Conditonii
Alternatiie
unless instead, rather and, also, fact, furthermore
Exp.A Alternattertii Chosen alternative
Argumentl: The fundamentals are pretty strong Argument2: don't see this bear market all
Exp. Conjunction
Exceptxco
except
Exp. Instantitaiiiinnn
instance, example particular, indeed, specifically
Top-level sense: Contingency
Restatestttmnnt
Temp Asynhhosynhrooous Precedence
then, before
Second-level sense: Cause
Temp. wwwnnhhooous.Suceesset.
after
Third-level sense: Result
Temp Synchronyy
meanwhile, when
Table 8:
The verbalizer mapping between implicit dis- course relation labels and connectives on 1 dataset, including 14 cross-level senses.
Figure S: The multi-level sense hierarchy of an IDRR instance in the PDTB 2.0 corpus.

---

A.4 Mutual Information Discriminator
We concatenate the global logic-related representa- tion HCtooic and the conmective-related representa- tion Hanask as input. Then we feed this to the dis- criminator Tw. which produces scores to maximize the MI estimator in Equation (5). The architecture of Tu is shown in Table 9.
Operation
Size
Activation
Input + Linear layer
1536 + 768
ReLU
Linear layer
768 - 768
ReLU
Linear layer
768
Table 9: The network architecture of discriminator T.


Reviews:
[{"review_id": "R9Vtt1git5", "reviewer": "Reviewer_skmg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answer:** We agree with your opinion. In many cases, implicit discourse markers don't have the same behavior as explicit one.  There is the difference in semantic distributions of explicit and implicit data. Next, we will discuss this issue and provide a potential solution.\n\nFor implicit data, there is no connnective between two arguments, which requires a profound understanding of the semantics between them to discern the discourse relation.  Besides, incorporating appropriate connectives doesn't affect the semantics of sentences. An instance of implicit data is as follows:\n\n```\nMany colleagues are angry at Mrs. Yeargin. [implicit connective = Because] She did a lot of harm.\nrelation: contingency\n```\n\nFor explicit data,  it can be divided into two categories:\n\n- Required connective: removing connectives will change the logical semantics of sentences. An instance is as follows:\n\n```\nI want to go home for the holiday. [explicit connective = Nonetheless] I will book a flight to Hawaii.\nrelation: Comparison\n```\n\n\"Nonetheless\" plays a crucial role in determining the Comparison relation. If \"nonetheless\" is omitted,  the logical semantics of arguments will be shift to reflect a Contingency relation.\n\n- Optional connective：removing connectives doesn't affect the logical semantics of sentences. An instance is as follows:\n\n```\nLet's go dinner [explicit connective = because] I'm hungury.\nrelation: Contingency\n```\n\nSince the presence or absence of connectives does not influence the semantics of arguments, a similar semantic distribution exists between explicit data with optional connectives and implicit data. For explicit data with required connectives,  connectives determine the semantics of  arguments, so it's inappropriate to predict connectives by utilizing them.\n\nA potential strategy is to separate explicit data with required connectives from those with optional connectives. Here we briefly introduce it.\n\n- Implicit Distribution Feature Extraction: Train a classifier on both the original implicit data and the implicit data augmented with added connectives, which will output the predicted probability distribution of discourse relation respectively. We can utilize the disparity between the two predicted probability distributions as the implicit distribution features.\n- Explicit Data Filtering: The GAN is employed to learn the implicit distribution features. After training, the discriminator learns to discern  whether a sample aligns with the implicit distribution feature. Then we feed both the original explicit data and the explicit data without the connective into the classifier to obtain two predicted probability distributions. According to the disparity between the two predicted probability distributions , we utilize the discriminator to filter out explicit instances that align with the implicit data distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a strong and very good scientific experiment \n- a work that tackle a very hard task\n- a good result in the end\n", "reasons_to_reject": "- a main hypothesis that could be discussed", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper present an Implicit Discourse Relation Recognition system based on Prompt-based Logical Semantics Enhancement\n\nthe paper is really well-written and easy to read.\nI really appreciate all the work done to properly present the different part of the implementation, and the rigorous comparison.\nI have one core remark against the work wihich is that you hypothesize that implicit discourse markers have exactly the same behavior as explicit one, which is clearly not the case. I agree that you can cover a lot of cases and the current State of the art is far from this, but you will face a mojor issue at some point for improvement.\n", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w0puwMHXaT", "reviewer": "Reviewer_q9JC", "summary": "", "questions": "Question 1: Could you provide more details about the rationale behind selecting the specific Connectives Mask strategy, where the [MASK] token is replaced with the connective token 90% of the time and unchanged 10% of the time? How was this ratio determined, and what experiments were conducted to validate its effectiveness?\n\nQuestion 2: In Section 4.3, you mentioned that the performance improvement of your method decelerates beyond a data scale of 500,000. Could you elaborate on how the semantic distribution of explicit and implicit data affects this performance improvement? Are there any potential strategies to further bridge the gap between explicit and implicit data to achieve better performance?\n\nQuestion 3: The paper mentions that the current approach converts two-token connectives into single tokens and abandons three-token connectives during connective prediction. How significant is the impact of this conversion on the overall performance? Have you explored other methods or techniques to handle multi-token connectives in the prompt-based approach?\n\nQuestion 4: In the Limitations section, you discussed that the paper focused on model evaluation and technical improvements and refrained from implementing aggressive filtering techniques on the text data. Have you considered or discussed the potential ethical implications of utilizing text data that may contain elements of offensiveness, toxicity, fairness, or bias in your research?\n\nQuestion 5: The paper highlights the outperformance of your PLSE method compared to ChatGPT for IDRR. Could you provide insights into the aspects where ChatGPT lacks the capacity to comprehend logical semantics in discourses? What do you think are the key factors contributing to the superiority of your approach over ChatGPT in this specific task?\n\nQuestion 6: Given that your approach achieved state-of-the-art results on the PDTB 2.0 and CoNLL16 datasets, do you have any plans or ongoing work to apply your method to other NLP tasks or domains? How generalizable do you believe your approach is, and what are the potential challenges or limitations in applying it to different tasks or languages?\n\nQuestion 7: Considering the experimental settings, are there any hyperparameters or model architecture choices that were thoroughly explored during the experiments but not explicitly mentioned in the paper? Are there any other aspects that you believe could be further optimized to potentially improve the overall performance of the PLSE method?\n\nQuestion 8: The paper mentions that the proposed method incorporates discourse relation knowledge based on Cloze-Prompt connective prediction and comprehensively learns global logical semantics through MI maximization. Could you discuss the relative importance or contributions of these two aspects in achieving the overall performance improvements?\n\nQuestion 9: In the Ethics Statement, you mentioned that the text data utilized in your research might possess elements of offensiveness, toxicity, fairness, or bias. Have you conducted any analysis or taken any measures to mitigate potential ethical concerns related to the usage of such data in the context of your research?\n\nQuestion 10: In the Conclusion section, you mentioned that future research explorations could involve bridging the gap between explicit and implicit data or exploring strategies to filter explicit data with similar distribution patterns to implicit data. Could you elaborate on potential research directions to address these challenges and further enhance the performance of the PLSE method in handling explicit and implicit data?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answer 1:** \n\n1. Connectives (e.g., but, and, because, meanwhile), due to their inherent logical semantics, play a pivotal role in establishing the discourse relation of arguments. It's noteworthy that each explicit data consists of a connective. Predicting these connectives enhances the model's comprehension of logical semantics. To optimally leverage the connective information within explicit data, we replaced the connective with (1) the [MASK] token 90 % of the time (2) the unchanged token 10% of the time.\n2. We refered to BERT's strategy for the token to be replaced (the chosen token is replaced with (1) the [MASK] token 80% of the time (2) a random token 10% of the time (3) the unchanged  token 10% of the time). Due to the strong logical correlation of connectives, we eliminated \"a random token 10% of the time\", which could affect the logical semantics of sentences. In order to avoid the model consistently encountering the masked connectives, we perserved \"the unchanged  token 10% of the time\". \n3. We conducted experiments with masked ratios of 80%, 90%, and 100%, respectively. The experimental results for the three ratios showed a remarkable closeness. The performance at a 90% ratio slightly outperformed others by 0.2-0.3 F1, so we set this ratio at 90%.\n\n**Answer 2:**\n\n2.1. For implicit data, there is no connnective between two arguments, which requires a profound understanding of the semantics between them to discern the discourse relation.  Besides, incorporating appropriate connectives doesn't affect the semantics of sentences. An instance of implicit data is as follows:\n\n```\nMany colleagues are angry at Mrs. Yeargin. [implicit connective = Because] She did a lot of harm.\nrelation: contingency\n```\n\nFor explicit data,  it can be divided into two categories:\n\n- Required connective: removing connectives will change the logical semantics of sentences. An instance is as follows:\n\n```\nI want to go home for the holiday. [explicit connective = Nonetheless] I will book a flight to Hawaii.\nrelation: Comparison\n```\n\n\"Nonetheless\" plays a crucial role in determining the Comparison relation. If \"nonetheless\" is omitted,  the logical semantics of arguments will be shift to reflect a Contingency relation.\n\n- Optional connective：removing connectives doesn't affect the logical semantics of sentences. An instance is as follows:\n\n```\nLet's go dinner [explicit connective = because] I'm hungury.\nrelation: Contingency\n```\n\nSince the presence or absence of connectives does not influence the semantics of arguments, a similar semantic distribution exists between explicit data with optional connectives and implicit data. For explicit data with required connectives,  connectives determine the semantics of  arguments, so it's inappropriate to predict connectives by utilizing them.\n\n2.2. A potential strategy is to separate explicit data with required connectives from those with optional connectives. Here we briefly introduce it.\n\n- Implicit Distribution Feature Extraction: Train a classifier on both the original implicit data and the implicit data augmented with added connectives, which will output the predicted probability distribution of discourse relation respectively. We can utilize the disparity between the two predicted probability distributions as the implicit distribution features.\n- Explicit Data Filtering: The GAN is employed to learn the implicit distribution features. After training, the discriminator learns to discern  whether a sample aligns with the implicit distribution feature. Then we feed both the original explicit data and the explicit data without the connective into the classifier to obtain two predicted probability distributions. According to the disparity between the two predicted probability distributions , we utilize the discriminator to filter out explicit instances that align with the implicit data distribution.\n\n**Answer 3:**  Firstly, for coverting two-tokens into single tokens (e.g., *for example* -> *example* , *in particular* -> *particular*), there is an inconsistency between pre-training and prompt-tuning. Secondly, there are numerous explicit instances with three-token connectives (e.g., *as soon as*, *in other words*). Abandoning them will lead to underutilization of pre-training data and PLMs. Due to the complexity of considering multi-token connectives, we didn't delve into this issue further. A potential method to handle multi-token connectives is as follows:\n\n- Train a classifier on the iput (*Argument1 Argument2*) to predict the number of tokens in connectives.\n- According to the predicted number, reconstruct a cloze-prompt input (*Argument1* [MASK] *Argument2*,  *Argument1* [MASK] [MASK] *Argument2* or *Argument1* [MASK] [MASK] [MASK] *Argument2*) to predict connectives.\n\n**Answer 4:** The pre-training data was collected from a corpus named \"English Gigaword\", which is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortium (LDC) at the University of Pennsylvania. This corpus is widely used in various Natural Language Processing (NLP) tasks for research purposes. Nonetheless, English Gigaword may still contain elements of offensiveness, toxicity, fairness, or bias. The following aggressive ﬁltering techniques can be used:\n\n- Machine Learning Models: There are models like Perspective API from Jigsaw that are trained to detect toxic comments. By using these, we can estimate the toxicity of a given piece of text and filter accordingly.\n- Rule-based Filters: Use lists of banned words or phrases.\n\n**Answer 5:** \n\n5.1. Chan et al. (2023) evaluated the ability of ChatGPT for EDRR (Explicit Discourse Relation Recognition) and IDRR (Implicit Discourse Relation Recognition). Due to the existence of connectives, ChatGPT achieved outstanding performance on EDRR, demonstrating that ChatGPT could solve this task easily by utilizing the signal from explicit connectives. For IDRR, the absence of connectives necessitates that ChatGPT possesses a deep understanding of logical semantics in discourses. It's a challenging task because ChatGPT may not understand the abstract sense of each discourse relation and grasp the language features from the discourses.\n\n5.2. The key factors are summarized as follows:\n\n- Connective prediction based on a cloze-prompt template: On the one hand, compared to directly predicting discourse relations, connective prediction simplifies the complexity of the inference. On the other hand, a connective typically emerges between two sentences, so the cloze prompt template is ideal for connective prediction, that capitalizes on the potential of data and PLMs.\n- Global logical semantics learning: A novel method based on MI maximization aims to address the local dependencies of predicted connective representations and capture global semantic information with logical coherence, that enhances the model's ability to comprehend logical semantics.\n\n**Answer 6:** \n\n6.1. Yes, we plan to explore how to extend our approach further, and broaden its application to the logical or causal reasoning task for large language models (LLMs).\n\n6.2. For example, the method for global logical semantics learning based on MI maximization is not confined to discourse relation recognition or the logical semantics learning, which can be generalized to certain reasoning tasks that rely on global information such as long document reading comprehension.\n\n6.3. Currently, a potential limitation of our approach is focused solely on encoder models. Considering how to adapt our method for tasks using the encoder-decoder or decoder-only model is worth further exploration.\n\n**Answer 7:** \n\n7.1. We sincerely apologize for not clarifying the settings clearly. The experimental settings are supplemented as follows:\n\n- All experiments were implemented in PyTorch framework by HuggingFace transformers.\n- Other hyperparameters, including the weight decay and the exponential decay rates β1 and β2, were consistent with the default configuration of AdamW.\n- For the warmup of the pre-training, we chose `get_cosine_schedule_with_warmup` as the learning rate schedulers provided by the `transformers` library.\n\nThe details of model architecture had been introduced in Section 4.2 and Appendix A.4 . We explored the number of heads in the Multi-Head Cross-Attention (MHCA) module as well as the number of layers in the mutual information discriminator, which had slight impact on the overall performance. Therefore, we didn't delve deeply into the discussion of the model structure choices.\n\n7.2. For *Connectives Mask* , *MaskedLM* , and *Global Logical Semantics Learning* , three tasks just shared the same weights for their loss functions. Considering different weights could potentially improve the performance of our method.\n\n**Answer 8:** \n\nCloze-prompt connective prediction:\n\n In Table 3 of our paper, previous work PCP (Zhou et al., 2022) adopted a prefix template in the following manner:\n\n```\nArg1:“the content of argument1” Arg2:“the content of argument2” [SEP] The conjunction between Arg1 and Arg2 is [MASK].\n```\n\nIt attains 64.95% F1, which is approximately 2.2% lower than our cloze-prompt template when compared fairly without explicit data pre-training. The primary reasons for the effectiveness of a cloze-prompt template are as follows:\n\nIn most cases, a connective (e.g., however, so) typically emerges between two sentences to serve the purpose of linking them together. For example, “The fundamentals are pretty strong, **so** I don't see this as a bear market at all.” and “But that ghost wasn't fooled, **because** he knew the RDF was neither rapid nor deployable nor a force.”. Therefore, the cloze-prompt template aligns more closely with natural language expression than the prefix-prompt template for the connective prediction. Furthermore, the cloze-prompt template bridges the gap between the pre-training and prompt-tuning, that can fully capitalize on the potential of masked language modeling.\n\nGlobal logical semantics learning through MI maximization:\n\nIn Section 5.2, we compared our method with an auxiliary classification task leveraging the [CLS] representation or the average representation of a sequence, which aims to introduce global information.\n\n| Model                   | Top-level (F1 ; Acc) | Second-level (F1 ; Acc) |\n| ----------------------- | -------------------- | ----------------------- |\n| ours                    | 71.40 ; 75.43        | 50.11 ; 64.00           |\n| w/o GLSL - $MTL_{cls}$  | 68.64 ; 73.23        | 47.53 ; 63.52           |\n| w/o GLSL - $MTL_{mean}$ | 69.68 ; 74.87        | 48.76 ; 63.43           |\n| w/o GLSL                | 69.17 ; 74.66        | 47.94 ; 63.71           |\n\nWhere $MTL_{cls}$ and $MTL_{mean}$ stand for the multi-task learning method that additionally utilizes the [CLS] representation or the mean representation of a sequence to predict senses. \n\nThe performance's improvement through $MTL_{mean}$ highlights the effectiveness of incorporating global semantics. Surprisingly, $MTL_{cls}$ detrimentally impacts the model's performance. The reason could be that the [CLS] representation encompasses a vast amount of information irrelevant to global logical semantics, thereby interfering with connective prediction. Our method considerably outperforms $MTL_{mean}$, which demonstrates that our MI maximization strategy by integrating global logical semantics is indeed beneficial for connective prediction.\n\n**Answer 9:** \n\nThe text data utilized in our research might possess elements of offensiveness, toxicity, fairness, or bias, but our model focused on IDRR, that output the predictions of connectives and then mapped connectives to discourse relations. Because the output was only associated with connectives and discourse relations, which exhibited neither offensiveness, toxicity, fairness, nor bias. Therefore, our research didn't result in potential ethical concerns.\n\n**Answer 10:**\n\nThis question closely resembles Question 2. Kindly refer to Answer 2 for further clarification.\n\n**Typos Grammar Style And Presentation Improvements:**\n\nThank you for your feedback. We will thoughtfully incorporate your suggestions to enhance the clarity and readability of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are as follows:\n\n1、Novel Approach: The paper introduces a novel and innovative approach for Implicit Discourse Relation Recognition (IDRR) using a prompt-based logical semantics enhancement method. The use of Cloze-Prompt templates, Connectives Mask task, and Global Logical Semantics Learning sets this work apart from existing approaches.\n\n2、Performance Improvement: The proposed PLSE method achieves state-of-the-art performance on two benchmark datasets, PDTB 2.0 and CoNLL16, outperforming several advanced baselines. This improvement in IDRR performance is valuable to the NLP community as it addresses an important problem in natural language understanding.\n\n3、Experimental Rigor: The paper provides a comprehensive experimental setup, including ablation studies, data scale analysis, and comparisons with existing methods. The experiments are well-designed and support the claims made in the paper, enhancing the credibility of the results.\n\n4、Applicability: The proposed approach leverages large-scale unannotated data, making it applicable to real-world scenarios where labeled data may be limited or expensive to obtain. This approach has the potential to be extended to other NLP tasks, further benefiting the NLP community.\n\n5、Clear Presentation: The language and structure of the paper are clear and well-organized, making it accessible to readers. The authors effectively convey their methodology, results, and conclusions, facilitating understanding by the NLP community.\n\nIf accepted, this paper would provide valuable insights and advancements in the field of IDRR, offering a new and effective method for leveraging large-scale unannotated data and logical semantics in pre-trained language models. The state-of-the-art results and novel contributions would contribute significantly to the NLP community's understanding of implicit discourse relations and open avenues for further research in this domain. Overall, the acceptance of this paper at the conference or Findings would be beneficial to the NLP community and enrich the conference's technical program.", "reasons_to_reject": "The weaknesses of this paper are as follows:\n\n1、Lack of Comparison: The paper lacks a comprehensive comparison with a broader range of existing state-of-the-art methods. While it does compare its proposed method with several recent baselines, it would be beneficial to include a more extensive comparison with a wider variety of approaches to demonstrate the superiority of the proposed method more convincingly.\n\n2、Limited Reproducibility: The paper falls short in providing sufficient details for reproducibility. Although it outlines the implementation details, some critical parameters and settings are not clearly specified, which may make it challenging for other researchers to reproduce the results accurately.\n\n3、Ethical Implications: While the paper discusses the technical aspects and contributions, it does not adequately address the ethical implications of the research. Given the nature of the topic, it would be essential to thoroughly assess and discuss potential ethical concerns and implications arising from the proposed method.\n\n4、Language and Presentation: The paper has some language and presentation issues, such as grammar errors and awkward sentence structures. Improving the language and presentation would enhance the clarity and readability of the paper.\n\n5、Missing References: The paper may benefit from including references to related works that have been overlooked or are relevant to the proposed method. Providing a more comprehensive literature review would strengthen the paper's foundation.\n\n6、Lack of Novelty: While the paper introduces a prompt-based logical semantics enhancement method for IDRR, some aspects of the method appear to be incremental improvements rather than significant novel contributions. The paper could better emphasize and highlight the novel aspects of its approach.\n\nThe main risks of having this paper presented at the conference or accepted into Findings include:\n\n1、Misleading Claims: Without a more extensive comparison and thorough reproducibility, the paper's claims may be misleading or overestimated, leading to potential misinterpretations by the NLP community.\n\n2、Ethical Concerns: As the paper deals with discourse relations, it is crucial to carefully examine and address any potential ethical concerns associated with the research. Failure to do so may raise ethical issues and impact the paper's acceptance.\n\n3、Limited Impact: Given the limited novelty and incremental nature of some contributions, the paper's impact on the NLP community may be limited. There is a risk that the paper may not significantly advance the state-of-the-art or offer substantial contributions.\n\n4、Clarity Issues: The language and presentation issues may make it challenging for readers and reviewers to understand the paper's content and contributions fully.\n\nOverall, while the paper has some strengths, it also exhibits several weaknesses that need to be addressed to ensure a solid contribution to the NLP community and mitigate potential risks associated with its presentation or acceptance.", "questions_to_authors": "Question 1: Could you provide more details about the rationale behind selecting the specific Connectives Mask strategy, where the [MASK] token is replaced with the connective token 90% of the time and unchanged 10% of the time? How was this ratio determined, and what experiments were conducted to validate its effectiveness?\n\nQuestion 2: In Section 4.3, you mentioned that the performance improvement of your method decelerates beyond a data scale of 500,000. Could you elaborate on how the semantic distribution of explicit and implicit data affects this performance improvement? Are there any potential strategies to further bridge the gap between explicit and implicit data to achieve better performance?\n\nQuestion 3: The paper mentions that the current approach converts two-token connectives into single tokens and abandons three-token connectives during connective prediction. How significant is the impact of this conversion on the overall performance? Have you explored other methods or techniques to handle multi-token connectives in the prompt-based approach?\n\nQuestion 4: In the Limitations section, you discussed that the paper focused on model evaluation and technical improvements and refrained from implementing aggressive filtering techniques on the text data. Have you considered or discussed the potential ethical implications of utilizing text data that may contain elements of offensiveness, toxicity, fairness, or bias in your research?\n\nQuestion 5: The paper highlights the outperformance of your PLSE method compared to ChatGPT for IDRR. Could you provide insights into the aspects where ChatGPT lacks the capacity to comprehend logical semantics in discourses? What do you think are the key factors contributing to the superiority of your approach over ChatGPT in this specific task?\n\nQuestion 6: Given that your approach achieved state-of-the-art results on the PDTB 2.0 and CoNLL16 datasets, do you have any plans or ongoing work to apply your method to other NLP tasks or domains? How generalizable do you believe your approach is, and what are the potential challenges or limitations in applying it to different tasks or languages?\n\nQuestion 7: Considering the experimental settings, are there any hyperparameters or model architecture choices that were thoroughly explored during the experiments but not explicitly mentioned in the paper? Are there any other aspects that you believe could be further optimized to potentially improve the overall performance of the PLSE method?\n\nQuestion 8: The paper mentions that the proposed method incorporates discourse relation knowledge based on Cloze-Prompt connective prediction and comprehensively learns global logical semantics through MI maximization. Could you discuss the relative importance or contributions of these two aspects in achieving the overall performance improvements?\n\nQuestion 9: In the Ethics Statement, you mentioned that the text data utilized in your research might possess elements of offensiveness, toxicity, fairness, or bias. Have you conducted any analysis or taken any measures to mitigate potential ethical concerns related to the usage of such data in the context of your research?\n\nQuestion 10: In the Conclusion section, you mentioned that future research explorations could involve bridging the gap between explicit and implicit data or exploring strategies to filter explicit data with similar distribution patterns to implicit data. Could you elaborate on potential research directions to address these challenges and further enhance the performance of the PLSE method in handling explicit and implicit data?", "paper_topic_and_main_contributions": "This paper focuses on Implicit Discourse Relation Recognition (IDRR) and proposes a novel prompt-based logical semantics enhancement method for this task. The main problem addressed in this paper is the challenge of recognizing implicit discourse relations between argument pairs in natural language text.\n\nThe main contributions of this paper are as follows:\n\n1、Proposing a prompt-based Cloze-Prompt template for reformulating input argument pairs to improve the semantic representation of arguments and facilitate connective prediction.\n2、Introducing a Connectives Mask task to incorporate discourse logical information into Pre-trained Language Models (PLMs) by learning intricate representations of connectives and aiding knowledge transfer to downstream tasks.\n3、Introducing a Global Logical Semantics Learning method based on Mutual Information (MI) maximization to capture global semantic information with logical coherence. This approach addresses the local dependencies of predicted connectives and enhances discourse relation representations.\n4、Conducting extensive experiments and achieving new state-of-the-art performance on the Penn Discourse Treebank 2.0 (PDTB 2.0) and CoNLL-2016 Shared Task (CoNLL16) datasets for IDRR.\n\nOverall, the paper presents a valuable contribution to the field of Implicit Discourse Relation Recognition by proposing a novel approach that leverages prompt-based connective prediction and mutual information maximization to improve discourse relation recognition performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\nBased on the content of the paper, there are no specific missing references that should be included in the bibliography or need to be discussed in more depth. The authors have appropriately cited prior works and relevant baselines for the task of Implicit Discourse Relation Recognition (IDRR) in the related work section. The paper demonstrates novelty in its approach and contributions, and the references provided adequately support the research.\n\nAs a reviewer, I do not identify any critical missing references that would significantly impact the evaluation or understanding of the paper's contributions. The authors have provided a comprehensive list of related works and baselines, which sufficiently contextualizes the significance of their proposed Prompt-based Logical Semantics Enhancement (PLSE) method for IDRR.", "Typos_Grammar_Style_and_Presentation_Improvements": "1、In section 3.2 Pre-training, there seems to be a typo in the sentence \"Therefore, we approximately collect 0.56 million explicit argument pairs from an unannotated corpus renowned as Gigaword (Napoles et al., 2012) by leveraging its vast collection of connectives.\" It should be \"approximately collected\" instead of \"approximately collect.\"\n\n2、In section 3.2 Pre-training, when describing the Connectives Mask task, there's a missing period at the end of the sentence \"The cross-entropy loss for Connectives Mask (CM) is as follows.\"\n\n3、In section 3.2 Pre-training, the equation for Masked Language Model (MLM) loss is missing a closing parenthesis. It should be: \"The cross-entropy loss for MLM is as follows: ...\"\n\n4、In section 4.3 Baselines, the sentence \"Here we mainly introduce the baselines that have emerged within the past two years\" should have \"baselines\" capitalized as \"Baselines.\"\n\n5、In section 5 Experimental Results, there's a typo in the sentence \"Based on these results, we derive the following conclusions: Firstly, our PLSE method has achieved new state-of-the-art performance on PDTB 2.0 and CoNLL16 datasets.\" It should be \"derive\" instead of \"derive.\"\n\n6、In section 5 Experimental Results, the sentence \"Finally, with the powerful ability of recent large language models (LLMs) on numerous tasks, Chan et al. (2023a) evaluated the ability of ChatGPT for IDRR task.\" should be rephrased for clarity. For example: \"Finally, we compare our method with ChatGPT (Chan et al., 2023a), a recent large language model, on the IDRR task.\"\n\n7、In section 6 Conclusion, the sentence \"Another limitation is that our method of connective prediction is based on a single token, which converts two-token connectives into single tokens (e.g., for example→example) and abandons three-token connectives (e.g., as soon as).\" could be made clearer by breaking it into two sentences for better readability.\n\n8、In section 6 Conclusion, the sentence \"We note that this work is a fundamental research work that focuses on model evaluation and technical improvements\" has repetition and could be revised to \"This study focuses on model evaluation and technical improvements in fundamental research.\"\n\nOverall, the language and presentation in the paper are generally clear and understandable. However, some sentences could be rephrased to enhance clarity and readability. The paper's structure is logical, and the motivation and explanations provided are sufficient to understand the research. The authors have done a good job presenting their contributions and experimental results, making it easy to follow and evaluate the finding", "Justification_for_Ethical_Concerns": "Based on the content of the paper, there were no explicit ethical concerns raised or discussed in the paper that would require special review by an ethics committee. The paper primarily focuses on technical aspects of natural language processing, and there were no indications of potential ethical challenges or controversial topics. As such, it may not be necessary to conduct a separate review for ethical concerns in this particular paper."}}, {"review_id": "Ufowx69AsB", "reviewer": "Reviewer_7gCC", "summary": "", "questions": "1) In this paper, a cloze-prompt template is adopted. The authors also claimed that it is more effective than other templates, for example, the prefix-prompts. It would be better to give more explanations or conduct the corresponding experiments.\n2) In the caption of Figure 2, what does ‘implicit data is randomly selected from the training data’ mean? Are all the implicit training data used during the prompt-tuning phase?\n3) It would be interesting to conduct experiments with fewer labeled data, for example, 10%, 20%, and 50% of implicit training data.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answer 1:** In Table 3 of our paper, previous work PCP (Zhou et al., 2022) adopted a prefix template in the following manner:\n\n```\nArg1:“the content of argument1” Arg2:“the content of argument2” [SEP] The conjunction between Arg1 and Arg2 is [MASK].\n```\n\nIt attains 64.95% F1, which is approximately 2.2% lower than our cloze-prompt template when compared fairly without explicit data pre-training. The primary reason for the effectiveness of a cloze-prompt template is as follows:\n\nIn most cases, a connective (e.g., however, so) typically emerges between two sentences to serve the purpose of linking them together. For example, “The fundamentals are pretty strong, **so** I don't see this as a bear market at all.” and “But that ghost wasn't fooled, **because** he knew the RDF was neither rapid nor deployable nor a force.”. Therefore, the cloze-prompt template aligns more closely with the natural language expression than the prefix-prompt template for the connective prediction. Furthermore, the cloze-prompt template bridges the gap between the pre-training and prompt-tuning, that can fully capitalizes on the potential of masked language modeling.\n\n**Answer 2:**\n\n1. We sincerely apologize for not clarifying this sentence. “implicit data is randomly selected from the training data” just means that the training data is randomly shuffled and then fed into the model.\n2. Yes, all the implicit training data was used during the prompt-tuning phase.\n\n**Answer 3:** Thanks for your advice. We supplement experiments with fewer labeled data.\n\nour method:\n\n| percentage of training data | PDTB-top(F1 ; Acc) | PDTB-second(F1 ; Acc) | CoNLL-Test(F1 ; Acc) | ConNLL-Blind(F1 ; Acc) |\n| --------------------------- | ------------------ | --------------------- | -------------------- | ---------------------- |\n| 10%                         | 58.58 ; 68.16      | 40.63 ; 56.4          | 46.74 ; 54.70        | 31.10 ; 54.12          |\n| 20%                         | 63.03 ; 68.74      | 41.90 ; 57.65         | 48.88 ; 56.40        | 35.74 ; 55.53          |\n| 50%                         | 66.90 ; 72.66      | 46.17 ; 61.79         | 52.85 ; 57.96        | 37.34 ; 57.41          |\n| 100%                        | 71.40 ; 75.43      | 50.11 ; 64.00         | 54.69; 61.49         | 39.19 ; 58.35          |\n\nonly prompt-tuning (w/o pre-training):\n\n| percentage of training data | PDTB-top(F1 ; Acc) | PDTB-second(F1 ; Acc) | CoNLL-Test(F1 ; Acc) | ConNLL-Blind(F1 ; Acc) |\n| --------------------------- | ------------------ | --------------------- | -------------------- | ---------------------- |\n| 10%                         | 57.12 ; 65.23      | 36.49 ; 55.05         | 37.27 ; 52.61        | 30.65; 49.18           |\n| 20%                         | 61.04 ; 67.06      | 37.85 ; 56.68         | 43.03 ; 56.39        | 35.10 ; 52.47          |\n| 50%                         | 64.25 ; 69.88      | 45.45 ; 58.81         | 48.48 ; 57.31        | 36.76 ; 55.06          |\n| 100%                        | 67.12 ; 71.03      | 46.54 ; 62.17         | 52.77; 59.01         | 38.23 ; 55.53          |\n\nFrom the tables above, our method, when trained on only 50% training data, achieves performance comparable to the baseline trained on full training data, which further demonstrates the effectiveness of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) In terms of implicit discourse relation recognition, the proposed mutual information-based loss for pre-training is novel and interesting.\n2) The experimental results are convincing, and can effectively support the claims in this paper.", "reasons_to_reject": "1) In my opinion, this is no sufficient reason to reject this paper.", "questions_to_authors": "1) In this paper, a cloze-prompt template is adopted. The authors also claimed that it is more effective than other templates, for example, the prefix-prompts. It would be better to give more explanations or conduct the corresponding experiments.\n2) In the caption of Figure 2, what does ‘implicit data is randomly selected from the training data’ mean? Are all the implicit training data used during the prompt-tuning phase?\n3) It would be interesting to conduct experiments with fewer labeled data, for example, 10%, 20%, and 50% of implicit training data.", "paper_topic_and_main_contributions": "This paper proposes a novel pre-training method for implicit discourse relation recognition, where large amounts of explicit discourse data, labeled with connectives naturally, is leveraged effectively. A self-supervised learning objective based on mutual information maximization is introduced to capture global semantics. The idea is interesting and the performance on commonly-used datasets is impressive.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers view the paper as a pioneering approach to Implicit Discourse Relation Recognition (IDRR) using a method of logical semantics enhancement.  While the concept of using explicit connective prediction has been explored several times before, this might be the first case of introducing the task to pre-training via the loss function. The results indicate that it achieves state-of-the-art performance on two benchmark datasets.

Pros:
Introduces a novel method for IDRR, leveraging logical semantics enhancement.
Achieves state-of-the-art performance on benchmark datasets.
Comprehensive experimental design, including diverse analyses.
Potential applicability to a wide range of real-world scenarios.
The paper is well-structured, making it accessible for readers (although one reviewer disagrees).
Cons:
Lacks comprehensive comparison with a broader range of state-of-the-art methods.
Issues with reproducibility due to lack of sufficient details.
Does not address potential ethical implications adequately.
Language and presentation concerns as pointed out by one reviewer.
Some aspects of the method might seem more incremental than novel. (which I agree with)
Decision: Accept-Main