Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading
Authors: Yangyang Luo
Shiyu Tian
Caixia Yuan
Xiaojie Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational Machine Reading
Task-oriented dialogue
Textual Entailment

Abstract: Conversational Machine Reading (CMR) requires answering a user's initial question through multi-turn dialogue interactions based on a given document. Although there exist many effective methods, they largely neglected the alignment between the $\textit{document}$ and the $\textit{user-provided information}$, which significantly affects the intermediate decision-making and subsequent follow-up question generation. To address this issue, we propose a pipeline framework that (1) aligns the aforementioned two sides in an explicit way, (2) makes decisions using a lightweight many-to-many entailment reasoning module, and (3) directly generates follow-up questions based on the document and previously asked questions. Our proposed method achieves state-of-the-art in micro-accuracy and ranks the first place on the public leaderboard of the CMR benchmark dataset ShARC.

Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading
Yangyang Luo, Shiyu Tian, Caixia Yuan, Xiaojie Wang" School of Artificial Intelligence, Beijing University of Posts and Telecommunications (luoyangyang, tiansy, yuancx, xjwang]dbupt.ed edu.
Abstract
(document) Statutory Maternity Leave
You qualify for Statutory Maternity L Leave if: * you're an employee not 'worker'e you give your employer the correct notice
Conversational Machine Reading (CMR) re- quires answering a user's initial question through multi-turn dialogue interactions based given document. Although there ist many effective methods, they largely ne- glected the alignment between document and the user-provided information, which sig- nificantly affects the intermediate decision- making and subsequent follow-up question gen- eration. To address this issue, we propose pipeline framework that (1) aligns the afore- mentioned two sides in explicit way, (2) makes decisions using lightweight many -to- many entailment reasoning module, and (3) di- rectly generates follow-up questions based the document and previously asked questions. Our proposed method achieves state-of-the-an in micro-accuracy and ranks the first place the public leaderboard' of the CMR benchmark dataset ShARC.
(scenario) I'm going to have my baby in two months. I'm still working right now and I just turned in the notice.
(initial question) Do I qualify for Statutory Maternity Leave?
(follow-up question) Are you an employee and not worker?
NO
MORE
(follow-up answer) Yes.
IWANT
YES
(final answer) Yes.
MORE
Figure 1: An example of Conversational Machine Read- ing from ShARC dataset (Saeidi et al., 2018).
1 Introduction
For decision-making. one common approach first segments the document into many text spans at different granularity levels (e.g., sentences or Ele- mentary Discourse Units (EDUs)). Then complex modules are adopted to predict the entailment state for each document span based on user scenario and previous dialogue history (both user-provided information). Finally, decisions are made based on the entailment states of all document spans. One effective module for predicting entailment states is transformer blocks (Vaswani et al., 2017), which are widely adopted (Gao et al., 2020b; Ouyang et al., 2021: Zhang et al., 2022). However, the aforementioned approach has overlooked the ex- plicit alignment between the document and the user- provided information, such as text spans marked with the same color as shown in Figure 1. Since not all user-provided information relevant ular document span, the lack of explicit aligmment leads to sparse attention and introduces noises that
The Conversational Machine Reading (CMR) task (Sacidi et al., 2018) requires agent to answer an initial question from users through multi-turn dia- logue interactions based given document. As shown in Figure 1, typical process involves two steps. (1) the agent first makes decision classifica- tion among IRRELEVANT. YES, NO and MORE, (2) if the decision MORE, the agent generates a question to clarify an unmentioned condition in the given document, otherwise responds directly. Re- cent research (Verma et al., 2020; Lawrence et al., 2019; Zhong and Zettlemoyer, 2019; Gao et al., 2020a; Gao et al., 2020b; Ouyang et al., 2021; Zhang et al., 2022) has explored how to improve the abilities of prerision-making and question gen- eration.
partic-
*Corresponding author
htttp::/sshare. data, giub. wt/lo/aeeeee leaderboard.
html

---

affect the prediction of the entailment state. Fur- thermore, recent work (Ouyang al., 2021) tries to leverage relational graph convolutional networks, which results in heavyweight decision module, therefore greatly imposing a substantial burden on computation and memory resourees.
previous SOTA method on key metrics, with a significant reduction in decision module pa- rameters.
2 Related Work
For question generation, most works (Zhong and Zettlemoyer, 2019; Gao et al., 2020a; Gao al., 2020b; Ouyang al., 2021) first extract an unmen- tioned span in the document and then rewrite into a follow-up question. The extract-then-rewrite method relies heavily on extracted spans, the fail- ure to properly extract an unmentioned span results in generating a redundant or even irrelevant follow- up question.
Machine Reading Comprehension (MRC) is a clas- sical and fruitful research field with various tasks and focuses, such as extractive tasks (Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Saha et al., 2018), cloze-style and multi- ple choice tasks (Xie et al., 2018; Hermann et al.. 2015; Richardson et al., 2013; Lai al., 2017: On- ishi al., 2016), multi-document tasks (Feng al., 2021; Nguyen al., 2016; Qiu et al., 2022; Dhin- gra et al., 2017). Among them, we focus on Con- versational Machine Reading (CMR) (Sacidi et al., 2018), which is a critical but more challenging task: (1) it requires determining complex intermediate states, such as whether the document is relevant to user's query, or whether it is necessary to make clarification before answering: (2) it requires multi- ple interactions with the user through dialogue in order output final answers; and (3) the document that the agent has to consult about usually has com- plicated discourse structures describing multiple rules and constraints.
To address these issues, we propose a pipeline approach consisting of a reasoning model based on Bipartite Alignment and many-to-many Entailment (BiAE) for decision mariinn and a directly fine- tuned model for question generation 2. Our ap- proach (1) explicitly aligns the document and the user-provided information by introducing supervi- son from external model, (2) uses lightweight core decision module with only linear layers, which predicts many-to-many entailment states using aligned information and feature vectors, (3) di- rectly uses the whole document and previously asked questions to generate follow-up questions without extracting underspecified document spans. Through extensive experiments on the CMR bench- mark dataset ShARC (Saeidi et al., 2018), we demonstrate that BiAE significantly outperforms baselines with lightweight decision modules by least 12.7% in micro accuracy and the finetuned model outperforms all baselines using extract-then rewrite generation method. Our contributions can be summarized as follows:
Due to the characteristic of determining the state before responding, the pipeline method consist- ing of decision-making and question generation is more suitable for this task, which is adopted by most existing methods and achieves great suc- cess (Zhong and Zettlemoyer, 2019; Gao et al., 2020a; Gao et al., 2020b; Ouyang et al., 2021). order to improve the ability of decision-maker makinn, Ouyang et al. (2021) focus on improving the repre- sentation of the document and use relational GCNs (Schlichtkrull et al., 2018) to construct the dis- course relations of the document. Other works focus on reasoning the entailment state of docu- ment rules, which highly relevant Recognizing Textual Entailment (RTE) (Bowman et al., 2015; Mou et al., 2016; Zhang et al., 2020; Wang et al., 2021). To do this, Gao et al. (2020a) modify a Recurrent Entity Network (Henaff et al., 2017), Gao et al. (2020b) use Transformer encoder, and Zhang et al. (2022) use T5 decoder.
We propose a method for constructing bipar- tite connection, which provides explicit align- ment for document and user provided infor- mation.
We propose the BiAE model which utilizes lightweight module to make decisions by introducing explicit alignment, and a direct method which generates questions using the document and previously asked questions.
To improve the ability of question generation, existing works (Zhong and Zettlemoyer, 2019; Gao et al., 2020a; Ouyang et al., 2021) extract a span and then rewrite it into a follow-up question, which heavily relies on the quality of the extraction.
Our approach ranks the first place on the leaderboard of ShARC and outperforms the
Chttps:///githen wtwp:ttuub..om/iiien//////..

---

Aligument Entailment Based Reasoning
2. Generation
(d) Decision
di
Kilgnment
dg
D.
d,
(b) Alignment
aligmment scores
Cabbaak
D,
Pretrained
entalmeniistates
Mep
U
Language Model
(c) Entailment
Pretrained Language Model
11
Decision Layer
Mi
follow question
Mi
NO probuesiiiies
(a) Eacoding
leamabll vectors
entalimet probablittes
The overall architecture of our proposed method. Our system consists of: (I) decision classification model based on Bipartite Alignment and Entailment (BiAE) and (2) follow-up question generation model. It is noteworthy that only when the classification result of BiAE is MORE will the question generation model be activated. cat&catt in ((d) is the operation of concatenation and attention.
Figure 2:
In comparison with these works, work fo- cuses on the explicit alignment of information from both the document and the user, and employs a simpler entailment reasoning structure. Then we adopt a new approach to directly generate follow- up questions based on the document and the ques- tions asked.
use the Segbot (Li al., 2018) to divide the doc- ument into several Elementary Discourse Units (FDUs), with each EDU containing exactly one condition. Suppose a document can be divided into m EDUs and these EDUs constitute a hypothesis set D. D D DI.-.- Dm-
We divide the scenario into individual sentences using NLTK3. S S S1, catenate the follow-up question and answer in dialogue turn and add the roles the conver- sation to form a premise provided by the user. H + Tí, T2, where T; = "System:" f "Client:" ag The two parts combined form the premise set U S; T with total number of n. Encoding. As shown in Figure 2a), we pre-trained language model (PLM) to encode the hypothesis D, initial question Ug. and premise set U. We insert a special token [H]] before cach hypothesis D; and [CLLS] before both the initial question and each premise U, separate the two parts by [SEP], resulting in the input of PLM as X with the length of L. The encoding of the input sequence is
We con-
3 Methodology
The CMR task can be formulated as follows: give input X = (D, Q, S, H). ). D is the document, Q is the user's ' initial question, S' is the user's scenario, H = (Ji, a,)," (Sny Ong) St is follow-up question that was already asked, a; YESS, NO). is the dialogue history, CMR system G makes response Y G(X).
We propose a classification model based on Bi- partite Alignment and Entailment (BiAE) for in- termediate decision-making. If the decision is IR- RELEVANT, YES or NO, the system provides direct response. If the decision is MORE, fine- tuned model is used for question generation. The overall architecture of classification and generation is displayed in Figure 2.
encoding PLM(X) E ER Lxd
(1)
3.1 Segmentation and Encoding
where d is the dimension of PLM hidden state. The representation of each hypothesis d,, initial question un and premise u; is determined by lecting the vector of special tokens [H] and [CLS]
Assuming that the document set of hypothe- ses and the user-provided information is a set of premises, segmentation step is taken first to con- struct hypothesis and premise sets before encod- ing. We not only segment documents (Gao et al.. 2020b), but also make clear segmentation of user- provided information. Figure shows an example of how both parts in Figure segmented. Segmentation. Following Gao et al. (2020b), we
in encoding. More specifically,
dj Select (eneoetinnnninngg.. Index (Di)) E Bd. (2)
uj Select( (eneceignn.diggg I (Ui)) R4, (3)

---

Segmentation
between sentence vector pairs and select the hy- pothesis with the maximal cosine distance as the alignment label for each user premise."
document
D2: Statutory ...
D2: You qualify for
D3: you're... you you give...
3.3 Many-to-many Entailment
scenario
I'm going S2i I'm still
The textual entailment task involves inferring the re- lationship of hypothesis-premiss pair, which is gen- erally classified into three categories: entailment, contradiction, and neutral (MacCartney and Man- ning, 2008). Entailment refers to the case where the hypothesis can be inferred from the premise, taking Figure as an example, the user's premise "I'm still working right and I just turned in the notice." " entails the document hypothesis "(You qualify for Statutory Maternity Leave if) you give your employer the correct notice". Contradiction represents the case where the hypothesis contra- diets the premise, while neutral indicates that the relationship of the hypothesis prrethesis-premise pair is un- known irrelevant. Inspired by Mou et al. (2016), adopt four simple yet effective features to pre- diet the entailment states as shown in Figure 2.1(c). Specifically, we initialize three learnable vectors eg, ec; eN Rd represent the three entailment states, use four well-designed features to predict the probabilities of the three states, and represent the entailment state of a hypothesis-premee probabilistic weighted sum of the three vectors. This process can be expressed
conversation
T1: System: Are you Client: Yes.
Figure 3: An example of document, scenario and con-
versation history segmentation.
where Sclect(encoding,) i) denotes selecting the hidden state at index i from encoding, and Index(-) denotes the index of in the input se- quence X. We use DeBERTaV3 (He al., 2021) as the PLM.
3.2 Explicit Alignment
The objective of explicit alignment is align a document hypothesis d; that describes a certain condition to a premise uj provided by the user. We calculate the unnormalized alignment matrix Á for each hypothesis-premise pair (di, uj) by the following formula:
pair as
Ay = WA wAld ba E R.
E.g WE e W uj; d; uj; d; ujl" + by: (7)
where WA and ba are parameters of a linear layer, A.y is the hidden value of each element in the align- ment matrix. Then we use the softmax function for each to get the final alignment score matrix A shown in Figure 2bb(
Eg = so softr (ET)) E R
(8)
where o denotes element-wise product, Eij E R33 is the logits of three states, and Fig
y Gj ! their probabilities after soft- max. The final state vector for single hypothesis across all premises weighted by alignment scores is represented as
[E(E) EE( (
A = fomtaar (A,) E R",
(5)
where the element A.j E 1]. We use contrastive learning to train bipartite alignment and the loss be formulated
e;
Aij
Σ
Fig
(K) eσ Rd.
(9)
Lalign Σ H(A,, (1 Ielign)
j=1
KE(E,C.N)
(6)
The expression for the entailment loss is
Lentoil
Σ ((Eij jenfail), )
(10)
where [p, q) represents the cross-entropy func- tion, 1 polign is the weakly supervised alignment la- bel.
(K))P
where P is the set of premise-hypothesis pairs, tentast denotes the weakly supervised entailment label. We adopt the three-state label proposed by Gao et al. (2020a) make such supervision.
In order construct geligm, we use Sentence- BERT (Reimers and Gurevych, 2019) compute the semantic similarity between the user provided premise set and the document hypothesis set of- fline. Specifically, we calculate the cosine distance
(j
"This method shows 92% consistency with manual selec- tion on subset of 100 randomly selected samples.

---

3.4 Decision Classification
4 Experiments
The decision unit in Figure 2.1(d) integrates all semantic vectors and all entailment states of the hypothesis set to obtain a holistic representation of the entire document, using the attention mecha- nism,
4.1 Dataset and Metrics
Dataset. Our experiments carried out on the CMR benchmark dataset ShARC (Saeidi al., 2018), which was crawled from government legal documents across 10 unique domains. This dataset (11) comprises 35% bullet point documents (e.g. the ex- ample shown in Figure I), while the rest are regular documents. The dialogues are constructed based (13) on annotation protocol (Saeidi et al., 2018) in the form of question-answer pairs with (or not) extra scenario. The sizes of the train, development, and test sets are 21,890, 2,270, and 8,276, respec- tively. The test set is withheld and not publicly available.
d: = Wa Wa eij² + ba E R, a = soflmar( (a) RR" az C [0,
(12)
s as [i ej] c Rd
Subsequently, the representation s is employed to generate the probabilities p of four aforementioned decision categories together with the semantic rep- resentation of initial question Uq. And the corre- sponding decision loss is
Metrics. For decision decision-making, making, Micro and Macro Accuracy are used for evaluation, whereas BLEU (Papineni et al., 2002) is used for evaluating ques- tion generation.
P - w + bD E R
(14)
Leccc = (ssoftmaz()))
(15)
4.2 Baselines
where 1d is the true decision label. Furthermore, bi- partite alignment and many-to-many entailment are employed to augment the decision-making process, and joint loss function is introduced incorporated with weight parameter X,
(1) Baseline-NMT (Saeidi al., 2018) is an end- to-end NMT-copy model based on LSTM and GRU, (2) Baseline-CM (Saeidi et al., 2018) is a pipeline combined model using Random Forest, Surface Lo- gistic Regression and rule-based generation. (3) BERTQA (Zhong and Zettlemoyer, 2019) is an extractive QA model. (4) UracNet (Verma et al., 2020) uses artificially designed heuristic-bases pat- terns. (5) BiSon (Lawrence et al., 2019) utilizes placeholders for bidirectional generation rather than autoregressive unidirectional generation. (6) 3 (Zhong and Zettlemoyer, 2019) performs rule extraction from documents, rule entailment from user information, and rule editing into follow-up questions jointly. (7) EMT (Gao et al., 2020a) uses gated recurrent network with augmented mem- ory that updates rule entailment state for decision- making by sequentially reading user information. (8) DISCERN (Gao et al., 2020b) subdivides a document into fine-grained EDUs and employs an inter-sentence transformer encoder for entailment prediction. (9) DGM (Ouyang et al., 2021) primar- ily employs relational GCNs (Schlichtkrull et al., 2018) to model the rhetorical structure of docu- ments for decision-making, (10) ET5 (Zhang et al., 2022) proposes an end-to-end generation approach with duplex decoders and shared encoder based on TS.
C dL + Lalign + Centail-
(16)
3.5 Question Generation
If the predicted decision is MORE, the system is required to propose follow-up question to obtain new premises for clarification and continuing the reasoning process. Although is intuitive to ex- tract a hypothesis with a neutral entailment state and then rewrite it into clarification question, the question generation process heavily depends on the extracted hypothesis. Current language mod- els, such as T5 (Raffel et al., 2020) and BART (Lewis al., 2020), have strong generative capa- bilities. Hence, we directly fine-tune T5 with the entire document D and the sequence of previously asked questions F = f1, f2...... ..., while treating the ground-truth follow-up question as the generation target. We use the generation loss implemented in Raffel et al. (2020) for training.
We also perform data augmentation to alleviate data sparsity. Specifically, we reduce the dialogue history by one turn to construct F for the data with decision labels other than MORE, and use the question in the last turn as the target question to be generated.
Baselines (8), (9) and (10) use heavyweight mod- ules for core sieciision-making Please refer to Ap-

---

Held-out Test Set
Dev Set(B)
Dev Set(L)
Model
mic
mac
B-1
B-4
mic
mac
mic
mac
NMT (Sacidi et al., 2018)
44.8
42.8
34.0
7.8
CM (Sacidi et al., 2018)
61.9
68.9
54.4
34.4
BERTQA (Zhong and Zettlemoyer, 2019)
63.6
70.8
46.2
36.3
63.6
70.8
UrcaNet (Verma et al.. 2020)
65.1
71.2
60.5
46.1
BiSon (Lawrence et al., 2019)
66.9
71.6
58.8
44.3
66.9
71.6
E3 (Zhong and Zettlemoyer, 2019)
67.6
73.3
54.1
38.7
67.6
73.3
EMT (Gao et al., 2020a)
69.1
74.6
63.9
69.1
74.6
DÍSCERN* (Gao et al., 20206)
73.2
78.3
64.0
49.1
74.9
79.8
77.2
80.3
ETS. (Zhang et al., 2022)
76.3
80.5
69.6
55.21
75.9
80.4
78.6
82.5
DGM (Ouyang et al., 2021)
77.4
81.2
63.3
48.4
75.5
79.6
78.6
82.2
BiAF(ours)
77.9
81.1
64.7
51.6
76.2
80.5
80.5
83.2
Table 1: Results on the held-out ShARC test and dev set. B-1, B-4, mic and mac are short for BLEUI, BLEU4, Micro Accuracy and Macro Accuracy. We conduct 5-fold cross validvaaii t-test experiments on the dev set since the test set is reserved, and the results show that < 0.05. Models with use heavyweight decision modules and ET5 uses dual decoders. The performance of each method using base and large version pre-trained language models the development set is marked by B and L, respectively.
pendix A for implementation details.
Model
paras
mic
mac
B-1
B-4
5 Results and Analysis
FastChat-TS
3B
12.7
29.8
+ +2-shot
3B
20.6
35.9
50.9
29.5
5.1 Main Results
Alpaca
7B
51.8
37.7
+2-shot
41.8
30.2
We report the results of BiAE and baselines on the blind held-out test set of the ShARC dataset in Ta- ble 1. BiAE significantly outperforms the baselines with lightweight decision modules, with at least 12.7% improvement in Micro Accuracy and 8. 8.7% improvement in Macro Accuracy. Compared baselines with heavyweight decision modules, BiAE achieves comparable results while greatly re- ducing the parameters from 27M (decision module DGM) to only 31.7K. Moreover, BiAE achieves statc-of-the-ar performance in terms of Micro Ac- curacy. For generation, T5BiAE outperforms all the methods using span extraction. Note that the gen- eration metrics are only calculated when both the classification decision and the true label is MORE. Therefore, since the test sets differ among the meth- ods used calculate generation scores, the results are not strietly comparable.
Vicuna
13B
24.8
39.0
11.9
7.7
+ 2-shot
13B
33.1
41.9
33.8
12.0
GPT-3.5-turbo
175B
35.6
45.2
58.6
37.1
2-shoo
175B
41.1
50.8
45.2
29.5
Table 2: Results of large language models on the ShARC development set. B-1, B-4, mic and mac are short for BLEUI, BLEU4, Micro Accuracy and Macro Accuracy.
0.7 (base) and 1.0 (large) than the strong bascline DGM.
We also report class-wise Micro Accuracy of BiAE and several baseline models on four different categories in Appendix B. BiAE greatly improves the abilities of deterministic decision-making (YES and NO).
To compare the decision-making ability of dif- ferent methods using base and large pre-trained language models fairly, we also report the results the development set in Table 1. Regardless of whether based on base or large pre-trained language model, BiAE significantly outperforms the baselines with lightweight decision modules, mcanwhile, achieves higher Micro Accuracy of 1.0 (base) and 1.9 (large) and Macro Accuracy of
Compare with Large Language Models. We have evaluated the performance of FastChat-TS (Zheng et al., 2023), Alpaca (Taori et al., 2023), Vicuna (Chiang al., 2023) and GPT-3.5-turbo (Ouyang et al., 2022) on the ShARC development set, and the results with 0-shot and 2-shot demon- strations are reported in Table 2. The four large language models show significantly lower results compared to our model. Please refer to Appendix

---

Model
micro-ace
macro-acc
Subset
#Count
Subset
#Count
BiAE(ELECTRA)
76.2
80.3
Bullet Point
999
Regular
1271
BiAE(DeBERTaV3)
76.2
80.5
Scenario
1839
NoScenario
431
w/o Align
74.3
78.9
History
1509
NoHistory
761
w/o Entail
74.1
78.3
All
2270
w/o Align & Entail
72.6
76.9
Table 5: Subset Size of ShARC Development Set.
Table Results of ablation experiments decision rea- soning based on the ELECTRA-bars and DeBERTaV3: base models on the ShARC development set.
State
B
02
Qf
Q5
Q
Success
0.45
0.75
0.29
0.50
0.80
1.00
Fail
0.24
0.72
0.24
0.60
0.75
0.89
Model
BLEU1
BLEU2
BLEU3
BLEU4
B5BE 62.8
55.8
51.7
48.6
Table 3 and some statisties of o² denotes the variance and Qn denotes the n-th quartile.
w/o aug.
61.4
54.1
49.8
46.6
Table 4: Results of ablation experiments for follow-up generation on the ShARC development set.
simple case study of generation.
5.3 Interpretation Document and User Information
for the prompt template used and some output cases.
To investigate the comprehension ability of BiAE for document and user information, we divide the development set into six subsets based whether the documents contain bullet points and whether the user information includes scenario or conversa- tion history. The sizes of each subset are shown Table 5.
5.2 Ablation Study
Effect of Alignment and Entailment. To plore the effects of bipartite alignment and many-to- many entailment on decision-making we conduct ablation experiments based on DcBERTaV3 on the development set, as shown in Table 3. The results indicate that both alignment and entailment have an impact on decision-making and the im- pact is greater when both considered together. Simultaneously removing both alignment and en- tailment losses leads decrease of 3.6 points in Micro Accuracy and 3.4 points Macro Accuracy. Furthermore, we also conduct ablation experiments on the encoders, the results based on ELECTRA base exhibits slight decrement when compared to those based on DeBERTaV3-bass This demon- strates that the improvement in reasoning perfor- mance is minimally influenced by encoders and mainly comes from our incorporation of explicit alignment and the modeling of many-to many-to-many en- tailment in the core decision module.
We calculate the Micro and Macro Accuracy of the strong baseline DGM and BiAE different subsets, as shown in Figure 4(a). The results show that BiAE performs better all subsets. Under- standing bullet point documents is more challeng- ing than regular documents, but BiAE reduces the gap by improving the Micro Accuracy of bullet point documents by 3.3 and regular documents by 2.2. Understanding scenarios is still a significant challenge (Sacidi et al., 2018; Gao et al., 2020b; Ouyang et al., 2021) because subsets without sce- narios have significantly higher accuracy than those with scenarios, BiAE achieves the most significant improvement (+3.9) on subsets containing history. These performance improvements are likely due our splitting of user information and the explicit alignment between documents and user informa- tion.
Effect of Data Augmentation for Generation. Only the data with the decision label MORE re- quires generating follow-up questions, accounting for 31.08% of the training set. We perform data augmentation address this data sparsity issue and organize ablation experiments, results are shown in Table Data augmentation improves all gen- cration metrics, with increases of 1.4, 1.7, 1.9 and 2.0 for BLEUI- -4, respectively. Appendix is a
5,4 Many-to-many Entailment In BiAE, the final decision based on: the en- coding of user initial question, the encoding and the final entailment state of each hypothesis in a document. To investigate the holistic textual entail- ment of the document hypothesis set on the final

---

0.95
0.95
mimieo
BiAT/micro
800
Success
BiABimaen
0.90
B'mrro
700
Fail
DGM/miero
DGMmmiero
Success
600
00
DGM/mmarr
Accuracy
0.85
Count
500
Fail
300
200
0.75
100
Bullet Point
Regular Document
All
Secnsrro NoSoemsrio History Noftisory
All
0.0
0.2
0.4
0.6
0.8
1.0
User Information
(b)
Figure (a)Accuracy of BiAE and DGM(St Baseline) on Different Subsets. (b)Histogram and Density Estimation of the Distribution under Successful and Failed Prediction States.
decision, we define and ( as follows:
underspecified document spans. BiAE significantly reduces the parameters in the core decision-making (17) module and achieves results comparable to strong baselines. Extensive experiments demonstrate the effectiveness of our framework. Through analysis, we believe that improving the ability of entailment reasoning among the overall hypotheses of doc- ument crucial for enhancing decision-makinn ability.
E I1s s!
m
JP E I(a 1.0)
(18)
[P]
where m is the number hypotheses docu- ment, P is a subset, sp and sy is the predicted and constructed label for the i-th hypothesis, I(-) de- notes the indicator function. measures the degree of correctness in entailment reasoning for an indi- vidual document and 3 represents the proportion of documents with perfect entailment reasoning in subset. Figure 4(b) illustrates the distribution and density estimation curve of under successful and failed prediction states. The statistics Table 6 and Figure 4(b) show that, compared with failed predic- tions, the values of a are more concentrated around 1.0 in successful predictions, indicating deeper un- derstanding of the entailment of all hypotheses in the document (corresponding to larger values), and hence leading higher prediction accuracy. In addition, gSueess is much larger than FFail while the difference between a Sucess and a Fail is small, indicating that the final decision not only requires one-to-one entailment but also relies on accurate many-to-many entailment of all hypotheses in the document.
Limitations
Although our approach exceeds the previous state- of-the-art model on the main metries, our work still has two limitations.
(1) We conduct experiments on the ShARC dataset, which is the benchmark for the CMR task but consists of relatively short documents. Due to limited computational resources, it is challenging for us to organize experiments with longer docu- ments larger datasets. However, we believe that BiAE has the potential perform well on longer documents. We will strive scale up our computa- tional resources and validate our approach on such datasets.
(2) Our proposed method for explicit alignment of documents and dialogues is based on semantic similarity. Although it demonstrates effectiveness in experiments, acknowledge that various knowledge bases, such knowledge graphs. provide alignment information beyond the seman- tic level. In the future, we will explore better align- ment methods by leveraging diverse knowledge bases.
6 Conclusion
We propose new framework for Conversational Machine Reading in this paper. Our classification model, BiAE, leverages many-to-many entailment reasoning, enhanced by explicit alignment, for decision-making And our T5BIAE is directly finc- tuned for generating follow-up questions to clarify

---

Ethics Statement
Yifan Gao, Chien ShiinnSheng Wu, Jingjing Li, Shafiq Joty, Steven C.H. Hoi, Caiming Xiong. Irwin King, and Michael Lyu. 2020b. Discern: Discourse-aware tailment reasoning network for conversational ma- chine reading. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 2439-2449, Online. As- sociation for Computational Linguistics.
Our research focuses on conversational machine reading. The dataset we used is publicly available and consists of documents from government web- sites and dialogues from crowdsoureing workers who have received fair and legal remuncration. We utilize open-source pre-trained models, including large language models. GPT-3.5 is only used for simple evaluation. We believe that work does not involve personal privacy or societal biases.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with tradient-disentaun embedding shar- ing.
A Acknowledgements
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the world state with recurrent entity networks. In Sth Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Truck Proceedings. OpenReview.net.
The work is partially supported by State Grid Corporation of China's Science and Technology Project "Construction of Electric Power Cogni- tive Large Model and key Techniques of Its Ap- plications on Operation. Maintenance and Detec- tion" (Project No: 5700-20231 1000-20231388A-1111111 3288A-1-1-NNN We thank the anonymous reviewers for their insight- ful comments and Max Bartolo for executing the evaluation on the reserved test set.
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt. Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th Interna- tional Conference on Neural Information Processing Systems Volume 1, NIPS' 15, page 1693-1701, Cam- bridge, MA, USA. MIT Press.
References
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon. Portugal. Association for Compu- tational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785- 794, Copenhagen, Denmark. Association for Compu- tational Linguistics.
Bhuwan Dhingra, Kathryn Mazaitis, and William W. Cohen. 2017. Quasar: Datasets for question answer- ing by search and reading. CoRR, abs//10070033 009004.
Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. MultiDoc2Dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference Empirical Methods in Natu- ral Language Processing, pages 61 6162-6176, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. 2019. Attending to future tokens for bidi- rectional sequence generation. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-JJCNLP) pages 1-10, Hong Kong. China. Association for Computational Linguisties.
Yifan Gao, Chien-Sheng Wu, Shafiq Joty, Caiming Xiong, Richard Socher, Irwin King, Michael Lyu, and Steven C.H. Hoi. 2020a. Explicit memory tracker with coarse-to-fin fine reasoning for conversational ma- chine reading. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 935 935. Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising soquence-to-sqqenne pre-training

---

for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Pennsylvania, USA. Association for Computational Linguistics.
Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, QiaoQiao She, Jing Liu, Hua Wu, and Haifeng Wang, 2022. DuReader-retrieval A large-scale Chinese bench- mark for passage retrieval from web search engine. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5338, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Jing Li, Aixin Sun, and Shafiq R. Joty. 2018. Segbot: A generic neural text segmentation model with pointer Inter-
network. In Proceedings of the Twenty-Sevent national Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4166-4172 ojcai.org.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21(140):1-61 176
Bill MacCartney and Christopher D. Manning. 2008 Modeling semantic containment and exclusion in nat- ural language inference. In Proceedings of the 22nd International Conference on Computational Linguis- rics (Coling 2008), pages 521-528, Manchester, UK. Coling 2008 Organizing Committee.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguistics.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 130-136, Berlin, Germany. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNNLP-IJCNTP)) pages 3982 3992, Hong Kong, China, Association for Com- putational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop Cognitive Computation: Integrat- ing neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Infor- mation Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016. volume 1773 of CEUR Workshop Proceedings. CEUR-WS WS..rg.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension text. In Pro- ceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computa- tional Linguistics.
Takeshi Onishi, Hai Wang. Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large- scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 2230-2235, Austin, Texas. Association for Computational Linguistics.
Marzich Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh. Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 15:27730-27744
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683- 1693, Melbourne, Australia. Association for Compu- tational Linguistics.
Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. 2021. Dialogue graph modeling for conversational machine reading. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 3158-3169, Online. Association for Computational Linguistics.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia,
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem. Rianne van den Berg. Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. The Semantic Web, pages 593- 607, Cham. Springer International Publishing.

---

Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-followinn model. Stan- ford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03//1 3/apppa himl, 3(6):7.
Annual Meeting of the Association for Computational Linguistics, pages 2310-2320, Florence, Italy. Asso- ciation for Computational Linguistics.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehen- sion dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages -000, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- lems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Dan- ish Contractor, Harshit Kumar, and Sachindra Joshi. 2020. Neural conversational QA: Learning to rea- son vs exploiting patterus. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7263-7269, Online. Association for Computational Linguistics.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. ArXiv, 1eee22044.44999
Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale cloze test dataset created by teach- ers. In Proceedings of the 2018 Conference Em- pirical Methods in Natural Language Processing, pages 2344-2356, Brussels, Belgium. Association for Computational Linguistics.
Xiao Zhang, Heyan Huang, Zewen Chi, and Xian-Ling Mao. 2022. ET5; A novel end-to-end framework for conversational machine reading comprehension. In Proceedings of the 29th International Confer- ence Computational Linguistics, pages 570-579, Gyeongju, Republic Korea. International Commit- tee on Computational Linguistics.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020. semantics-aware bert for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(00))9628-966
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang. Joseph E. Gonzalez, and lon Stoica. 2023. Judging llm-as-a-judge with mt-hench and chatbot
Victor Zhong and Luke Zettlemoyer. 2019. E3:
Entailment-driven extracting and editing for conver- sational machine reading. In Proceedings of the 57th

---

A Implementation Details
C Prompt Template and Examples
We use the O-shot prompt template as shown Figure 5 and 2-shot prompt template as shown in Figure 8 to create inputs for large language mod- els. Figure 6 and Figure 7 are two O-shot output examples.
Adam optimizer (Kingma and Ba, 2015) and lin- ear schedule with warmup used for the train- ing process. For the decision-makien task, BiAE is fine-tuned based on DeBERTaV3 for epochs with dropout rate set to 0.3. Batch size is set to 20 for base model and 8 for large model. We try various loss weights in Equation 16, including À = 0.5, 1.0, 2.0, 3.0, and find that 2.0 works best. The learning rates 1e-3 2 5e and 1e-1 are attempted, and we find that 5 is optimal for base model, while 2 is best for large model. For the question generation task, T5 large fine-tuned for 3 epochs, with batch size set to 6, learning rate set to 2e and all other parameters set to default. All experiments are conducted on a NVIDIA GeForce RTX 3090. It takes 6-7 hours to fine-tune BiAE (DeBERTaV3-large for 5 epochs, 1-2 hours to fine- tune Ta-lerge (without data augmentation) for epochs, and 7-8 hours to fine-tune Ta-large (with data augmentation) for 3 epochs. The data augmen- tation is performed during the construction of the Dataset and the time required for negligible.
D Generation Case Study
We conduct a case study on 100 samples with the lowest BLEU scores to analyze the reasons, and the main categories are summarized as follows:
Incomplete generation of questions: 2%
Generated questions lacking key words: 2%
Generated questions lacking non-key words:
8%
Generated questions with the same semantics as the true questions but different expressions: 8%
Generated questions unrelated to the docu-
ment: 13%
Redundant generation of questions (already asked or unnecessary): 25%
B Deterministic and Uncertainty Reasoning
Generated questions describing other reason- able and unasked conditions: 42%
Table 7 shows Micro Accuracy of our model and several baseline models four different categories. All models demonstrate high reasoning abilities for IRRELEVANT (over 95%). Compared to the strong baseline, BiAE performs slightly worse in reasoning uncertainty problems (MORE, the rea- soning ability to clarify questions), but greatly improves the abilities of deterministic decision- making (YES and NO). This phenomenon may stem from the fact that the selected entailment fea- tures exhibit a higher sensitivity towards determin- istic reasoning.
should be noted that evaluating the generation task solely based on BLEU is not sufficient. Cat- egories and 7 (50% in total) represent accept- able generated questions but receive lower BLEU scores.
You are customer service consultant. Give proper response to answer the initial question proposed by the client based the given document. You can only reply 'Irrelevant', "Yes', 'No" directly or ask follow-up question without explanation. Don't explain!
Document:
Model
IRRELEVANT
YES
NO
MORE
Total
[document]
BERTQA
96.4
62.6
Initial question: (scenario] (initial question]
96.4
65.9
70.6
60.5
68.0
UracNet
95.7
63.3
68.4
58.9
The following is the conversation that has already happened:
EMT
98.6
70.5
73.2
70.8
74.2
You: [follow up question 1]
Discer
71.9
75.8
73.3
75.2
Client: (follow-up answer 1]
DGM
97.8
75.2
77.9
76.3
77.8
BiAE(ours)
97.1
84.1
80.5
71.2
80.5
Figure S: The 0-shot prompt template for large language models.
Table 7: Class-wise Accuracy of BiAE and baselines the ShARC development set.

---

You customer service consultant. Give proper response to answer the initial question proposed by the client based on the given document. You can only reply 'Irrelevant', "Yes' 'No' directly or ask follow-up question without explanation. Don't explain!
You customer service consultant. Give proper response the initial question proposed by the client based the given document. You only reply 'Ireeeeant', "Yes', 'No' directly ask follow-up question without explanation. Don't explain!
Document:
## Statutory Maternity Pay
To qualify for SMP must:
Document:
earn on averoge at least Eug week
you are a female Vietnam Veteran child who has birth defect or you are child of a female Vietnam Veteran with a birth defect, child may eligible for VA-financed health care.
give the correct notice
give proof you're 're pregnant
have worked for your employer continuously for at least
26 weeks continuing into the 'qullifying week' the 15th week before the expected week of childbirth'
Initial question: I registered as a single not knowing my son would have a birth defect. Is my child eligible for VA-financed health care?
Initial question: have worked for Her Majesty's Revenue and Customs for the past three years without single day off and am expected to have my baby in about nine weeks. Do qualify for SMP?
The following is the conversation that has already happened:
You: Are you a female Vietnam Veteran?
Client: Yes.
The following is the conversation that has already happened: You: Do you earn on average at least 2113 a week?
You: Does your child have a birth defect?
Client Yes.
Client: Yes.
You: Did you give the correct notice?
True answer: Yes.
Client: Yes.
Output FastChat-TS; Irrelevant.
Output Alpaca: Yes, your child cligible for financed
True answer: Did give proof you're pregnant? Output FastChat-TS: Irrelevant.
health care.
Output Vicuna:
Yes, your child may be eligible for Vi-financed
Output of Alpaca: Yes, you qualify SMP.
health care.
Output of Vicuna: Based on the information provided, appears that the client qualifies for SMP. However, it important to note that the client must also provide proof that they are pregnant and have worked for their employer continuously for at least 26 weeks into the qualifying week.
Output of GPT-3.5-turbas Yes.
Figure 6: An output case of large language models (0- shot).
Output of GPT-3.5-turbo, You: Have provided proof that you're pregnant?
Figure 7: An output case of large language models which requires generating follow-up question (0-shot).

---

You are customer service consultant. Give proper response to answer the initial question proposed the client based on the given document. can only reply 'Irrelevant', ask follow question without explanation. Don't explain!
directly
#*# Example
Document:
Eligibility
You must have:
unconditional offer of place with licensed Tier sponsor
enough money support yoursel and pay for course the amount will vary depending on your circusstances.
Initial question: have un unconditional offer for course with licensed sponsor who ponel of experts eligible Tier (General) student viso?
The following is the already happened conversation: You: Do you have an unconditional offer a place on course with a licensed Tier sponsor?
Client: Yes.
you enough money support yourself and pay for the course?
Client: No.
The final response: No.
Example
Document:
order to quality for this benefit. program, your husiness private non-profit organization must have sustained physical domage and be locoted in a disaster declored county.
Initial question: My housing benefit doesn't currently cover my rent. progrom meet my needs?
No conversation has taken place.
The final response: you own business or private non non-profit organization?
proper response for the following initial question. only reply 'Irrelevan", "Yes', 'No' directly ask follow-up question without exp lnatton..
Document: [documeni]
initial question: [scenorio) finitial question]
The following the conversation that has already happened: You: (Sollow-up question 3)
Client: (follow up onswer 1J
The final response:
Figure 8: The 2-shot prompt template for large language models.


Reviews:
[{"review_id": "EoMjpfRVcg", "reviewer": "Reviewer_DMCN", "summary": "", "questions": "+ In comparison with large language models, No examples or demonstrations are employed in the template as shown in Figure 5. Will there be a surge of performance if several shots of examples are given?\n+ Why the document is the set of hypotheses while the dialogue history is treated as a premise?\n+ The formula and the notation in the manuscript are a little bit cluttered. I suggest deferring the implementation details to the appendix.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns and answer your questions point by point.\n\n# Reasons To Reject Point 1:\nThank you very much for your thoughtful suggestions. We value fair comparisons a lot. Therefore, for the reasoning model (BiAE), we conducted ablation experiments based on ELECTRA as the encoder on the development set (results shown below). The experimental results indicate that the reasoning (decision) results based on ELECTRA are slightly lower than those based on DeBERTa, but still higher than DGM. This not only allows us to make a fair comparison with the previous SOTA method (DGM), but also demonstrates that the improvement in reasoning performance mainly comes from the introduction of explicit alignment and modeling of many-to-many entailment in our core decision module. It should be noted that the parameter size of our core decision module (31.7K) is only 0.06% of DGM (54M). One of our main contributions lies in significantly reducing the parameter size of the core decision module while achieving comparable results to DGM. Additionally, we have conducted significance test using 5-fold cross-validation with the result of p < 0.05 in a t-test. We will incorporate these ablation experiments into the revised version. Once again, thank you for your feedback.\n| Model               | Micro Acc. | Macro Acc. |\n|---------------------|------------|------------|\n| DGM (electra-base)  | 75.5       | 79.6       |\n| BiAE (electra-base) | 76.2       | 80.3       |\n| BiAE (DeBERTa-base) | 76.2       | 80.5       |\n| DGM (electra-large) | 78.6       | 82.2       |\n| BiAE (electra-large)| 79.9       | 82.8       |\n| BiAE (deBERTa-large)| 80.5       | 83.2       |\n\n\n# Reasons To Reject Point 2:\nWe conducted a case study on 100 samples with the lowest BLEU scores to analyze the reasons, and the main categories are summarized as follows:\n* (1)\tIncomplete generation of questions: 2%\n* (2) Generated questions lacking key words: 2%\n* (3) Generated questions lacking non-key words: 8%\n* (4) Generated questions with the same semantics as the true questions but different expressions: 8%\n* (5) Generated questions unrelated to the document: 13%\n* (6) Redundant generation of questions (already asked or unnecessary): 25%\n* (7) Generated questions describing other reasonable and unasked conditions: 42%\n\nAs you have rightly pointed out, evaluating the generation task solely based on BLEU is not sufficient. Categories 4 and 7 (50% in total) represent acceptable generated questions but receive lower BLEU scores. We will explain this issue adequately in the revised version. Thank you for your thoughtful considerations.\n\n# Question 1:\nThank you for your question. The purpose of evaluating on the large language models was to assess its ability to solve the CMR task. Since the best results on the large language models (GPT-3.5, Micro-Acc 35.6 Macro-Acc 45.2) were still far below our results (Micro-Acc 80.5, Macro-Acc 83.2), we did not conduct additional few-shot experiments before submitting the paper. However, your question has motivated us to perform few-shot experiments and supplement the results for the large language models, which will make our findings more rigorous.\n\nThe results of the 2-shot experiments on the large language models are as follows. The results show that providing examples can improve the performance, but the best results are still far below ours.\n| LLM              | Micro Acc. | Macro Acc. | BLEU-1 | BLEU-4 |\n|--------------------|------------|------------|--------|--------|\n| FastChat-T5 (3B)   | 12.7       | 29.8       | -      | -      |\n| + 2-shot           | 20.6       | 35.9       | 50.9   | 29.5   |\n| Alpaca (7B)        | 51.8       | 37.7       | -      | -      |\n| + 2-shot           | 41.8       | 30.2       | -      | -      |\n| Vicuna (13B)       | 24.8       | 39.0       | 11.9   | 7.7    |\n| + 2-shot           | 33.1       | 41.9       | 33.8   | 12.0   |\n| GPT-3.5 (175B)     | 35.6       | 45.2       | 58.6   | 37.1   |\n| + 2-shot           | 41.1       | 50.8       | 45.2   | 29.5   |\n\n\n# Question 2:\nWe have excerpted the definition proposed by previous researchers: \"Textual entailment is defined as a directional relationship between pairs of text expressions, denoted by T (the entailing 'Text') and H (the entailed 'Hypothesis'). We say that T entails H if humans reading T would typically infer that H is most likely true\". (Please refer to Ido Dagan; Dan Roth; Fabio Zanzotto; Mark Sammons, Morgan & Claypool, 2013. Recognizing Textual Entailment: Models and Applications) A hypothesis is a statement that can be tested and verified through empirical evidence, while a premise is a statement used to support or justify a conclusion in an argument. In the CMR task, evidence is continuously provided through dialogue to determine if the user-provided information can entail the document. For the sake of convenience, we consider the document as a set of hypotheses to be verified, and the information provided by the user as premises to support the verification of the document. \n\n# Question 3:\nThank you for your feedback. We will carefully review all the formulas and expressions again, and strive to present them in a more concise manner whenever possible. Regarding the suggestion of placing implementation details in the appendix, we have also considered this before submitting the paper. Once again, thank you for your suggestions, and we will reorganize the placement of this section in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper introduces a novel framework for CMR that explicitly aligns the document and user-provided information, which greatly improves decision-making and question generation.\n+ The use of many-to-many entailment reasoning and lightweight decision modules in the pipeline framework is innovative and practical.\n+ The experimental results show that the proposed method outperforms baselines and achieves state-of-the-art performance on the ShARC dataset.", "reasons_to_reject": "+ The potential unfair comparison between the proposed model and the baselines. The proposed method uses a deBERTa for encoding and finetunes a T5 for question generation, which are nearly state-of-the-art backbone models. However, the baseline methods use much weaker backbones, rendering the comparison meaningless.\n+ For evaluating question generation, I doubt that simple BLEU metric may be not enough. Human evaluation or at least case study is necessary.", "questions_to_authors": "+ In comparison with large language models, No examples or demonstrations are employed in the template as shown in Figure 5. Will there be a surge of performance if several shots of examples are given?\n+ Why the document is the set of hypotheses while the dialogue history is treated as a premise?\n+ The formula and the notation in the manuscript are a little bit cluttered. I suggest deferring the implementation details to the appendix.", "paper_topic_and_main_contributions": "The paper proposes a pipeline framework for Conversational Machine Reading (CMR) that addresses the issue of alignment between the document and user-provided information. The proposed method achieves state-of-the-art results on the CMR benchmark dataset ShARC.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VPtEWiomwN", "reviewer": "Reviewer_xtUY", "summary": "", "questions": " Question A: What exactly is meant by the user-provided information and user scenarios (168-169)? It seems that the sample dialogue in Figure 1 is not enough to introduce the concept.\n Question B: How do you define the difference between the heavyweight and lightweight decision modules? You mention that there is a \"significant reduction in decision module parameters\" (lines 108-109), but how big is the difference?\n Question C: Why was DeBERTaV3 chosen for the experiments and how does it compare to the models used in the baseline methods?\n Question D: How do you generate the direct responses in case of IRRELEVANT, YES, NO labels (lines 175-177)?\n Question E: What is the meaning of \"the weakly supervised alignment label\" (lines 248-149)? How is it computed?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thought-provoking comments. We will explain your concerns and answer your questions point by point.\n\n# Question A & Reasons To Reject Point 1 Bullet point 1 :\nIn lines 41-44, we explained that user-provided information includes user scenarios and conversation history. In this regard, we assumed that no additional explanation is needed for user scenarios as the statement itself should be sufficient for understanding. However, your concern is also valid and worth careful consideration. We will address this part specifically in the revised version and provide further explanations.\n\n# Question B:\nWe differentiate between heavyweight and lightweight based on the parameter size and structural complexity of the core decision module. From a structural perspective, DGM utilizes a combination of graph networks and Transformers, ET5 uses the T5 Decoder, and DISCERN employs Transformers. In comparison, we only utilize a few linear layers. In terms of parameter size, as mentioned in lines 420-421, our decision module has a parameter size of 31.7K, which is only 0.06% of DGM's parameter size(54M).\n\n# Question C:\nThank you very much for your question. We chose DeBERTaV3 because of its superior performance on tasks such as RTE. We recognize the importance of fair comparisons, so for the reasoning model, we conducted ablation experiments using an encoder based on ELECTRA on the development set (results below). The experimental results show that the reasoning results based on ELECTRA are slightly lower than those based on DeBERTa but still higher than DGM. This approach allows us to make fair comparisons with the previous SOTA model, DGM, while also demonstrating that the improvement in reasoning performance primarily comes from introducing explicit alignment and the modeling of many-to-many entailment in our core decision module. We will include these ablation experiment results in the revised version. Thank you again for your question.\n| Model               | Micro Acc. | Macro Acc. |\n|---------------------|------------|------------|\n| DGM (electra-base)  | 75.5       | 79.6       |\n| BiAE (electra-base) | 76.2       | 80.3       |\n| BiAE (DeBERTa-base) | 76.2       | 80.5       |\n| DGM (electra-large) | 78.6       | 82.2       |\n| BiAE (electra-large)| 79.9       | 82.8       |\n| BiAE (deBERTa-large)| 80.5       | 83.2       |\n\n\n\n# Question D:\nIf the decision result is IRRELEVANT, YES or NO, we will directly reply with the corresponding label itself: \"Irrelevant\", \"Yes\", or \"No\".\n\n\n# Question E & Reasons To Reject Point 1 Bullet point 2 :\nThank you for your question. \"The weakly supervised alignment labels\" refer to the labels used for training explicit alignment, which are not provided in the dataset and are constructed using SentenceBERT. Specifically, we compute the cosine distance between all (document rule, user provided information) pairs. For each piece of user provided information, we select the document rule with the maximum cosine distance as its alignment label. Since the labels constructed using this method contain noise (the document rule with the maximum cosine distance may not necessarily be the one that truly needs alignment), they are referred to as \"the weakly supervised alignment labels\". Additionally, we validate the quality of the weakly supervised alignment labels in two ways. First, we use a heuristic approach to make such labels. Specifically, we calculate the edit distance between all (document rule, user provided information) pairs and assign the document rule with the minimum edit distance as the label for each piece of user provided information. The labels constructed using the heuristic approach show 85% consistency with the labels constructed using SentenceBERT. Second, we randomly selecte 100 samples with labels constructed using SentenceBERT and manually evaluate their consistency. Among them, 92% of the constructed labels match the manually selected ones.\n\nIn addition to using SentenceBERT to assign hard labels based on the highest cosine distance, we also conducted an explicit alignment experiment using normalized soft labels and the KL divergence loss function. The results are as follows: \n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| Hard Label + Cross Entropy | 76.2       | 80.5       |\n| Soft Label + KL Divergence           | 70.3      | 74.9      |\nIn future work, we will continue to explore better alignment methods.\n\n# Reasons To Reject Point 1 Bullet point 3 :\nThe method we proposed consists of a pipeline framework with a reasoning model and a generation model. Data augmentation is only used during the training of the generation model. We briefly mentioned data augmentation in lines 338-342, but we may not have explained it clearly enough. Here, we would like to provide a more detailed explanation.\n\nFirstly, the reasoning model produces four types of decision states: YES, NO, IRRELEVANT, and MORE. The generation model is only activated when the decision result is MORE, and only the data with true decision labels as MORE are used for generation. This subset of data accounts for only 31.08% of the training set.\n\nDuring the generation phase, in order to utilize more training data, we perform data augmentation. Specifically, for the data with true labels of IRRELEVANT, YES or NO, assuming their dialogue history consists of n turns, denoted as H = [(q1, a1), ..., (qn, an)], we shorten the dialogue history by one turn to obtain a new dialogue history, H' = [(q1, a1), ..., (qn-1, an-1)]. We then take the question qn from the last turn as the new ground truth generation question. This helps increase the amount of data available for training the generation model.\n\n# Reasons To Reject Point 2 :\nThe previous SOTA method, DGM, used a core decision module consisting of graph networks and Transformers. In contrast, our approach only utilizes a few linear layers, resulting in a core decision module with parameters amounting to only 0.06% of those in DGM. We would like to emphasize that, compared to models using heavyweight decision modules such as DGM, Discern, and ET5, we achieve comparable results. Furthermore, compared to models employing lightweight core decision modules like EMT and E3, we observe a significant improvement of 12.7%.\n\n\n# Reasons To Reject Point 3 :\nThank you for pointing out the problem. We have calculated the time required for the pipeline of preprocessing, reasoning, and question generation on the Dev set (2270 samples) during the inference stage (using DeBERTa-large+T5-large). The entire inference process takes less than 5 minutes. Preprocessing the data takes about 2 minutes, making the decision classification on the Dev set takes 37 seconds, and generating questions for the data with reasoning results as MORE takes 70 seconds. Except for the preprocessing stage, which runs on the CPU, the GPU configuration and hyperparameters for inference and generation are consistent with what we described in section 4.3.\n\nAdditionally, we have calculated the time required for fine-tuning the reasoning model (BiAE) and the generation model. Fine-tuning BiAE (DeBERTa-large) for 5 epochs takes 6-7 hours, fine-tuning T5-large (without data augmentation) for 3 epochs takes 1-2 hours, and fine-tuning T5-large (with data augmentation) for 3 epochs takes 7-8 hours. The data augmentation is performed during the construction of the Dataset and the time required for it is negligible. The GPU configuration and hyperparameters for fine-tuning and reasoning/generation are consistent with what we described in section 4.3.\n\nWe will include this information in the revised version. Thank you again for your feedback.\n\n\n# Reasons To Reject Point 4 :\nThank you for highlighting the shortcomings. To address this, we have conducted significance test experiments. However, since the test set is reserved for official evaluation by ShARC (which requires submission of the model and code), we are unable to perform significance test experiments on the test set. Instead, we divided the training set into 5 folds to conduct 5-fold cross-validation t-tests, with result of p < 0.05. We will include this ablation experiment in the revised version. Thank you again for your feedback.\n\n\n# Reasons To Reject Point 5 :\nThank you for your attention to detail. According to the submission instructions of EMNLP, the content excluding the Limitations and Ethics Statement sections needs to be kept within 8 pages. We have ensured that our submission complies with this requirement. If it does not meet the specified page limit, our submission may face desk rejection.\n\nThank you for your meticulousness in identifying typos, grammar, style, and presentation improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An interesting method that explicitly aligns documents with the user-provided information.\n2. It shows SOTA results in terms of micro-accuracy on the ShARC dataset.\n3. The paper provides a very good and comprehensive overview of the related work.\n4. The authors show the ablation studies and provide a detailed analysis of their results.", "reasons_to_reject": "1. This paper includes some concepts that are not clearly defined/explained, e.g., the following ones (also see the questions to the authors):\n    - User-provided information and user scenarios need to be explained in more detail.\n    - The meaning of \"the weakly supervised alignment label\" (lines 248-149) is not clear.\n    - Data augmentation is not well explained. Although the authors claim that \"data augmentation improves all generation metrics\" (474-475) it's not clear how exactly they augment the data.\n2. The statements about the performance improvements seem a little bit over-emphasized. The authors claim that \"BiAE significantly outperforms baselines with lightweight decision modules by at least 12.7% in micro accuracy\" (090-092). However, on the ShARC leaderboard the difference between the proposed model and the previous SOTA is only 0.5% not 12.7%.\n3. The overall computational budget is only partially reported (e.g., no information about how many hours it takes to tune all the parts of the pipeline and the additional costs of the fine-tuning and data augmentation).\n4. The authors mention \"significant improvements\" but as far as I understand they did not perform any significance tests.\n5. The paper is slightly longer than 8 pages.", "questions_to_authors": " Question A: What exactly is meant by the user-provided information and user scenarios (168-169)? It seems that the sample dialogue in Figure 1 is not enough to introduce the concept.\n Question B: How do you define the difference between the heavyweight and lightweight decision modules? You mention that there is a \"significant reduction in decision module parameters\" (lines 108-109), but how big is the difference?\n Question C: Why was DeBERTaV3 chosen for the experiments and how does it compare to the models used in the baseline methods?\n Question D: How do you generate the direct responses in case of IRRELEVANT, YES, NO labels (lines 175-177)?\n Question E: What is the meaning of \"the weakly supervised alignment label\" (lines 248-149)? How is it computed?", "paper_topic_and_main_contributions": "The authors propose a pipeline framework that aligns the document with the user-provided information. This framework uses an entailment reasoning module and it can generate follow-up questions based on the document and the previously asked questions. The proposed approach shows the best performance in terms of micro-accuracy on the ShARC dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "We utilizes (lines 579-580)"}, "extra_text": {}}, {"review_id": "87fmDJLutM", "reviewer": "Reviewer_AKtE", "summary": "", "questions": "* Did you explore other ways to construct l^{align} instead of simple sentence similarity? Maybe an NLI model?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns and answer your question point by point.\n\n# Reasons To Reject Point 1\nOur pipeline framework consists of a reasoning model and a generation model. DA is only used for training the generation model. The explicit alignment and many-to-many entailment approaches we proposed are designed for the reasoning model. Due to the greater importance placed on the performance of the reasoning model, we have focused extensively on conducting experiments related to the reasoning model. The generation model is only activated when the reasoning result is \"MORE.\" The experiments related to generation account for only a small fraction of all our experiments and are not our primary focus.\n\n# Reasons To Reject Point 2\nThe motivation behind our work can be summarized into two points. Firstly, existing methods for reasoning have not taken into account the explicit alignment between document rules and user information, leading to the introduction of noise and sparse attention. Secondly, the core decision modules of existing methods (e.g., DGM using graph networks + transformers) have become increasingly complex, consuming more computational resources and memory. Therefore, in addition to the explicit alignment modeling you mentioned, our main contributions include designing a many-to-many entailment analysis module, significantly reducing the parameter size of the core decision module (only 0.06% of DGM). Experimental results show that our lightweight core decision module achieves comparable results to DGM.\n\nFurthermore, we also conducted ablation experiments on the encoder of the reasoning model, as we recognize the importance of fair comparison. The results of the ablation experiments based on ELECTRA on the development set are as follows. The reasoning results based on ELECTRA are slightly lower than those based on DeBERTa but still higher than DGM. This allows for a fair comparison with the previous SOTA method, DGM, and also demonstrates that the improvement in reasoning performance mainly comes from our incorporation of explicit alignment and the modeling of many-to-many entailment in the core decision module. We will include these experimental results in the revised version. Thank you for your feedback.\n| Model               | Micro Acc. | Macro Acc. |\n|---------------------|------------|------------|\n| DGM (electra-base)  | 75.5       | 79.6       |\n| BiAE (electra-base) | 76.2       | 80.3       |\n| BiAE (DeBERTa-base) | 76.2       | 80.5       |\n| DGM (electra-large) | 78.6       | 82.2       |\n| BiAE (electra-large)| 79.9       | 82.8       |\n| BiAE (deBERTa-large)| 80.5       | 83.2       |\n\n\n# Reasons To Reject Point 3\nThank you for pointing out the shortcomings. To address this, we have conducted significance test experiments. However, as the test set is reserved (evaluation on the test set requires submitting the model and code to the official ShARC platform), we are unable to perform significance test experiments on the test set. Instead, we divide the training set into 5 folds and conduct a t-test using 5-fold cross-validation, resulting in p < 0.05. We will incorporate these ablation experiments into the revised version.\n\n# Question\nThank you for your question. As using sentence embeddings to calculate cosine distance is simple and effective, we did not consider other approaches. Exploring better alignment methods will be left for our future work. However, we have conducted quality checks on the alignment labels in two ways.\n\nFirstly, we use a heuristic approach to label the alignments. Specifically, we calculate the edit distance between all document rules and user information and select the document hypothesis with the minimum edit distance as the label. The labels generated by the heuristic method show 85% consistency with the labels generated using SentenceBERT.\n\nSecondly, we randomly select 100 samples where labels are generated using SentenceBERT and manually evaluate their consistency. Among them, 92% of the generated labels are consistent with the manually selected ones.\n\nFurthermore, we conduct experiments using ELECTRA-base to explore the results of labeling with different versions of Sentence-BERT. Specifically, we use all-mpnet-base-v2 (420 MB, Avg. Performance: 63.30, SOTA) and all-MiniLM-L6-v2 (80M, Avg. Performance: 58.80, 5 times faster than SOTA). The results are as follows:\n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| all-mpnet-base-v2 | 76.0       | 79.9       |\n| all-MiniLM-L6-v2 | 76.2      | 80.5      |\n\nIn addition to using the hard labels obtained by selecting the document hypothesis with the highest cosine distance as labeled by SentenceBERT, we also conduct explicit alignment experiments using normalized soft labels and the KL divergence loss function. The results are as follows:\n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| Hard Label + Cross Entropy | 76.2       | 80.5       |\n| Soft Label + KL Divergence           | 70.3      | 74.9      |\nIn future work, we will continue to explore better alignment methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written and is easy to read and follow\n* The method proposed is relatively light-weight and easy to implement. It shows greater performance as compared to more complex baselines.\n* The methodology is simple and well described\n* The ablations are thorough\n* The analysis towards the end is much appreciated and insightful.", "reasons_to_reject": "* It is unclear if the advantages are from the modeling or data augmentation. Since the central contribution of the paper is in the modeling aspect (explicit alignment), I expect the authors to compare with the baselines where all the baselines have data augmentation for a more fair comparison. I do however note that DA might be a central aspect of this work.\n* This work feels like a combination of existing works and not very novel. Since most components (except the explicit alignment) are taken from prior work, I expect the authors to properly and fairly compare other approaches that have the same rest of the components.\n* The main results in table 1 don't seem statistically significant. Can you provide statistical tests?", "questions_to_authors": "* Did you explore other ways to construct l^{align} instead of simple sentence similarity? Maybe an NLI model?", "paper_topic_and_main_contributions": "The paper proposes a an approach in the three stage framework for conversational machine reading. The first stage involves aligning the document context with the user information explicitly, the second stage involves predicting agent action (yes, no, irrelevant or more), and the third stage is generating questions if the agent decides to ask follow up questions. \n\nThe authors call the lightweight decision making approach - Bipartite Alignment and many-to-many Entailment.\n\nThe method proposed by the authors is lightweight as compared to the baselines and outperforms larger baselines on the ShARC task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I'm not much aware of this area.", "Typos_Grammar_Style_and_Presentation_Improvements": "line 118 - add and\nline 122 - more challenging than?"}, "extra_text": {}}, {"review_id": "sdCTSc98Wl", "reviewer": "Reviewer_QzcE", "summary": "", "questions": "Question A: Did you experiment with varying weights for each term within the joint loss function as mentioned in equation (16)?\n\nQuestion B: Have you considered an end-to-end strategy that merges the two steps, perhaps through joint training?\n\nQuestion C: Could you elucidate the rationale behind opting for the maximal cosine distance as the alignment label? (“Specifically, we calculate the cosine distance between sentence vector pairs and select the hypothesis with the maximal cosine distance as the alignment label for each user premise.”)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thought-provoking comments. We will explain your concerns and answer your questions point by point.\n\n# Reasons To Reject Point 1:\nThank you for your feedback. Due to limited computational resources, it is challenging for us to organize more experiments. Conducting experiments on additional datasets will be left for our future work. In order to provide more comprehensive evidence of the effectiveness of BiAE, we divide the training set into five parts, conducting a 5-fold cross-validation experiment. We performe statistical significance t-tests between the results of these five experiments and the results of DGM, with p < 0.05. This experiment partially supplements the demonstration of the effectiveness of our approach. Thank you again for your feedback.\n\n\n# Reasons To Reject Point 2:\nThank you for your feedback. In fact, we have validated the quality of alignment labels through two methods. Firstly, we use a heuristic approach to make labels. Specifically, we calculate the edit distance between all document rules and user information and assign the document rule with the minimum edit distance as the label. The labels generated by the heuristic method show 85% consistency with the labels constructed using SentenceBERT. Secondly, we randomly select 100 samples with labels constructed using SentenceBERT and manually evaluate their consistency. Among them, 92% of the constructed labels are consistent with the human-selected ones. The issue you mentioned may indeed lie within the 8% noise, but this approach has proven to be simple and effective. Thank you again for your feedback.\n\nFurthermore, we conduct experiments using ELECTRA-base to explore the results of labeling with different versions of Sentence-BERT. Specifically, we use all-mpnet-base-v2 (420 MB, Avg. Performance: 63.30, SOTA) and all-MiniLM-L6-v2 (80M, Avg. Performance: 58.80, 5 times faster than SOTA). The results are as follows:\n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| all-mpnet-base-v2 | 76.0       | 79.9       |\n| all-MiniLM-L6-v2 | 76.2      | 80.5      |\n\nIn addition to using the hard labels obtained by selecting the document hypothesis with the highest cosine distance as labeled by SentenceBERT, we also conduct explicit alignment experiments using normalized soft labels and the KL divergence loss function. The results are as follows:\n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| Hard Label + Cross Entropy | 76.2       | 80.5       |\n| Soft Label + KL Divergence           | 70.3      | 74.9      |\nIn future work, we will continue to explore better alignment methods.\n\n# Question A:\nThank you for your question. We conducted several sets of experiments in the preliminary stage to explore the relative importance of different loss components. Specifically, we set up three configurations: \n* (1) L = λ * L_dec + L_align + L_entail,\n* (2) L = L_dec + λ * L_align + L_entail,\n* (3) L = L_dec + L_align + λ * L_entail, \nwhere λ is a weight parameter. The experimental results showed that L_dec is the most important term. Therefore, we only reported the results of the first configuration (1) in our paper.\n\n# Question B:\nThank you for your question. We have explored end-to-end modeling by incorporating an reasoning module between the shared encoder and decoder. For the first three types (YES, NO, MORE), the target of the decoder is to generate a special token. We also tried adding adapters for the core decision module and the decoder. The experimental results of using a T5-large encoder, a T5-large decoder, and a BiAE core decision module are as follows.\n| Method                   | Micro Acc. | Macro Acc. | BLEU1 | BLEU4|\n|--------------------------|------------|------------|------------|------------|\n| End2end, Multi-task | 72.56       | 76.96       |40.41       |31.98       |\n| End2end, Multi-task, +Adapter|75.68       | 79.59      | 48.45      | 40.36     |\nThe results show that the reasoning task and generation task have mutually negative impacts, and neither can achieve the desired results.\n\n# Question C:\nThe cosine distance is commonly used to measure the semantic similarity between sentence pairs. In the vector space of sentence embeddings, vectors that are semantically similar have smaller angles and thus larger cosine distances. The objective of constructing explicit alignment labels is to consider semantic similarity, where we aim to match the most similar document rules with user information. Calculating cosine distances is a simple and effective method, which is why we chose SentenceBERT to obtain sentence embeddings. Additionally, we validated the quality of alignment labels through two approaches. Firstly, we used a heuristic method for labeling, where we calculated the edit distance between all document rules and user information and selected the document with the minimum edit distance as the label. The alignment labels generated by the heuristic method show 85% consistency with the labels constructed using SentenceBERT. Secondly, we randomly selected 100 samples with labels constructed using SentenceBERT and manually evaluated their consistency. Among them, 92% of the constructed labels are consistent with the human-selected ones.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is clear and well structured.\n\n2. It encompasses the latest state-of-the-art baseline models.\n\n3. The introduced two-step methodology offers a fresh perspective, potentially pioneering a new learning paradigm in CMR research.\n\n4. Demonstrable high performance (First place in the leaderboard) on the CMR dataset, ShARC, outperforming existing methods.\n\n\n\n\n", "reasons_to_reject": "The paper does not include experiments on other notable CMR benchmark datasets. \n\nThe weakly supervised alignment label, generated using Sentence-BERT, might introduce biases and errors, particularly in longer sentences with negation. It would be beneficial to conduct additional ablation studies, perhaps employing alternative models or introducing noise to this loss, to further investigate its performance.\"", "questions_to_authors": "Question A: Did you experiment with varying weights for each term within the joint loss function as mentioned in equation (16)?\n\nQuestion B: Have you considered an end-to-end strategy that merges the two steps, perhaps through joint training?\n\nQuestion C: Could you elucidate the rationale behind opting for the maximal cosine distance as the alignment label? (“Specifically, we calculate the cosine distance between sentence vector pairs and select the hypothesis with the maximal cosine distance as the alignment label for each user premise.”)", "paper_topic_and_main_contributions": "This paper presents a two-tiered approach to the Conversational Machine Reading (CMR) task. The first is a Classification (BiAE) model which aligns the document to user-provided data and employs an entailment reasoning module for decision-making. The second utilizes T5 model and BiAE's results for question generation.\n\nContributions:\n\n1. Introduce an explicit alignment strategy for improved interaction between document content and user data with weakly supervised alignment label.\n\n2. Implement a lightweight many-to-many entailment reasoning module that enhances decision-making based on user queries.\n\n3, A novel approach to generating follow-up questions that consider both the content of the document and previous user questions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uw5pYFU9Do", "reviewer": "Reviewer_HUMs", "summary": "", "questions": "Question A: Is there any effort dedicated to checking for the quality of weakly supervised alignment labels? \n\nQuestion B: The representation of segments in document D (d_1, d_2, … d_n) is often utilized independently, such as in alignment and decision processes. Is it reasonable to utilize these representational vectors independently? In the Segmentation part of Section 3 (Methodology), the Segbot does not decontextualize Elementary Discourse Units of a given document D (Refer to Decontextualization: Making Sentences Stand-Alone (Choi et al., TACL 2021) for the concept of decontextualization). Is that necessary to decontextualize these EDUs?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns and answer your questions point by point.\n\n# Reasons To Reject Point 1:\nThank you for your feedback. Due to limited computational resources, it is challenging for us to organize more experiments. Conducting experiments on additional datasets will be left for our future work. In order to provide more comprehensive evidence of the effectiveness of BiAE, we divide the training set into five parts, conducting a 5-fold cross-validation experiment. We perform a statistical significance t-test between the results of these five experiments and the results of DGM, resulting in p < 0.05. This experiment partially supplements the demonstration of the effectiveness of our approach. Thank you again for your feedback.\n\n# Reasons To Reject Point 2:\nThank you very much for your insightful suggestion. We acknowledge that fair comparison is crucially important. Therefore, for the reasoning model, we conducted ablation experiments based on ELECTRA encoders on the development set (results shown below). The experimental results show that the reasoning results based on ELECTRA are slightly lower than those based on DeBERTa, but still higher than DGM. On one hand, this can fairly compare with the previous SOTA, DGM; on the other hand, this can illustrate that the improvement in reasoning performance mainly comes from our introduction of explicit alignment and the modeling of many-to-many entailment into the core decision module. It should be noted that the parameter size of our core decision module (31.7K) is only 0.06% of that of DGM (54M). One of our main contributions is achieving comparable results to DGM while greatly reducing the parameter size of the core decision module. We will add this ablation experiment to the revised version. Thank you again for your feedback.\n| Model               | Micro Acc. | Macro Acc. |\n|---------------------|------------|------------|\n| DGM (electra-base)  | 75.5       | 79.6       |\n| BiAE (electra-base) | 76.2       | 80.3       |\n| BiAE (DeBERTa-base) | 76.2       | 80.5       |\n| DGM (electra-large) | 78.6       | 82.2       |\n| BiAE (electra-large)| 79.9       | 82.8       |\n| BiAE (deBERTa-large)| 80.5       | 83.2       |\n\n\n# Question A:\nThank you for your feedback. We have validated the quality of alignment labels through two methods. Firstly, we use a heuristic approach to make labels. Specifically, we calculate the edit distance between all document rules and user information and assign the document rule with the minimum edit distance as the label. The labels generated by the heuristic method show 85% consistency with the labels constructed using SentenceBERT. Secondly, we randomly select 100 samples with labels constructed using SentenceBERT and manually evaluate their consistency. Among them, 92% of the constructed labels are consistent with the human-selected ones. \n\nFurthermore, we conduct experiments using ELECTRA-base to explore the results of labeling with different versions of SentenceBERT. Specifically, we use all-mpnet-base-v2 (420 MB, Avg. Performance: 63.30, SOTA) and all-MiniLM-L6-v2 (80M, Avg. Performance: 58.80, 5 times faster than SOTA). The results are as follows:\n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| all-mpnet-base-v2 | 76.0       | 79.9       |\n| all-MiniLM-L6-v2 | 76.2      | 80.5      |\n\nIn addition to using the hard labels obtained by selecting the document hypothesis with the highest cosine distance as labeled by SentenceBERT, we also conduct explicit alignment experiments using normalized soft labels and the KL divergence loss function. The results are as follows:\n| Method                   | Micro Acc. | Macro Acc. |\n|--------------------------|------------|------------|\n| Hard Label + Cross Entropy | 76.2       | 80.5       |\n| Soft Label + KL Divergence           | 70.3      | 74.9      |\nIn future work, we will continue to explore better alignment methods.\n\n# Question B.1:\nAlthough the embeddings of document rules are independently used for explicit alignment and many-to-many entailment, the rules can attend to each other as context during encoding. In addition, the attention mechanism is also used to synthesize the information for the final decision classification. Moreover, previous baselines (such as DGM and DISCERN) can be seen as dependently used, but our results are comparable to their best results, and even our main metric is slightly better than theirs.\n\n# Question B.2:\nThank you for your question. Based on our understanding of decontextualization in the original paper, decontextualization refers to the process of adding crucial contextual information (such as referential and bridging information) to stand-alone sentences, enabling them to maintain their meaning even when detached from context. Its purpose is \"to be able to interpret, or present, text independently from the rich context in which it occurs.\"\n\nIn the current CMR task, during the encoding phase, we do not reduce the context for each document fragment (EDU). Each EDU can still see all the surrounding context. The only difference is that after encoding, we use embeddings separately for explicit alignment and many-to-many entailment. In other words, the independent encoding of EDUs that we use still contains contextual information.\n\nHowever, your question has also prompted us to take decontextualization into consideration if we conduct future experiments on longer documents. If there are limitations on input length and thus we have to split the document, making it impossible to input the complete context, decontextualization may indeed be necessary in such scenarios.\n\n\n# Typos Grammar Style And Presentation Improvements:\nThank you for your feedback. In the revised version, we will take multiple perspectives into consideration and remove unnecessary description. We will make modifications to simplify the formulations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tBiAE is a novel approach to the problem of CMR. The ablation study conducted on BiAE offers valuable insights into CMR.\n\n2.\tThis paper introduces a novel approach for constructing a lightweight decision module, which has significant merit for future direction of CMR.", "reasons_to_reject": "1.\tThe experiments with BiAE are trivial. Experiments using only one dataset is not enough to prove the effectiveness of BiAE over baselines in this paper.\n\n2.\tIt is still unclear if the superior performance by BiAE is thanks to the novel architecture proposed or the superior encoding module (DeBERTaV3). This is because previous state-of-the-art models use previous core modules, such as DGM uses ELECTRA,  ETM uses BERT base, … Further study about the core modules is needed as we know that DeBERTaV3 significantly outperform previous models on the task of RTE.\n\nNote: If the author(s) can address any of these concerns during the rebuttal phase, the final revision may be a strong candidate for the conference.", "questions_to_authors": "Question A: Is there any effort dedicated to checking for the quality of weakly supervised alignment labels? \n\nQuestion B: The representation of segments in document D (d_1, d_2, … d_n) is often utilized independently, such as in alignment and decision processes. Is it reasonable to utilize these representational vectors independently? In the Segmentation part of Section 3 (Methodology), the Segbot does not decontextualize Elementary Discourse Units of a given document D (Refer to Decontextualization: Making Sentences Stand-Alone (Choi et al., TACL 2021) for the concept of decontextualization). Is that necessary to decontextualize these EDUs?", "paper_topic_and_main_contributions": "The paper proposes Bipartite Alignment and many-to-many Entailment (BiAE) for the Conversational Machine Reading (CMR) task. BiAE uses DeBERTaV3 and T5. BiAE features a lightweight decision module, showcasing a substantial performance improvement over its other small models on the ShARC benchmark. Moreover, BiAE surpasses the performance of previous state-of-the-art heavyweight models across critical metrics. The authors also highlight the importance of explicit alignment of the document and the user-provided information in CMR.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In section 3, why do the author(s) have to define D = w_1 w_2…w_{n_D}, Q = w_1 w_2…w_{n_Q}, S = w_1 w_2…w_{n_S}? This paper's definition of w is unnecessary and could potentially be excluded, enhancing readers' comprehension of the fundamental concepts presented."}}, {"review_id": "ZSDZYCUuIq", "reviewer": "Reviewer_1skF", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns and answer your question point by point.\n\n# Reasons To Reject Point 1:\nThe motivation behind our work can be summarized into two points. Firstly, existing methods for modeling have not taken into account the explicit alignment between document rules and user information, leading to the introduction of noise and sparse attention. Secondly, the core decision modules of existing methods (e.g., DGM using graph networks + transformers) have become increasingly complex, consuming more computational resources and memory. Therefore, in addition to the explicit alignment modeling, our main contributions include designing a many-to-many entailment analysis module, significantly reducing the parameter size of the core decision module (only 0.06% of DGM). Experimental results show that our lightweight core decision module achieves comparable results to DGM.\n\nFurthermore, we also conducted ablation experiments on the encoder of the reasoning model, as we recognize the importance of fair comparison. The results of the ablation experiments based on ELECTRA on the development set are as follows. The reasoning results based on ELECTRA are slightly lower than those based on DeBERTa but still higher than DGM. This allows for a fair comparison with the previous SOTA method, DGM, and also demonstrates that the improvement in reasoning performance mainly comes from our incorporation of explicit alignment and the modeling of many-to-many entailment in the core decision module. We will include these experimental results in the revised version. Thank you for your feedback.\n| Model               | Micro Acc. | Macro Acc. |\n|---------------------|------------|------------|\n| DGM (electra-base)  | 75.5       | 79.6       |\n| BiAE (electra-base) | 76.2       | 80.3       |\n| BiAE (DeBERTa-base) | 76.2       | 80.5       |\n| DGM (electra-large) | 78.6       | 82.2       |\n| BiAE (electra-large)| 79.9       | 82.8       |\n| BiAE (deBERTa-large)| 80.5       | 83.2       |\n\n\n# Reasons To Reject Point 2:\nIn terms of model performance, our main contributions are twofold. Firstly, we significantly reduced the parameter size of the core decision module by introducing explicit alignment and many-to-many entailment, with our core decision module accounting for only 0.06% of DGM's. Secondly, we achieved comparable results to strong baselines while also surpassing the previous SOTA (DGM) on the main metric.\n\nIn addition, we conducted significance test experiments. As the test set is reserved (evaluation on the test set requires submitting the model and code to the official ShARC platform), it is hard for us to perform significance tests on the test set. However, we divided the training set into five parts and conducted a 5-fold cross-validation t-test, resulting in p < 0.05. The result indicates that our performance is significantly higher than that of DGM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a lightweight many-to-many entailment reasoning module for decision-making. \n2. The proposed approach achieves the state-of-the-art results on ShARC dataset.", "reasons_to_reject": "1. The novelty is limited. As mentioned in this work, most part of this paper follows the previous approach EMT, DISCERN, ET5, and DGM. The biggest contribution is the proposed lightweight many-to-many entailment reasoning module.\n2. The improvement is marginal compared to previous work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a pipeline framework for Conversational Machine Reading (CMR) that explicitly aligns the user's initial question with the provided document and utilizes a lightweight many-to-many entailment reasoning module for decision-making. The framework also enables direct follow-up question generation based on the document and previously asked questions. By addressing the alignment issue and incorporating effective reasoning and question generation techniques, the proposed method achieves state-of-the-art results on ShARC dataset.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the reviews, the paper proposes a pipeline framework for Conversational Machine Reading (CMR) that addresses the issue of alignment between the document and user-provided information. The proposed method achieves state-of-the-art results on the CMR benchmark dataset ShARC. 

The reviewers generally appreciate the proposed framework, highlighting its novelty, practicality, and improved performance over baselines. They also commend the clear writing style and thoroughness of the experiments and analysis.

However, there are some concerns raised by the reviewers. One reviewer questions the potentially unfair comparison between the proposed model and baselines due to the differences in backbone models. Another reviewer raises questions about the clarity of certain concepts and urges the authors to provide more details and clarity. Furthermore, there are suggestions to include more datasets for evaluation and to consider alternative models or noise in the weakly supervised alignment label. 

In terms of novelty, some reviewers express ambivalence, noting that while the proposed many-to-many entailment reasoning module is novel, other components of the pipeline are incremental. The pros mentioned by the reviewers include the clear writing, improved performance, and insights provided by the ablation studies.

In summary, the paper presents a promising pipeline framework for CMR, addressing the alignment issue and achieving state-of-the-art results on the ShARC dataset. The reviewers appreciate the clear writing and thorough experiments, but some concerns and suggestions for improvement have been raised.
Decision: Accept-Findings