Bridging The Gap: Entailment Fused-T5 for Open-retrieval Conversational Machine Reading Comprehension
Year: 2023

Abstract:
Open-retrieval conversational machine reading comprehension (OCMRC) simulates reallife conversational interaction scenes. Machines are required to make a decision of Yes/No/Inquire or generate a follow-up question when the decision is Inquire based on retrieved rule texts, user scenario, user question and dialogue history. Recent studies try to reduce the information gap between decision-making and question generation, in order to improve the performance of generation. However, the information gap still persists because these methods are still limited in pipeline framework, where decision-making and question generation are performed separately, making it hard to share the entailment reasoning used in decision-making across all stages. To tackle the above problem, we propose a novel one-stage end-to-end framework, called Entailment Fused-T5 (EFT), to bridge the information gap between decisionmaking and question generation in a global understanding manner. The extensive experimental results demonstrate that our proposed framework achieves new state-of-the-art performance on the OR-ShARC benchmark. Our model and code are publicly available1.

heading: 1 Introduction
text: Open-retrieval conversational machine reading comprehension (OCMRC) (Gao et al., 2021) investigates real-life scenes, aiming to formulate multi-turn interactions between humans and machines in open-retrieval settings. As shown in Figure 1, given the user scenario and user question, machines are required to first retrieve related rule texts in the knowledge database, and then make a decision of Yes/No/Inquire or generate a follow-up question when the decision is
∗Corresponding author. 1github.com/Yottaxx/EFT
Inquire based on retrieved rule texts, user scenario, user question and dialogue history.
Previous studies (Saeidi et al., 2018; Verma et al., 2020; Lawrence et al., 2019; Zhong and Zettlemoyer, 2019; Gao et al., 2020a,b; Ouyang et al., 2021; Gao et al., 2021; Zhang et al., 2021) typically adopt pipeline frameworks based on pretrained language models (PrLM) (Devlin et al., 2019; Clark et al., 2020; Lewis et al., 2020; Liu et al., 2020) , as shown in Figure 4, these frameworks usually consist of three stages, includ-
15374
ing decision-making, span extraction and question rephrasing. Different entailment reasoning strategies are utilized to improve the performance of decision-making. Span extraction and question rephrasing are conducted for question generation. These pipeline frameworks are either completely independent of the three stages (Zhong and Zettlemoyer, 2019; Gao et al., 2020a,b; Ouyang et al., 2021; Gao et al., 2021), or try to reduce the information gap between decision-making and question generation through representation-fused methods (Zhang et al., 2021) among three stages .
However, the information gap still persists because these methods are still limited in pipeline framework, where decision-making and question generation are performed separately, making it hard to share the entailment reasoning used in decision-making across all stages.
To tackle the above problem, we propose a novel one-stage end-to-end framework, called entailment fused-T5 (EFT) to bridge the information gap between decision-making and question generation in a global understanding manner. Specifically, our model consists of a universal encoder and a duplex decoder. The decoder consists of an activated entailment reasoning decoder and a fused answer generation decoder. The implicit reasoning chains of both decision-making and question generation in the multi-fused answer generation decoder are explicitly supervised by ac-
tivated entailment reasoning through the shared entailment representation of our encoder. Moreover, a relevance-diversity fusion strategy is utilized to further improve the implicit reasoning abilities among the multiple retrieved rules for the fused answer generation decoder through the implicit ranking method. Thus, our model can reason in a global understanding manner. The extensive results, as illustrated in Figure 3, demonstrate that our proposed framework EFT achieves new stateof-the-art performance on the OR-ShARC benchmark.
Our contributions are summarized as follows:
• We propose a novel one-stage end-to-
end framework, called entailment fused-T5 (EFT) to bridge the information gap between decision-making and question generation through a global understanding manner.
• We further investigate a relevance-diversity fusion strategy (RD strategy) to improve the implicit reasoning abilities of our model.
• Extensive experiments demonstrate the effectiveness of our proposed framework on the OR-ShARC benchmark.

heading: 2 Related Work
text: Conversation-based Reading Comprehension Conversation-based reading comprehension (Saeidi et al., 2018; Sun et al., 2019; Reddy et al., 2019; Choi et al., 2018; Cui et al., 2020; Gao et al., 2021) aims to formulate human-like interactions. Compared to traditional reading comprehension, these tasks extend the reading comprehension scenarios with dialogue interactions. There are typically three main types of these tasks: span-based QA tasks (Choi et al., 2018; Reddy et al., 2019), multi-choice tasks (Sun et al., 2019; Cui et al., 2020), or hybrid-form tasks (Saeidi et al., 2018; Gao et al., 2021).
Conversational Machine Reading Comprehension CMRC (Saeidi et al., 2018) is the hybrid form of conversation-based reading comprehension, which requires the machines to make a decision or generate a follow-up question based on rule text, user scenario, user question and dialogue history.
In this paper, we focus on the open-retrieval conversational machine reading (OCMRC) task (Gao et al., 2021), which further extends the CMRC task into a real-life scenario. Machines are required to first retrieve related rule texts in a knowledge base based on user questions and user scenarios, then machines are required to make a decision of Yes/No/Inquire, or a follow-up question if the decision is Inquire based on the relevant rule texts, user scenario, user question and dialogue history.
Due to the hybrid-form task, the previous methods (Zhong and Zettlemoyer, 2019; Gao et al., 2020a,b; Ouyang et al., 2021; Zhang et al., 2021) typically adopt pipeline architectures, including decision-making, span extraction and question phrasing. Various kinds of entailment reason-
ing strategies are proposed to improve the performance of decision-making. Despite the effectiveness of entailment reasoning, the performance is still limited because of the information gap between decision-making and question generation. Recent studies (Zhang et al., 2021, 2022) explored entailment reasoning sharing methods to reduce the gap between decision-making and question generation, but the performance is limited due to its frame flaws. In this paper, we propose a novel one-stage end-to-end model, called entailment fused-T5 (EFT), the details are written in the next sections.

heading: 3 Methods
text: In open-retrieval CMRC, the machines are first required to retrieve related rule texts in a knowledge base, given user question and user scenario. Then machines are required to make decisions or generate follow-up questions based on retrieved rule texts, user scenario, user question and dialogue history. Thus, we conduct a retriever to first retrieve related rule texts from the knowledge database, and then generate the final answer through our end-to-end reader EFT. The training procedure is shown in Algorithm 1.

heading: 3.1 Retriever
text: We first concatenate the user question and user scenario as the query to retrieve related rule texts in the knowledge base. The knowledge base is divided into the seen subset and the unseen subset. This is to simulate the real usage scenario: users will ask questions about rules they have already seen, or rules that are completely new. We only use seen rules in the training process. In this work, we utilize DPR (Karpukhin et al., 2020) to retrieve related rule texts. In contrast to previous approaches (Zhang et al., 2021) that employ TFIDF negatives as DRP hard negatives and restrict the scope of retrieved negatives to a limited data space, we adopt a different strategy. We randomly sample rule texts from the known knowledge base to serve as the negatives. Each step will randomly sample m numbers negatives in the training stage. We retrieve the top 20 relevant rule texts for each query, which is further used by our reader.

heading: 3.2 Reader: EFT
text: In this stage, each item is formed as the tuple {R,S,Q,D}. R donates the rule text can-
didates. R = {r1, r2, ..., rk}, where r donates the rule text item of R. S and Q represent user scenario and user questions, respectively. D donates the dialogue history. Given {R,S,Q,D}, EFT will directly generate a decision of Yes/No/Inquire or follow-up question when the decision is Inquire. EFT consists of a universal encoder and a duplex decoder. The Duplex decoder consists of an activated entailment reasoning decoder and a fused answer generation decoder. In this way, the whole implicit reasoning chains of the fused answer generation decoder will be fine-grained supervised by activated entailment reasoning with the shared entailment representation. Thus, the fused answer generation decoder will reason in a global understanding manner. The details are shown in Figure 4.

heading: 3.3 Encoding
text: Given {R,S,Q,D}, we random sample k items in R, and concatenate each of them with S,Q,D as c, thus the item collection is formed as C = {c1, c2, ..., ck}. Specifically, each r in R is first parsed to elementary discourse unit (EDU) by a pre-trained model (Li et al., 2018). The final input format is shown in Figure 4. To prevent the leakage of location information in the fused an-
swer generation decoder, and enhance the information extraction ability of the decoder, we utilize a relevance-diversity fusion (RD) strategy to randomly shuffle the order of items which are sampled from RD candidate rule texts, the details are written in Sec 3.4. Given C, we utilize T5 encoder as our backbone to get the representation. The presentation of special token are utilized as sentence-level representation Hs = {hs1, hs2, ..., hsk} for activated entailment reasoning decoding. The word-level representation Hw = {hw1, hw2, ..., hwk} are utilized for fused answer generation decoding.

heading: 3.4 Decoding
text: We utilize duplex decoding to explicitly supervise our answer generation stage, which introduced the explicit entailment reasoning information in implicit answer generation reasoning. The answer generation will either generate a decision of Yes/No/Inquire or a follow-up question when the decision is Inquire. The activated entailment reasoning decoder will reason the entailment states of the EDUs. The duplex decoder is trained in a multi-task form. And the activated entailment reasoning only activates in training stage.
Activated Entailment Reasoning Each EDU will be classified into one of three entailment states, including ENTAILMENT, CONTRADICTION and NEUTRAL. To get the noisy supervision signals of entailment states, we adopt a heuristic approach2. This is proposed to simulate fulfillment prediction of conditions in multi-sentence entailment reasoning, which can explicitly supervise the implicit reasoning chains of the answer generation.
Previous studies typically introduce entailment reasoning in all rule text segmented EDUs. This will greatly increase the proportion of NEUTRAL labels and affect the model effect, because nearly all of the entailment states of EDUs in retrieved irrelevant rules are NEUTRAL, and introducing more noise in the training stage. In our method, entailment reasoning will only activate for the golden rule text. Utilizing this setting, one benefit is to balance the categories of entailment reasoning, and the other is to supervise the implicit reasoning of the fused decoder, which can help the fused decoder infer correct rule text from multiple retrieved rule texts.
Given the sentence-level representation Hs, we utilize inter-attention reasoning to fully interact with various information in r, including EDUs, user question, user scenario and dialogue history. We utilize an inter-sentence Transformer (Devlin et al., 2019; Vaswani et al., 2017) to get the interacted sentence-level representation Gs. Then, we use a linear transformation to track the entailment reasoning states of each EDU in activated rule text.
ei = Wch̃ei + bc ∈ R3, (1) where the Wc is trainable parameters, ei is the predicted score for the three labels of the i-th states.
Fused Answer Generation Given the wordlevel representation Hw = {hw1, hw2, ..., hwk} of R, we concatenate the Hw as the fused representation fw. In this manner, our answer generation decoder can reason through all the k items through an implicit ranking mechanism. It is worth mentioning that each item of Hw is first fully interacted among rule text, user question, user scenarios and dialogue history without other multi-rule noise through our encoder.
To improve the information implicit reasoning abilities of our model, we further investigate
2The noisy supervision signal is a heuristic label obtained by the minimum edit distance.
Algorithm 1 Training procedure of EFT Input: Contextualized context C, learning rate τ , Output: Final answer A, activated entailment
reasoning state E, EFT encoder parameters θe, EFT fused answer generation decoder parameters θa, EFT activated entailment reasoning decoder parameters θd
1: Initialize θe,θa,θd 2: while not converged do 3: for i = 1, 2, . . . , N do 4: hsi,hwi = f(ci,θe) s.t. ∀c ∈ C 5: ei = f(hsi,θd) 6: ai = f(hwi,θa)
7: end for 8: g ← ∇θL 9: θe ← θe − τg 10: θd ← θd − τg 11: θa ← θa − τg 12: end while
the relevance-diversity fusion strategy (RD fusion strategy), which consists of relevance-diversity candidate rule texts, order information protection and fused answer generation. The rule text candidates are consists of top k relevant rule texts and randomly sampled rule texts, which are called RD candidate rule texts. Thus, the candidates are full-filled with relevant and diverse rule texts. On the premise of ensuring relevance among the rule texts, the diversity of learnable information sampling combinations is further improved. Moreover, the order of items fused in fw may lead to information leakage and affect the reasoning ability of the decoder in the training stage, so as we mentioned in the last section, we will randomly shuffle the order of items when inputting to the encoder to protect the order information. In the evaluation stage, only the top 5 unshuffled relevant rule texts will be utilized for answer generation.
The fused answer generation is utilized to generate either the decision or the follow-up question. We employ T5 decoder as our answer generation decoder. Given encoder fused representation fw, and the final answer a, including decision or follow-up question, the answer is composed of the variable-length tokens xi, the probabilities over the tokens are shown in the blow:
p(a) =
m∏
1
p(xi|x<i, fe; θ), (2)
where θ donates the trainable parameters of our decoder.

heading: 3.5 Training Objective
text: Activated Entailment Reasoning The activated entailment reasoning is supervised by crossentropy loss, by given the entailment stages ci :
Lenatil = − 1
N
N∑
i=1
log softmax(ci)r, (3)
where r is the ground truth of entailment state.
Fused Answer Generation The fused answer generation training objective is computed as illustrated in below:
Lanswer = − M∑
i=1
log p(xi|x<i, fw; θ), (4)
The overall loss function is:
L = Lanswer + λLentail. (5)

heading: 4 Experiment and Analysis

heading: 4.1 Data
text: Dataset The experiment dataset is OR-ShARC (Gao et al., 2021), the current OCMRC benchmark. The corpus is crawled from the government website. There is a total of 651 rule texts collected in the knowledge base. For the validation and test set, the golden rule texts are split into unseen or seen. This is to simulate the real usage scenario: users will ask questions about rules they have already seen, or rules that are completely new. The train, dev and test size is 17,936, 1,105 and 2,373, respectively. Each item consists of utterance id, tree id, golden rule document id, user question, user scenario, dialog history, evidence and the decision.

heading: 4.2 Setup
text: Evaluation The evaluation consists of two parts: decision-making and question generation. We utilize Micro-Acc and Macro-Acc for the results of decision-making, and use F1BLEU (Gao et al., 2021) for question generation. The F1BLEU is conducted to evaluate the question generation performance when the predicted decision is Inquire.
precisionBLEU =
∑M i=0 BLEU(yi, ŷi)
M , (6)
Where M is the total number of Inquire decisions made by our model. yi is the predicted question, ŷi is the corresponding ground truth prediction. The recall of BLEU is computed in a similar way.
recallBLEU =
∑N i=0 BLEU(yi, ŷi)
N , (7)
where N is the total number of Inquire decision from the ground truth annotation,
The calculation of F1BLEU is shown below:
F1BLEU = 2× precisionBLEU × recallBLEU precisionBLEU + recallBLEU .
(8)
Implementation Details We utilize the T5-base (Raffel et al., 2020) as our reader backbone, and additionally add an activated entailment reasoning decoder, whose parameters are randomly initialized. We utilize BERT (Devlin et al., 2019) as our retriever backbone , whose parameters are initialized from DPR (Karpukhin et al., 2020). For the RD strategy, we use the top-20 retrieved rule texts and 30 randomly sampled rule texts as our fused candidates in the training stage, every step we will randomly select 5 samples from the candidates. We only use seen rule texts in the knowledge base for the training stage. And we only use top 5 retrieved rule text for the inference stage. The fused number k is set as 5 for fused answer generation for both training and inference. We use AdamW (Loshchilov and Hutter, 2018) to finetune our model. The learning rate is hierarchically designed, the learning rate of T5 is 2e-4, and the learning rate of activated entailment decoder is 2e5. We tried from 0.1 to 1.0 for λ, and find 0.9 is the best hyper-parameter. The beam-size is set as 5 for the answer generation.

heading: 4.3 Results
text: All results on the OR-ShARC benchmark are illustrated in Table 1, including dev and test set with metrics for both decision-making and question generation.
Experimental results demonstrate that our proposed methods achieve new SOTA on the ORShARC benchmark. EFT outperforms OSCAR by 3.6% in Micro-Acc, 3.6% in Macro-Acc for decision-making on the dev set, and outperforms OSCAR by 2.6% in Micro-Acc, 2.7% in MacroAcc for decision-making on the test set. In particular, our proposed EFT achieves considerable
improvement in BLEU scores. EFT outperforms OSCAR by 27.7% in F1BLEU1, 36.9% in F1BLEU4 for the question generation on the dev set, and outperforms OSCAR by 20.8% in F1BLEU1, 26.5% in F1BLEU4 for the question generation on the test set. We further to investigate the classwise accuracy performance of EFT, as shown in Table 4. Experiments show that the accuracy of each category in OR-ShARC is improved by conducting EFT framework, compared with reported baselines.
To further investigate the performance for our proposed EFT in seen and unseen settings, the performance of the split subset 3 is illustrated in Table 3. Compared with OSCAR, the seen subset performance are greatly improved through our framework EFT. EFT greatly outperforms OSCAR by 29.1% in F1BLEU1, 36.4% in F1BLEU4 for the question generation on the seen test set. In addition, compared with the previous pipeline architectures utilized in MURDEN and OSCAR, our model not only improves the performance, but also makes the framework of OCMRC more lightweight. We reduce the number of model parameters from 330M to 220M, which is decreased by 33.3%. The performance on the seen subset of
3Only BLEU scores are reported in OSCAR.
EFT is 35.0% higher in micro-acc than seen subset, 35.3% higher in macro-acc than unseen subset. Our retrieval results are illustrated in Table 6 and Table 5. The details are illustrated in Appendix A and Appendix B, respectively.

heading: 4.4 Ablation Studies
text: The ablation studies of EFT on the dev set of ORShARC benchmark are shown in Table 2. There are four settings of our EFT is considered:
• EFT-wo/s trains the model without relevance-diversity (RD) candidate rule texts. Only top-5 randomly shuffled relevant rule texts are considered in the training stage.
• EFT-wo/s+a trains this model additionally remove activated entailment reasoning.
• EFT-wo/s+a+i trains this model further cancels random shuffle in the training stage.
• EFT-wo/s+a+i+f trains this model without multi rule fused answer generation, only top1 retrieved rule text is considered.
Analysis of RD Candidate We investigate the necessity of the RD candidate rule texts. This strategy is utilized to improve the implicit reasoning abilities of our decoder by improving the learning space of fused candidates. On the premise of ensuring the relevance among the rule texts, the diversity of learnable information sampling combinations is further improved. As shown in Table 2, compared with EFT, the performance of both decision-making and question generation decline when RD candidate rule texts are removed, highlighting the effectiveness of RD candidate rule
texts in enhancing the information-seeking abilities of our fused answer generation decoder. By removing RD candidate rule texts, the micro-acc is decreased by 0.5, the macro-acc is decrease by 0.4, the F1BLEU1 is decreased by 1.7, and the F1BLEU4 is decreased by 2.0. The above results emphasize the indispensability of RD candidate rule texts.
Analysis of Activated Entailment Reasoning EFT-wo/s+a trains this model additionally remove activated entailment reasoning. As illustrated in Table 2, compared with EFT-wo/s+a, the performance of both decision-making and question generation are dropped without activated entailment reasoning, the micro-acc is decreased by 2.2, the macro-acc is decrease by 2.3, the F1BLEU1 is decreased by 1.4, and the F1BLEU4 is decreased by 0.7. The above results suggest that the implicit reasoning of conversational machine reading comprehension could be enhanced by introducing explicit fine-grained supervised signal in a global understanding manner.
Analysis of Order Information Protection The order of fused representation used in fused answer generation decoder may lead to information leakage and affect the reasoning ability of the decoder. In order to avoid the problem of poor information seeking ability caused by excessive learning of position information of the model, we randomly shuffle the order of fused representation to protect the order information in the training stage. As illustrated in Table 2, compared with EFT-wo/s+a, EFT-wo/s+a+i decrease the performance of both decision-making and question generation without order information protection, the micro-acc is decreased by 0.5 , the macro-acc is decrease by 0.6, the F1BLEU1 is decreased by 1.2, and the F1BLEU4 is decreased by 1.3. The above results indicates the importance of order information protection.
Analysis of Fused Generation Fused Generation is utilized to introduce the ability to pro-
cess multiple rule contextualized information. The multiple rule contextualized information are fused as a single fused information. EFT-wo/s+a+i+f trains this model without multi rule fused answer generation, only top1 retrieved rule text is considered. In this manner, the performance is limited with the retrieval performance. Compared with EFT-wo/s+a+i+f, the performance of both decision-making and question generation of EFTwo/s+a+i are significantly improved by introducing fused answer generation strategy, the microacc is increased by 9.2 , the macro-acc is increased by 8.9, the F1BLEU1 is increased by 12.0, and the F1BLEU4 is increased by 11.2. The above results suggests the necessity of fused answer generation strategy.

heading: 5 Conclusion
text: In this paper, we propose a novel end-to-end framework, called EFT, to bridge the information gap between decision-making and question generation through the shared entailment representation in a global understanding manner. Extensive experimental results on the OR-ShARC benchmark demonstrate the effectiveness of our proposed framework EFT. In our analysis, the implicit reasoning ability of both decision-making and question generation is significantly improved by sharing external explicit entailment knowledge through our novel framework EFT.
Limitations
As shown in Table 3, the results demonstrate the effectiveness of our proposed EFT, but the performance of the unseen subset is still limited by comparing it with the performance of seen subset, which suggests plenty of room for improvement. Data augmentation or generalization methods based on semi-supervised methods could be effective to solve the problem in the future.

heading: Acknowledgements
text: The work is supported by National Key R&D Plan (No. 2020AAA0106600), National Natural Science Foundation of China (No.62172039, U21B2009 and 62276110).

heading: A The Performance of Retriever (DPR-R)
text: Table 6 presents the detailed performance of DPR-R, including the performance of Top-k on dev set and test set. Different from previous methods (Zhang et al., 2021) that utilize DRP but only use TF-IDF retrieved negatives, we use random negatives sampled from seen rule texts in knowledge base. Experimental results illustrate that DPR-R outperforms DPR by 16.5% in top5 accuracy on test set, and reaches competitive results with TF-IDF+DPR.

heading: B The Performance of Retriever (DPR-R) on Subset
text: We further analyzed the performance of DPR-R on the seen and unseen subset. As shown in Table 5, experimental results demonstrate the effectiveness of DPR-R on seen sets, the top1 accuracy reached 97.6 on the seen subset of test set. But the performance still have a large latent space of improvement on unseen sets.

heading: C Additional Experiment Details
text: We implement EFT with the PyTorch4 library and using pre-trained Transformers from the Hugging Face5 repositories. The retriever DPR-R is based on the DPR6 repositories. The data of OR-ShARC are from the OR-ShARC7 repository. The above repositories provide the data, models and licenses. The whole training process takes about several hours on eight Nvidia A100 GPUs.
4github.com/pytorch/pytorch 5github.com/huggingface/transformers 6github.com/facebookresearch/DPR 7github.com/YifanGao/open_retrieval_
conversational_machine_reading
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section after section 5

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations section after section 5

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: We use chatGPT to verify the grammar of our abstract.
B 3 Did you use or create scientific artifacts? Appendix.C

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix.C

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix.C
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix.C and Section 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 1, Table2 in Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix.C and Section 4.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Data augmentation or generalization methods based on semi-supervised methods could be effective to solve the problem in the future