Focus-aware Response Generation in Inquiry Conversation
Year: 2023

Abstract:
Inquiry conversation is a common form of conversation that aims to complete the investigation (e.g., court hearing, medical consultation and police interrogation) during which a series of focus shifts occurs. While many models have been proposed to generate a smooth response to a given conversation history, neglecting the focus can limit performance in inquiry conversation where the order of the focuses plays there a key role. In this paper, we investigate the problem of response generation in inquiry conversation by taking the focus into consideration. We propose a novel Focusaware Response Generation (FRG) method by jointly optimizing a multi-level encoder and a set of focal decoders to generate several candidate responses that correspond to different focuses. Additionally, a focus ranking module is proposed to predict the next focus and rank the candidate responses. Experiments on two orthogonal inquiry conversation datasets (judicial, medical domain) demonstrate that our method generates results significantly better in automatic metrics and human evaluation compared to the state-of-the-art approaches.

heading: 1 Introduction
text: Thanks to the high effectiveness of machine learning techniques, natural language processing (NLP) has made tremendous progress in a variety of tasks; for example, in conversation response generation which empowers many applications such as chatbots (e.g., Siri). The performance of response generation was significantly improved after applying neural network models such as recurrent neural networks (RNN) (Cho et al., 2014; See et al., 2017) and Transformers (Vaswani et al., 2017; Ji et al., 2020). However, existing studies on response generation mainly concentrate on relevance and fluency, rarely paying attention to the focus, which is important from the viewpoint of the rationality of generated responses.
Inquiry conversation (inquiry dialogue) is a common form of conversation that aims to complete the investigation (Hamami, 2014) (e.g., court hearing, medical consultation, police interrogation). Focus shifts tend to occur often in inquiry conversation, and their order plays a key role. For example, as shown in Fig.1, a judge will not issue the verdict before the defendants have finished pleading, and the doctor will not prescribe drugs before stating a diagnosis. The latent focuses of utterances often affect dialogue development, and hence it is beneficial to incorporate the notion of focus in the response generation process.
In this paper, we focus on response generation in inquiry conversation, and we aim at improving the rationality of generated content. For practical reasons, we only generate the responses of the leading role speaker in a conversation (e.g., judge, doctor). When addressing this problem one faces the following challenges: (1) The focuses are sequential yet latent. The next response should be generated considering the focuses underlying in the conversation history, and the next focus needs to be predicted. (2) The focuses are discrete and different focuses correspond to different responses. Thus, the generator needs to determine the focus before and then generate a response guided by the established focus.
To address these challenges, we propose a novel focus-aware response generation (FRG) method by jointly optimizing a multi-level encoder, a set of focal decoders (to generate responses with different focuses), and a synergistic focus ranking module. Specifically, the multi-level encoder is designed to better learn the latent focuses from the conversation history based on the aggregated characteristics of speakers and the content in each block (defined in Sec.3) through a speaker level attention layer and a block level attention layer. Then, each decoder in the set of focal decoders generates a candidate response guided by its corresponding focus. Finally,
12585
the focus ranking module ranks all the candidate responses generated by the focal decoders and predicts the next focus for the final output.
To test the proposed method, we employ two inquiry conversation datasets from two diverse domains - court hearing and medical consultation. Due to the difficulty and high cost of annotating focuses in different domains which typically require input from domain experts, we use a two-stage training paradigm to assure the generalizability of our method. We first warm-up the decoders together with a large number of unlabeled data to ensure the generation ability, and then we fine-tune them separately on a small number of labeled data to ensure the generation quality of particular focus. Extensive experiments show that the proposed FRG model achieves the best performance on both automatic metrics and human evaluation.
To sum up, our contributions are as follows: • We investigate the response generation task in
inquiry conversation by involving the focus in the generation process.
• We propose a novel focus-aware response generation (FRG) method by jointly optimizing a multi-level encoder, a set of focal decoders, and a synergistic focus ranking module.
• We validate the performance of the proposed method with extensive experiments on two orthogonal inquiry conversation datasets. The ex-
periments indicate the high domain adaptability of our approach.
• To motivate other researchers to investigate this task, we make the code publicly available 1.

heading: 2 Related Work

heading: 2.1 Conversational NLG
text: Neural language generation (NLG) has been widely studied and applied in many tasks including machine translation (Wu et al., 2016; He et al., 2018; Shen et al., 2019), question answering (McCann et al., 2018; Bagchi and Wynter, 2013) and text summarization (Rush et al., 2015; Liu and Lapata, 2019; Wu et al., 2020, 2022). Existing NLG methods can be divided into rule-based and neural-based. The rule-based methods generate content through manually formulated templates (Yang et al., 2019; Becker, 2002). Such responses tend to be smooth and regular, but the cost of formulating templates is quite high. The neural-based methods take the advantage of deep learning (Shen et al., 2021; Zhang et al., 2022a,b; Li et al., 2022a,b; Zhang et al., 2023; Qian et al., 2023; Ma et al., 2021), which requires far less labor and enables flexibility. Bahdanau et al. (2015) firstly applied the attention mechanism into the NLG task. See et al. (2017) proposed a PointerGenerator Networks (PGN), which can solve the Out-Of-Vocabulary (OOV) problem.
1https://github.com/wuyiquan/FRG
In conversational scenarios, many relevant NLG techniques have been also proposed, such as dialogue summarization (Chen and Yang, 2020), chatbots (Li et al., 2016), and response generation (Zhou et al., 2018b). In our work, we focus on the task of response generation for inquiry conversation.

heading: 2.2 Response Generation
text: Response generation is a key task in NLG, which aims to generate a response based on the conversation history (Zhou et al., 2018a,b; Zeng and Nie, 2021). Several approaches have been proposed to improve generation performance. Xing et al. (2017) proposed Topic-Aware Neural Response Generation (TAS2S) which incorporates pre-processed topic words to generate the response. Lau et al. (2017) introduced a Topically Driven Neural Language Model (TDLM) method, which can generate a response based on the predicted topic embedding. Lei et al. (2021) applied a hierarchical speakeraware encoder to model the conversation. Zhao et al. (2017) propose a dialogue act-guided generation work, which aims to improve the diversity of the response. Wu et al. (2021) proposed a controllable grounded response generation framework, which uses an explicit hint phrase to generate. Due to the popularity of pre-training, several pre-trained models have been employed for response generation task, such as TransferTransfo (Wolf et al., 2019) and DialoGPT (Zhang et al., 2020b).
In this work, we emphasize the focus shifts among the blocks in the conversation, therefore the block level attention module is proposed for capturing their sequences. In addition, our model uses a set of focal decoders to generate a ranking list of responses corresponding to the predicted focus which is more applicable in practical use.

heading: 3 Problem Formulation
text: In this section, we define the problem of response generation in inquiry conversation. We first describe the key concepts as below: Inquiry conversation is a form of conversation that aims to complete an investigation (Hamami, 2014) (e.g., court hearing, medical consultation). Focus is the center of the conversation at a certain stage of its progress. The focuses tend to shift during the conversation. Leading role is the speaker (e.g., interrogating speaker such as a judge, doctor) who controls the
focus shifts in the inquiry conversation. Block consists of several consecutive utterances and is regarded as the smallest unit of the focus shifting. Therefore, the conversation can be divided into several blocks according to the actions of the leading role speaker. Response utterance refers to the interrogating utterance of the leading role speaker (examples are shown in Figure 1).
The problem of response generation in inquiry conversation is then defined here as follows:
Given the conversation history U = {(ut, st)}nut=1 where {(ut, st)} is tth pair of utterance ut and the role of speaker st, the task is to determine the next focus f and based on it generate the corresponding response denoted as r = {wt}mt=1 for the leading role.

heading: 4 Method
text: In this section, we describe our focus-aware response generation (FRG) model. Fig.2 shows the overall framework. Our model consists of a shared multi-level encoder, a focus ranking module, and a set of focal decoders. The model works in a multitask learning manner. The ranking module and decoders take the output of the encoder as an input.

heading: 4.1 Multi-level Encoder
text: The multi-level encoder consists of four layers, which encode the input from different levels. Firstly, we introduce two kinds of special tokens: (1) Speaker token 
 indicates the end of a speaker’s utterance, where s is the id of the speaker. (2) Block token <b> refers to the end of a block. A block consists of several consecutive utterances with the same focus and is set automatically according to the speaking action of the leading role speaker (e.g., judge, doctor). For example, in Fig.2, the blocks are created every time the judge speaks. The input is transformed to:
I = {u1,<s1>,u2,<s2>,<b>,u3, <s3>, ...,unu ,<snu>,<b>},
where u is the utterance, s is the corresponding speaker, nu is the number of utterances. Note that since we only generate responses for the leading role speaker, I will always end with a <b>.
The input is a sequence of tokens. We then first transform the tokens into embeddings. The special tokens mentioned above are randomly initialized.

heading: 4.1.1 Utterance Level Layer
text: In this layer, the embeddings of tokens are fed into a bidirectional LSTM (Bi-LSTM) (Huang et al., 2015), producing a token-level representation of the input ht = Bi-LSTM(I).
To obtain a representation for each utterance, we take the output of the speaker token for that utterance. Thus, the utterance-level representation of the input is hu = {htk}, k∈XS , where XS is the set of speaker token indices in I .
To obtain a representation for each block, we take the output of the block token for that focus block. Thus, the block-level representation of the input is hb = {htk}, k∈XB , where XB is the set of block token indices in I .

heading: 4.1.2 Speaker Level Attention Layer
text: In the conversation, different speakers will play different roles. In order to obtain the speaker level representation, we create a special mask M according to the speaker’s id. M is a matrix with the dimension of [nu, nu]. For any mi,j in M :
mi,j =
{ 1 si = sj
0 si ̸= sj . (1)
where si is the speaker of the utterance of ui.
Given the utterance-level representation hu and the mask M , the speaker-level representation hs is calculated as follows:
hs = softmax ( Q⊤s KsM√
dks
) Vs
Qs = WQsh u,Ks = WKsh u, Vs = WV sh u (2) where WQs, WKs, WV s are learnable parame-
ters, and dks is the dimension of Ks.

heading: 4.1.3 Block Level Attention Layer
text: In inquiry conversation, we assume the focus shifts only when the leading role speaker speaks, by which we divide the conversation history into several blocks.
Given the block-level representation hb, we run a self-attention on it, and the final block-level representation hb ′ is calculated as follows:
hb ′ = softmax ( Q⊤f Kf√
dkf
) Vf
Qf = WQfh b,Kf = WKfh b, Vf = WV fh b
(3) where WQf , WKf , WV f are learnable parameters, and dkf is the dimension of Kf .

heading: 4.1.4 Conversation Level Layer
text: In this layer, we concatenate the output of the former layers to get hcon. For each hti in the h
t, we concatenate it with its corresponding speaker-level representation and block-level representation:
hconi = [h t i;h s x(i);h b′ y(i)] (4)
where x is a function mapping the index of ht and hs, y is a function mapping the index of ht and hb ′ and [·; ...; ·] represents the concatenation operation. Then we use another Bi-LSTM layer to obtain the final representation of the input h = Bi-LSTM(hcon).

heading: 4.2 Focal Decoders
text: In order to make the model generate a reasonable response, we use a set of decoders with the same structure that aim to generate responses guided by different focuses. We call them focal decoders. Specifically, the number of decoders is equal to the number of predefined focuses.
Given the representation of the input h and the decoding state st, we apply the attention mechanism (Bahdanau et al., 2015). At each step t, the attention distribution at is calculated as follows:
eti = v T tanh (WHhi +WSst + battn) at = softmax ( et ) (5)
where v, WH , WS , battn are learnable parameters. The context vector h∗t is the weighted sum of h,
such that h∗t = ∑ i a t ihi.
Then, the context vector h∗t is concatenated with the decode state st and fed to linear layers to produce the vocabulary distribution pvoc:
pvo = softmax(V ′(V [st;h∗t ]) + b) + b ′) (6)
where V , V ′, b, b′ are all learnable parameters. We use a generation probability (See et al., 2017) to solve the OOV problem. Given the context h∗t , the decode state st and the decoder’s input xt, the generation probability pgen is calculated as follows:
Pgen = σ(w T h∗h ∗ t + w T s st + w T x xt + bptr) (7)
where wh∗ , ws, wx and bptr are learnable parameters, and σ is the sigmoid function.
The final probability for a word w for the current time step is obtained:
P (w) = Pgen∗pvoc(w)+(1−Pgen) ∑
i:wi=w
ati (8)
Given the same h, the decoders will generate different outputs due to the different parameters. We explain how to warm-up and independently fine-tune the decoders in the training part.

heading: 4.3 Focus Ranking Module
text: Given the representation of the input h, the focus ranking module will produce the probability of each focus through a fully connected layer and a softmax operation. The ranking score rs = {rs1, rs2, ..., rsnf } is obtained as rs = softmax(FC(mean(h))), where FC denotes a fully-connected layer. Then, the outputs of the decoders can be sorted by the rs.

heading: 4.4 Two-Stage Training Paradigm
text: Since the annotation of the focus is difficult and costly, we adopt a two-stage training paradigm to assure the high generalization ability of our method.
In the first stage, we use a large number of unlabeled data to train the model without the ranking module, aiming to let the decoders acquire a good generation ability. Here, all the decoders share the same parameters.
For the decoders, the loss for time step t is the negative log-likelihood of the target word w∗t : Lt = −logP (w∗t ), (9) and the overall generation loss is:
Lgen = 1
T
T∑
t=0
Lt, (10)
where T is the length of the response utterance. In the second stage, we use a small number of labeled data to train the ranking module and finetune the encoder and decoders trained in the first stage. In this stage, each decoder corresponds to a different focus, and the decoders will be trained by the data annotated to their corresponding focus.
For the ranking module, we use cross-entropy as the loss function:
Lrank = − nf∑
i=1
yi log (rsi) , (11)
where yi = 1 if i = f , otherwise, yi = 0. f is the annotated focus. nf stands for the number of focuses.
For the set of focal decoders, we take a mask operation when calculating the loss of each decoder.
The actual loss for the decoder di is:
Li = { Lgen(i) f = i 0 f ̸= i , (12)
where i is the corresponding focus of di and Lgen(i) is the generation loss of di.
Thus, the total loss in the second training stage is:
Ltotal = nf∑
i=1
Li + λ ∗ Lrank, (13)
where we set λ to 0.1 ∗ nf .

heading: 4.5 Inference
text: During inference, the decoders apply beam search with the size of 4 to generate candidate outputs, which will be sorted by the ranking score rs.

heading: 5 Experiments

heading: 5.1 Dataset
text: We use the following two datasets for experiments: Court Hearing and Medical Consultation.
Court Hearing dataset:2 Court hearing is a judicial event where the judge inquires the plaintiff and the defendant in order to clarify the facts of the case. The annotated data we use is released by Duan et al. (2019) 3. The input is the conversation history, and the output is the next response utterance of the judge. There are seven focuses in this dataset: Principal, Interest, Common debt claim, Guarantee liability, Liquidated damage, Creditor qualification, Limitation of action.
Medical Consultation dataset:4 Medical consultation is a conversation between a patient and a doctor. The annotated dataset we use is released by the competition: Conference on Knowledge Graph and Semantic Computing 2021 (CCKS21)5. There are five focuses for this dataset: Symptom, Medicine, Test, Attribute, Disease.
The statistics of the two datasets are shown in Tab.1. We randomly separate each dataset into a training set, a validation set, and a test set according to a ratio of 80%:10%:10%. The annotated data is ensured not to be in the test set.
2This dataset is provided by the High People’s Court of a province in China.
3https://github.com/zhouxinhit/Legal_Dialogue_ Summarization.
4The raw data can be downloaded in https://github. com/UCSD-AI4H/Medical-Dialogue-System.
5The data can be downloaded in https://www.biendata. xyz/competition/ccks_2021_mdg/data/

heading: 5.2 Evaluation Metrics

heading: 5.2.1 Automatic Evaluation
text: We adopt ROUGE6, BLEU (Papineni et al., 2002) and BERTScore (Zhang et al., 2020a) as the automatic metrics. Specifically, we report the values of ROUGE-1 and ROUGE-L for ROUGE; BLEU-1 and BLEU-N (average of BLEU-1 to BLEU-4) for BLEU; P, R and F1 for BERTScore.

heading: 5.2.2 Human Evaluation
text: We conduct a human evaluation to analyze the quality of the generated responses. We randomly sample 500 test cases from each dataset. For each case, we present the responses generated by 5 representative methods7 together with the ground truth to 5 annotators. The evaluation is conducted following two perspectives: (1) Rationality level. The rationality indicates the logical coherence between the conversation history and the generated response. Annotators are asked to give a score on the rationality of the generated response. (2) Fluency level. Annotators are asked to give a score on the fluency of the generated response. Both scores range from 1 to 5 (1 for the worst and 5 for the best).

heading: 5.3 Baselines
text: We employ the following methods as baselines for comparison with our approach:
L-Distance (Levenshtein distance) is used to measure the difference between two texts. Given the input of the test case, we find out the case in the training dataset with the smallest L-distance and take its response as the output. This method performs in a text retrieval manner. LSTM+ATT (Sutskever et al., 2014) and PGN (See et al., 2017) are RNN-based models. T5 (Raffel et al., 2020) and GPT-2 (Radford et al., 2019) are transformerbased models for NLG task. We also fine-tune them on the task datasets. TransferTransfo (Wolf
6https://pypi.org/project/rouge/ 7We shuffle all the results to make fair evaluation for all
the methods.
et al., 2019) and DialoGPT (Zhang et al., 2020b) are dialogue pre-trained models. We fine-tune them on task datasets. TDLM (Lau et al., 2017) predicts focus embedding first, then sends the focus embedding to the decoder to form responses. TAS2S (Xing et al., 2017) predicts focus words first, then takes the focus words as the external vocabulary to the decoder. MPG (Ide and Kawahara, 2021) uses multi-task learning to simultaneously predict the focus and generate the response.8
FRG-top1 indicates that we choose as the output the content generated by the decoder which has the highest ranking score, while FRG-top3 means that we take the three top-ranked candidates at the same time. The latter simulates a practical scenario that a user could select an appropriate answer from the suggested candidates.
We also conduct the ablation experiments on FRG-top1 as follows: FRG w/o RM removes the ranking module and replaces the set of decoders with a single decoder. FRG w/o ML removes the speaker level attention layer and block level attention layer. FRG w/o BL removes the block level attention layer. FRG w/o SL removes the speaker level attention layer.
8The focus is called as topic in TDLM and TAS2S, while it is called as emotion label in MPG.

heading: 5.4 Experimental Results
text: In this section we analyze the experimental results9. Quantitative evaluation. Tab.2 demonstrates the results of response generation on both Court Hearing and Medical Consultation datasets with ROUGE, BLEU, and BERTScore.
Based on the results, we make the following observations: (1) L-Distance method has the worst performance in both datasets, which means that simply retrieving the response from the dataset based on the context similarity is not promising. (2) RNN-based baselines and Transformer-based baselines achieve similar performance in this task yet much lower than the performance of FRG. It demonstrates that with the help of a multi-level encoder and focal decoders, FRG is capable of estimating the focus of the leading role speaker and thus generating more precise content. (3) Models that employ annotations achieve better performance, which proves the usefulness of considering the focus. (4) TDLM and TAS2S show that merging the focus embedding into the decoder brings only a small improvement, which suggests the positive effect of the focal decoders. (5) Moreover, FRG also seems to have good domain adaptability by achieving the best performance on both Court Hearing and Medical Consultation datasets compared with the baselines.
To investigate the effects of the number of annotations in the second training stage, we study the performance change in Fig. 4 and draw the following conclusions: (1) A small number of annotations can bring significant improvement to the
9The detailed parameter settings are shown in Appendix and the code will be released.
Conversation History
Conversation History
model (e.g., boosting ROUGE-L from 16.3 to 25.8 for the Court Hearing dataset). With the increase of the number of annotations, the performance of the model continues to improve. (2) The effect of annotations on judicial domain data is stronger than that for the medical domain. It indicates that the number and the granularity of the focuses used may influence the performance.
Qualitative evaluation We show the result of human evaluation in Tab. 3, and report the following observations: (1) Although L-Distance has high performance in fluency due to its retrieval method, it achieves very poor results in focus rationality. (2) Thanks to the focal decoders, FRG significantly improves the performance at the rationality level. (3) FRG also achieves better performance on fluency level compared to other generative methods. (4) Kappa coefficient κ between any two human annotators is above 0.8, which indicates the high quality of human evaluation.
Ablation Study We report the results of ablation
study in Tab. 2 noticing a dramatic decrease in the performance of FRG w/o RM (e.g., decrease from 33.3 to 19.6 on R-1 in Court Hearing dataset), which points to the high importance of ranking module and focal decoders. Similarly, the FRG w/o ML, FRG w/o BL and FRG w/o SL also experience a decrease in performance, albeit less than FRG w/o RM. This confirms the effectiveness of the proposed block level attention layer and speaker level attention layer in the encoder.

heading: 5.5 Case Study
text: Fig. 3 shows two cases of the responses generated by our method (FRG) and by the four baseline methods to provide a more intuitive understanding of the performance of each method. We find that the output of L-Distance is irrelevant to the conversation history. The utterances generated by PGN, GPT-2 and TAS2S are more likely to repeat the content already spoken in the conversation history. FRG is able to generate more reasonable content thanks to the guidance of the focus.

heading: 5.6 Error analysis
text: To explore the limitations of our model, we also analyze generated responses that had a high error rate10, then we summarize the problems that occur, and also explore optimization solutions.
After conducting statistical analysis, we make the following observations: (1) FRG performs worse when external information needs to be used.
10We collect the samples in human evaluation whose either rationality or fluency score of FRG-top1 equals 1.
In the Court Hearing dataset, 27% of errors are related to this problem (e.g., "According to the law, the maximum interest rate shall not exceed four times the interest rate of similar bank loans."). At the same time, 38% of errors in Medical Consultation dataset are related to such problem (e.g., "According to the instructions, Trimebutine Maleate tablets and Golden Bifid can be taken after meals."). (2) 36% of errors in Court Hearing dataset and 47% of errors in Medical Consultation dataset occur when a long response needs to be generated (e.g., more than 25 tokens). (3) Long conversation history (e.g., more than 10 utterances) will also cause a high error rate. This is the case of 42% of errors in Court Hearing dataset and 53% of errors of Medical Consultation dataset.
To address these problems, constructing a retrieval database and enhancing the long dependence of language models can be promising for the future.

heading: 6 Conclusion and Future Work
text: In this paper, we investigate the response generation task in inquiry conversation from a focal view and propose a novel focus-aware response generation (FRG) method. We design a multi-level encoder to represent the conversation history at different levels, as well as a set of focal decoders to generate responses guided by different focuses. Thanks to the focus ranking module, the generated responses are sorted for the final output. The experiment results show the effectiveness of our method.
In the future, we will explore the following directions based on the FRG method: (1) Adding external knowledge to constrain the ranking module and (2) Using the feedback of users to optimize the ranking module in practical applications.

heading: 7 Limitations
text: In this section, we discuss the limitations of our work as follows: • As described in the paper, our proposed method
requires annotations of the latent focus; a small number of annotations (around 250 labeled samples per focus) can already bring a significant improvement (see Fig.4). Therefore when applying our approach to other domains it is necessary to prepare at least a few annotations.
• As mentioned in the error analysis section, the model is unable to generate unseen entities, such as specific drug names or laws. Further improve-
ment should be made to solve this problem for practical use.

heading: Acknowledgments
text: This work was supported in part by Key R&D Projects of the Ministry of Science and Technology (2020YFC0832500), National Natural Science Foundation of China (62006207, 62037001, U20A20387), the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJUSIAS-0010), Project by Shanghai AI Laboratory (P22KS00111), Program of Zhejiang Province Science and Technology (2022C01044), the Fundamental Research Funds for the Central Universities (226-2022-00143, 226- 2022-00142) and MOE Engineering Research Center of Digital Library.
Finally, we would like to thank the anonymous reviewers for their helpful feedback and suggestions.

heading: A Appendices
text: A.1 The Settings of Parameters All models are trained on 2 V100 GPU(16GB). The settings of parameters of our model are shown in Tab. 4.The train/eval/decode step is the same as https://github.com/becxer/ pointer-generator.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 4

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 4
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 4
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will explore the following directions based on the FRG method: (1) Adding external knowledge to constrain the ranking module and (2) Using the feedback of users to optimize the ranking module in practical applications