Incomplete Utterance Rewriting as Sequential Greedy Tagging
Year: 2023

Abstract:
The task of incomplete utterance rewriting has recently gotten much attention. Previous models struggled to extract information from the dialogue context, as evidenced by the low restoration scores. To address this issue, we propose a novel sequence tagging-based model, which is more adept at extracting information from context. Meanwhile, we introduce speakeraware embedding to model speaker variation. Experiments on multiple public datasets show that our model achieves optimal results on all nine restoration scores while having other metric scores comparable to previous state-of-theart models. Furthermore, benefitting from the model’s simplicity, our approach outperforms most previous models on inference speed.

heading: 1 Introduction
text: Recent years have witnessed increasing attention in dialogue systems mainly due to its promising potential for applications like virtual assistants or customer support systems (Hauswald et al., 2015; Debnath et al., 2018). However, studies (Carbonell, 1983) show that users of dialogue systems tend to use incomplete utterances which usually omit (a.k.a. ellipsis) or refer back (a.k.a. co-reference) to the concepts that appeared in previous dialogue contexts. (also known as non-sentential utterances, (Fernández et al., 2005). Thus, dialogue systems must understand these incomplete utterances to make appropriate responses.
To tackle the problem, the task of Incomplete Utterance Rewriting(IUR, also known as context rewriting) (Su et al., 2019; Pan et al., 2019; Elgohary et al., 2019), which aims to rewrite an incomplete utterance into an utterance that is semantically equivalent but self-contained to be understood without context, has recently become an increasing focus of NLP research. As depicted in Table 1, the incomplete utterance u3 not only omits the subject “深圳”(Shenzhen), but also refers to the
semantics of “经常阴天下雨”(always raining) via the pronoun “这样”(this). The downstream dialogue model only needs to take the last utterance by explicitly recovering the dropped information into the latest utterance. Thus, the burden of long-range reasoning can be primarily relieved, making the downstream dialogue modeling more accurate.
The previous top work on building IUR model mainly includes generation-based methods and tagging-based methods. Generation-based solution (Su et al., 2019; Pan et al., 2019; Elgohary et al., 2019) consider this task as a standard text-generation problem, adopting a sequence-tosequence model with a copy mechanism (Gulcehre et al., 2016; Gu et al., 2016; See et al., 2017). However, those methods generate the rewritten utterance from scratch, which introduces an over-large search space and neglects the critical trait that the main structure of a rewritten utterance is always the same as the incomplete utterance.
In order to break through those limitations, tagging-based approach (Liu et al., 2020; Hao et al., 2021; Jin et al., 2022; Zhang et al., 2022; Wang et al., 2022) was proposed. For specifically, here we consider models like RUN (Liu et al., 2020) as a tagging-based method. Its semantic segmentation
7265
task can be analogous to the sequence tagging task. The main difference is that the semantic segmentation task is tagging in two-dimensional coordinates, while the sequence annotation task is tagging in one-dimensional coordinates.
The previous top tagging-based approach generally formalizes IUR as learning the edit operation and corresponding location. The tagging-based approach enjoys a smaller search space than the generation-based approach and can better utilize the information that the main structure of a rewritten utterance is always the same as the incomplete utterance.
Despite their success, existing approach that learning edit operation and the corresponding location has difficulty handling situations where multiple inserts correspond to one position. Moreover, models like RUN adopt a heavy model that takes ten convolution layers in addition to the BERT encoder, which will increase its training time and slows down its infer speed. More critically, although BERT (Devlin et al., 2019) has shown to be powerful in extracting information, the generally low restoration scores prove that previous BERTbased models are ineffective in extracting the information needed for IUR from historical utterances. Finally, the experimental results of SA-BERT (Gu et al., 2020) demonstrate that explicitly modeling speaker changes has a specific enhancement effect on modeling multi-turn dialogue tasks. The previous approach did not model this critical information.
To address these issues, we propose a novel sequence tagging model named SGT(Sequential Greedy Tagging), which is not based on learning editing operations and can significantly improve
the restoration score and inference speed. Our solution was derived from the following thinking: First, we consider that in the dialogue process, any complete utterance is composed by only of a few fragments. For example, "I love you" includes three components: subject, verb, and object. Even if it is expanded with modifications and qualifications, its composition is still minimal. Based on this insight, we thought it would be possible to build a model to identify the fragments and their order from dialogue history to form a target completed utterance. And then, splice those fragments together in sequence and get the complete utterance. Meanwhile, in order to keep the number of fragments constituting the target rewritten utterance relatively small, we adopt the greedy tagging strategy. Our model will identify all the fragments and their order required to form a completed utterance; each fragment is the longest fragment found in the given order. We might as well call this fragment GLCS (Greedy Longest Common Subsequence). Specifically, we use the tag type to represent the order of GLCS for composing the target rewrite utterance, For example, the first GLCS that constitutes a rewritten utterance would be tagged as A, and the second is B, the third is C, and so on. In the above manner, we converted IUR into a simple sequence tagging task, as illustrated in Figure 1. After the model has identified all GLCSs from the dialogue history through this strategy, the target rewritten utterance can be obtained by splicing each GLCS in alphabetical order according to its tag.
Furthermore, we introduce speaker-aware embedding to model the speaker changes in different rounds. Finally, to better perceive the boundaries of each tagging mention, we add two simple losses in addition to the sequence labeling loss.
In summary, our contributions are as follows:
1. We proposed SGT, a novel paradigm to model IUR. Due to the simplicity and effectiveness of modeling, our approach can fully utilize the sequence labeling capabilities of BERT to extract information from historical utterances and thus restore incomplete utterances with more accuracy. Experiments on several datasets show that our method significantly improved the ability to extract mentions from context, which are argued to be harder to copy by (Pan et al., 2019).
2. To the best of our knowledge, we are the
first to introduce speaker-aware embedding to model IUR.
3. Finally, benefit from the model’s simplicity. Our inference speed is faster than most previous models.

heading: 2 Related Work
text: Earlier efforts (Su et al., 2019; Elgohary et al., 2019) treated dialogue utterance rewriting as a common text generation problem and integrated seq-toseq models with copy mechanisms to model this task. Later work (Pan et al., 2019; Zhou et al., 2019; Huang et al., 2021) explore task-specific features for additional gains in performance. For example, (Pan et al., 2019) adopt a pipeline-based method. The idea is to detect keywords first and then append those words to the context and adopt a pointer generator that takes the output of the first step to produce the output. However, this two-step method inevitably accumulates errors.
SRL (Xu et al., 2020) trains a semantic labeling model to highlight the central meaning of keywords in the dialogue as a sort of prior knowledge for the model. To obtain an accurate SRL model for dialogues, they manually annotate SRL information for more than 27,000 dialogue turns, which is costly and time-consuming.
RUN (Liu et al., 2020) convert this task into a semantic segmentation problem, a significant task in computer vision. In particular, their model generates a word-level edit matrix, which contains the operations of insertion and substitution for each original utterance. Rather than word embeddings, RAU (Zhang et al., 2022) directly extracts ellipsis and co-reference relationships from Transformer’s self-attention weighting matrix and edits the original text accordingly to generate complete utterances. RUN++ (Wang et al., 2022) Introduce contrastive learning and keyword detection tasks to model the problem jointly. Both RAU and RUN++ make significant improvements in most metrics on several datasets. Although some additional effective strategies exist. It is still in the same paradigm as RUN, learning edit matrix by cast IUR as a semantic segmentation task.
RAST (Hao et al., 2021) is the first work to convert dialogue utterance rewriting into a sequence tagging task. It takes experimentation to prove that most models for this task suffer from the robustness issue, i.e., performance drops when testing on a different dataset. By contrast, RAST is more
robust than the previous works on cross-domain situations. Moreover, this work design additional reinforcement learning task to improve fluency. Despite all these efforts, its overall in-domain performance still lags behind methods that learn edit operation matric (Liu et al., 2020).
To better enhance pre-trained language models for multi-turn response selection in retrieval-based chatbots. A model named Speaker-Aware BERT (SA-BERT) (Gu et al., 2020) proposed to make the model aware of the speaker’s changed information, which is an essential and intrinsic property of multiturn dialogues.
Although RAST has a different learning paradigm from works that learn edit matrix, it still tries to learn the edit operation and corresponding location by sequence tagging. As mentioned before, our method is sequence tagging-based but takes an entirely new paradigm that would not learn edit operations. Besides, inspired by SA-BERT (Gu et al., 2020), we introduce speaker embedding to this task. Finally, we introduce two simple sequence labeling tasks to model this problem jointly.

heading: 3 Methodology

heading: 3.1 Task Definition
text: Here we give the formal definition of how we model the IUR problem with the SGT approach. Taking all history utterances H = (U1, U2, ..., Un) as input, SGT aims to learn a function to rewrite Un to R: f(H) → R. R is the target rewritten utterance in the infer stage. In particular, Un is the last utterance of all history utterances and the utterance that needs to be rewritten in the IUR task. R is the reference rewritten utterance Uref in the training phase and the target rewritten utterance in the inference phase.

heading: 3.2 Model Architecture
text: Figure 2 shows the overall architecture of our model.
Contextual Encoder Since pre-trained language models have been proven to be effective in many NLP tasks, our experiment employs BERT (Devlin et al., 2019) to be encoder. For a fair comparison, we take the same BERT-base encoder as the previous sota work (e.g., RUN, RAU, RUN++) to represent each input. Concretely, given input token list H = (x1, x2, · · · , xM ) which concatenated by all utterances of dialogue history and inserted a special
token [SEP] between each utterance for separate utterances in different turns. The BERT encoder is firstly adopted to represent the input with contextualized embeddings and the calculation of this part is defined as:
E = (e1, · · · , eM ) = BERT (H) (1)
Speaker Aware Embedding To distinguish utterances between different speakers, our approach stitches a one-dimensional one-hot vector at the hidden dimension with the output representation of the BERT encoder. This design is based on two considerations. On the one hand, most of the dialogue in the dataset is back-and-forth conversations between only two people. On the other hand, adding speaker embedding at the input layer and performing domain adaptation like SA-BERT will make the encoder different from the BERT-based model, which would contradict the fair comparison conditions we assumed earlier in paragraph 3.2. The calculation of this part is defined as follows::
EA = Concat (Dropout (E) , SA) (2)
In the above equation, E ∈ RM×768 is the output representation from the contextual encoder. SA ∈ RM×S denotes the speaker-aware embedding. We concatenate E and SA alongside its hidden dimension to get EA ∈ RM×(768+S). Sequential Greedy Tagging Our main task is sequential greedy tagging, this can be generally
defined as: Psgt = f(H) (3)
Specifically, H = (x1, x2, · · · , xM ) is the input token list that concatenated by the dialogue’s history utterances. The model learns a mapping function f to predict from H to the token-level sequence labeling matrix Psgt ∈ RMXN , where M is the token number of sequence H , and N is the number of tag types. The objective function is defined as:
Lsgt = 1 M ×N M×N∑
i=0
CE ( P isgt, Y i sgt ) (4)
Where Y isgt is the target type of the i-th sample at the token level. CE is the notation of cross-entropy loss which is the same for both equations 5 and 6.
GLCS Detection and GLCS Edge Detection To better lock in the span of target GLCS needed to make up the rewritten utterance, we introduced multi-task learning.
Firstly, as depicted by the red components on the right side of Figure 2, the GLCS Detection module (GD) is a binary classification task to distinguish whether a token should belong to a target GLCS. The module GD outputs Pgd ∈ RMX1. LD is essentially a sequence tagging problem, and the loss function of the GLCS detection is as follows:
Lgd = 1
M
M∑
i=0
CE ( P igd, Y i gd ) (5)
Y igd is the golden mentions label of the i-th sample. P igd is the predicted mentions label of the i-th sample.
Secondly, as depicted by the green components on the right side of Figure 2, the GLCS Edge Detection module (GED) is a binary classification task with a structure similar to GD. Specifically, a target that consists of a single token or only two tokens will be marked throughout as 1; only tokens at its start position and end position will be marked as 1 when more than three tokens, left with the others as 0. The loss function of the GED is as follows:
Lged = 1
M
M∑
i=0
CE ( P iged, Y i ged ) (6)
Y iged is the golden mentions label of the i-th sample. P iged is the predict mentions label of the i-th sample.
Final Learning objectives Finally, we combine all tasks and train them simultaneously by taking the summation of all loss functions, and the final loss function is shown below:
Lfinal = Lgd + Lged + Lsgt (7)

heading: 3.3 Data Construction
text: The construction of the training data for the SGT task is shown in Figure 3. First, in step S1, we make U
(1) ref = Uref , then find the LCS between each history utterance and U1ref separately. Also, this LCS needs to satisfy being a prefix of U (1)ref . After step S1, we can get the first GLCS “深圳”(Shenzhen), and we set the label of its corresponding position to "AA." Then, in step S2, we make U (2) ref = (U (2) ref remove the prefix “深圳”(Shenzhen)). Performing the same GLCS search process, we can obtain the second GLCS “冬天就是”(winter is) and set its label as "BBBB." Analogously, we can get the third GLCS “经常阴天下雨”(always cloudy and raining) and set its label as "CCCCCC" at Step S3. Finally, the historical utterances are stitched together as the input of the SGT task. The corresponding labels obtained from steps S1, S2, and S3 are used as the labels of the sequence labeling task.
Points need to be clarified: (i) Granularity The token sequence is char level for Chinese and word level for English and numbers, both in the GLCS
S1: Find LCS between each utterance and the prefix of uref (1)
S2: Find LCS between each utterance and the prefix of uref (2)
S3: Find LCS between each utterance and the prefix of uref (3)
ref (3) ref (2)
ref (2) ref (1)u = u remove the prefix “深圳”
u = u remove the prefix “冬天就是”
u1 深圳最近天气怎么样 A A OOOOOOO
O O OOOOOO
O O OOOO
u2 最近经常阴天下雨
u3 冬天就是这样
uref (1) 深圳冬天就是经常阴天下雨
u1 深圳最近天气怎么样 A A OOOOOOO
O O OOOOOO
B B OOBB
u2 最近经常阴天下雨
u3 冬天就是这样
uref (2) 冬天就是经常阴天下雨
S4: Concat each utterance as input and concat each tag as label
u1 深圳最近天气怎么样 A A OOOOOOO
O O CCCCCC
B B OOBB
u2 最近经常阴天下雨
u3 冬天就是这样
uref (3) 经常阴天下雨

heading: A A O O O O O OOB B B BCCCCCCOOOOOO
text: matching phase of S1, S2, and S3 and in the training phase of data obtained from S4, which is the same as RUN; (ii) Duplicate Matching In case of duplicate matches, e.g., if U1 and U2 have the same desired GLCS, the principal is the latter, the better.

heading: 4 Experiments
text: In this section, we conduct through experiments to demonstrate the superiority of our approach.
Datasets We conduct experiments on three public datasets across different domains: Chinese datasets in open-domain dialogues: MULTI (Pan et al., 2019) and REWRITE (Su et al., 2019) , English Task-Oriented Dialogue TASK (Quan et al., 2019). For a fair comparison, We adopt the same data split for these datasets as our baselines. The statistics of these datasets are displayed in Table 2.
Baselines To prove the effectiveness of our approach, we take the State-of-the-art models as strong baselines including SRL (Xu et al., 2020), SARG (Huang et al., 2021), PAC (Pan et al., 2019), RAST (Hao et al., 2021), T-Ptr-λ (Su et al., 2019), RUN (Liu et al., 2020) and RUN++ (Wang et al., 2022) .
Evaluation We employ credible automatic metrics to evaluate our approach. As in literature (Pan et al., 2019), we examine SGT using the widely used automatic metrics BLEU, ROUGE, EM and Restoration F-score. (i) BLEUn (Bn) evaluates how similar the rewritten utterances are to the golden ones via the cumulative n-gram BLEU score (Papineni et al., 2002). (ii) ROUGEn (Rn) measures the n-gram overlapping between the rewritten
utterances and the golden ones, while ROUGEL (RL) measures the longest matching sequence between them (Lin, 2004). (iii) EM stands for the exact match accuracy, which is the strictest evaluation metric. (iv) Restoration Precisionn, Restoration Recalln and Restoration F-scoren (Pn, Rn, Fn) emphasize more on words from dialogue context which are argued to be harder to copy (Pan et al., 2019). Therefore, they are calculated on the collection of n-grams that contain at least one word from context utterance. As validated by Pan et al. (2019), above automatic metrics are credible indicators to reflect the rewrite quality.
Implementation Our implementation was based on PyTorch (Paszke et al., 2019) and fastNLP (Xipeng Qiu, 2018). In practice, we adopt the exact connection words setting with RUN and append the list of connection words to the head of H , as part of it. Considering that only two speakers are in the datasets related to our experiments, we set the hidden_size of SA to 1. For encoding different tagging types, We choose IO encoding, the simplest tag encoding schema, which tags each token as either being in (I-X) a particular type of named entity type X or in no entity (O). Since the distribution of tag types is severely unbalanced (e.g. (O) accounts for more than 81% on MULTI), we employed weighted cross-entropy loss and tuned the weight on development sets. We used Adam (Kingma and Ba, 2014) to optimize our model and set the learning rate as 2e-5. We set the dropout rate as 0.3 for the dropout operation on the equation 2. For a fair comparison, the BERT used in our model is BERT-base which is the same as our baselines.

heading: 4.1 Model Comparison
text: Table 3, Table 4, and Table 5 show the experimental results of our approach and baselines on MULTI and REWRITE. As shown, our approach greatly surpasses all baselines on practically all restoration scores significantly. Taking MULTI as an example, our approach exceeds the best baseline RUN++(PCL) on restoration score by a significant margin, reaching a new state-of-theart performance on almost all restoration metrics. Our approach improves the previous best model by 9.79 points and 9.89 points on restoration F3 and F2, respectively. Furthermore, our approach reaches comparable performance on other auto metrics. As demonstrated by the result of REWRITE, our approach achieves comparable performance
.
on the B4, R2, and RL scores and a new state-ofthe-art performance on B1 and R1 scores. Even for the most strict metric EM on REWRITE, our approach reached comparable performance with RUN, demonstrating the comprehensive ability of our model. Besides, our approach achieves better results against all baselines on TASK, as depicted in Table 5. Specifically, we achieve state-of-the-art performance on the EM score and exceed the previous best model by 16.7 points on the restoration F1 score. Finally, the combined performance of our model on the three datasets above demonstrates that our model can perform well on datasets with
varied languages and tasks.

heading: 4.2 Closer Analysis
text: We conduct a series of experiments to analyze our model thoroughly. First, we conduct a detailed ablation study to validate the efficacy of the components in our model. Then, in the same run-time setting, we compare the inference speed of our model to that of representative baselines.
Ablation Study By analyzing table 6, we can find that “w/o sa”,“w/o gd” or “w/o ged” basically hurts the effect of the model, and these can initially corroborate that each of these modules is beneficial to our model.
Meanwhile, we can find that “w/o gd+ged” significantly reduces R3, indicating that these two subtasks are very helpful for discovering the potential target GLCS. Further, we find that although removing “sa” alone has little effect on the restoration score, comparing the results of removing “gd+ged” and removing “sa+gd+ged” reveals that the fit with the missing speaker-aware information significantly reduces the restoration score. The F3 decreases from 74.7 to 71.8, which indicates that the information of different speakers or rounds is
crucial to extract the target GLCS correctly, and combining “sa” embedding and “gd+ged” subtasks can significantly improve the model’s ability to obtain the target GLCS fragments from the context.
Finally, we find that even though the absence of the three critical components “sa+gd+gd” leads to an overall decrease in model performance, our model still achieves a better restoration score than the RUN model, which further validates the effectiveness of our sequential greedy tagging learning strategy for modeling and solving UIR problems.
Inference Speed As shown in Table 7, both SGT and RUN significantly outperform traditional generation algorithms regarding inference speed and B4 score. At the same time, the most time-consuming
computation of SGT in the inference phase, except for the BERT encoder, is only one layer of a linear transformation, which dramatically saves the inference time compared with RUN, which has Unet (Ronneberger et al., 2015) structures after the context encoder. Therefore, we can see that the inference time of SGT is significantly less than that of RUN. The latency of a single rewriting task is reduced by 20ms, while the B4 score slightly better.

heading: 5 Conclusion
text: In this paper, we convert the IUR problem into a simple sequence tagging task, SGT. The simplicity and effectiveness of the modeling paradigm not only improve the inference speed and allow the pretrained BERT encoder to fully exploit its widely validated information extraction ability which can significantly improve the restoration score and ensure that other metrics are competitive. We also introduced speaker-aware embedding to explicitly model speaker changes and verified that it has some improvement effect on the IUR task.
In the future, we will explore the following directions:
1. Adopt the GD task in this paper to extract essential fragments and then pick the best permutation of fragments with a language model or using a PAC-like pointer network for fragment integration to get rid of the problem of category imbalance is caused by representing the order with tag lists.
2. Combining SGT’s efficient fragment extraction paradigm with generation.
Limitations
Although our model has made some progress, it still has some limitations. First of all, SGT uses the tag type to represent the connection order of GLCS fragments when forming a complete utterance, and the average statistics on the three datasets we used show that more than 99% of the complete utterance can be composed with less than three GLCS fragments. That will lead to situations that need to combine multiple GLCSs (e.g., more than 3) to form a complete utterance, which cannot be fully trained or fall into unbalanced tag categories. Second, like other tagging-based models, the fragments that make up the complete utterance must exist in history utterances or connection words, which does not work well for situations where it is necessary to combine context information and introduce new words to express their complete utterance.
Ethics Statement
We guarantee that the approach in our research is original and that the experimental results of the SGT model are reproducible. All the experimental data of the baseline model can be reproduced from the relevant open-source code or found in the cited paper. Finally, The list of authors of the submissions does not include any individuals who did not contribute substantially to work submitted.

heading: Acknowledgements
text: First, we thank all the anonymous reviewers for their valuable comments. Moreover, we are grateful for all the previous work related to exploring the IUR task, which has inspired us a lot.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 8
3 B1. Did you cite the creators of artifacts you used? 8
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will explore the following directions:
1. Adopt the GD task in this paper to extract essential fragments and then pick the best permutation of fragments with a language model or using a PAC-like pointer network for fragment integration to get rid of the problem of category imbalance is caused by representing the order with tag lists. 2. Combining SGT’s efficient fragment extraction paradigm with generation. Limitations
Although our model has made some progress, it still has some limitations. First of all, SGT uses the tag type to represent the connection order of GLCS fragments when forming a complete utterance, and the average statistics on the three datasets we used show that more than 99% of the complete utterance can be composed with less than three GLCS fragments. That will lead to situations that need to combine multiple GLCSs (e. g. , more than 3) to form a complete utterance, which cannot be fully trained or fall into unbalanced tag categories. Second, like other tagging-based models, the fragments that make up the complete utterance must exist in history utterances or connection words, which does not work well for situations where it is necessary to combine context information and introduce new words to express their complete utterance