How Well Apply Simple MLP to Incomplete Utterance Rewriting?
Year: 2023

Abstract:
Incomplete utterance rewriting (IUR) aims to restore the incomplete utterance with sufficient context information for comprehension. This paper introduces a simple yet efficient IUR method. Different from prior studies, we first employ only one-layer MLP architecture to mine latent semantic information between joint utterances for IUR task (MIUR). After that, we conduct a joint feature matrix to predict the token type and thus restore the incomplete utterance. The well-designed network and simple architecture make our method significantly superior to existing methods in terms of quality and inference speed1.

heading: 1 Introduction
text: Multi-turn dialogue modeling is a research area focusing on developing systems that can engage in multiple conversation turns with humans. This type of modeling is often used in the field of humanmachine interaction to improve the ability of artificial intelligence systems to communicate with humans in a natural and intuitive way. One of the challenges of multi-turn dialogue modeling is to accurately understand and respond to the context and meaning of the conversation, as well as to handle incomplete or ambiguous utterances that may be used for brevity or to convey meaning. As shown in Table 1, the incomplete utterance u3 refers to the semantic of "新冠肺炎" (COVID-19) with "那" (that). The limited context provided by a single utterance, such as u3, can lead to referential ambiguity and semantic incompleteness in downstream applications like retrieval-based dialogue systems, as demonstrated in a study by Ni et al. (2022). In addition, Su et al. (2019) has revealed that coreference and ellipsis are prevalent in more than 70% of utterances, particularly in pro-drop
∗Corresponding author 1Our code is available at https://github.com/
IMU-MachineLearningSXD/MIUR
languages like Chinese. These linguistic phenomena in conversation present a significant challenge for the development of practical conversational AI systems.
To address this issue, recent works (Kumar and Joshi, 2016; Su et al., 2019; Pan et al., 2019; Xu et al., 2020) proposed the Incomplete Utterance Rewriting (IUR) task, which aims to transform an incomplete or context-dependent statement into a self-contained, semantically equivalent one that can be understood without any additional context. As shown in Table 1, IUR (u3 → u′3) task makes the downstream dialogue modeling more precise.
Despite previous works achieving promising results, the speed of autoregressive generation remains a limiting factor. To improve the speed, Huang et al. (2021) fuses the sequence labeling and non-autoregressive generation, which predicts missing elements in incomplete utterance and rewritten utterance. In addition, Liu et al. (2020) formulates IUR as semantic segmentation task based on U-Net (Ronneberger et al., 2015) and achieves better performance at a faster speed. However, above mentioned models are still not simple enough.
In this paper, we propose a simple yet efficient
1567
solution that our model first employs MLP architecture to simultaneously mine the semantic associations between the context utterances and the incomplete utterance, and capture attention information between them. After MLP architecture, we obtain the joint feature maps and further construct the token-pair edit matrix. Finally, the above matrix is edited according to prediction edit type tokens to generate the final rewritten utterance. Experiments show that our approach achieves better performance on several datasets across different domains and languages with low resource costs and a much faster inference speed.

heading: 2 Methodology
text: In this section, we elaborate on our proposed approach. As shown in Figure 1, our method mainly consists of two modules: MLP backbone network and joint feature matrix. For a multi-turn dialogue utterances (u1, u2, ..., ut), we concatenate all the context utterances to produce an m-length word sequence c = (c1, c2, ..., cm) and employ a special mask [SEP ] to separate different context utterances. Meanwhile, all the incomplete utterances are denoted as an n-length word sequence x = (x1, x2, ..., xn).

heading: 2.1 MLP Backbone Network
text: We first concatenate the context utterances and the incomplete utterances to construct a joint m + n length word sequence H =
(c1, c2, ..., cm, x1, x2, ..., xn). Besides, pretrained language models have been found to be highly effective in various natural language processing tasks. Hence, we employ BERT (Devlin et al., 2019) to initialize the word vector matrix H, where H ∈ R(m+n)×768. MLP backbone network contains two MLP blocks. Specifically, the first MLP block is responsible for mining the global semantic association information between context utterances c and incomplete utterance x. The second MLP block aims to learn the confidence level for each word embedding. This further enables the model to focus on important word information. It is important for the follow-up edit type classification, including substitute, insert and none. Each MLP block contains two fully-connected layers and a nonlinearity applied independently. For clarity and simplicity, we exclude the transposition process and the whole process can be represented as:
I∗,i = H∗,i+W2σ(W1LN(H∗,i)), Kj,∗ = Ij,∗+W4σ(W3LN(Ij,∗)), (1)
where i = 1, 2, .., 768, j = 1, 2, ..,m + n and σ represents GELU (Hendrycks and Gimpel, 2016). In addition, MLP backbone contains other standard architectural components: skip-connections (He et al., 2016) and LayerNorm (LN ) (Ba et al., 2016).
In contrast to the approach taken by Tolstikhin et al. (2021), who treated the word vector matrix H as an image and employed 1× 1 convolution on
non-overlapping image patches, we directly input the word vector matrix H into the MLP backbone network. Our operation avoids the loss of semantic spatial information resulting from 1×1 convolution. Furthermore, since the number of words in each utterance varies, we utilize padding operation and copy mechanism (Gu et al., 2016; Zeng et al., 2018) to maintain a consistent sequence length. It is worth noting that our approach employs a one-layer MLP backbone network.

heading: 2.2 Joint Feature Matrix
text: Furthermore, to further capture the relevance between word embeddings, we employ three similarity functions: dot product similarity (dot Sim.), cosine similarity (cos Sim.), and linear similarity (linear Sim.). The word-to-word embeddings relevance between each context utterance’s word embedding Kcm and each incomplete utterance’s word embedding Kxn are captured using a 3- dimensional joint feature matrix J(cm, xn) represented as follows:
J(cm, xn) = [Kcm ·Kxn ; cos(Kcm ,Kxn); linear(Kcm ,Kxn)]. (2)
Finally, we employ BatchNorm (Ioffe and Szegedy, 2015) on joint feature matrix J(cm, xn) to expedite and stabilize the training process. The batch is obtained by computing the mean and variance of the batch activation, which captures global information. After applying the BatchNorm operation, the matrix J(cm, xn) is flattened, and each feature vector is mapped to one of three token types: Substitute, Insert, or None. This generates the token-pair edit matrix.

heading: 2.3 Supervised Label
text: Prior to training our model in the supervised fashion, we need to create word-level labels through the following process to construct our training set. Specifically, we first calculate the longest common subsequence (LCS) between the incomplete utterance and the rewritten utterance. Then, we align the incomplete utterance, the rewritten utterance, and the LCS using a greedy strategy. Finally, we identify the corresponding tokens in the rewritten utterance and mark them accordingly. Please refer to Algorithm 1 in Appendix A for a detailed description.

heading: 3 Experiments

heading: 3.1 Experimental Setup
text: Datasets We conduct the experiments on three IUR benchmarks from different domains and languages, including RESTORATION-200K (Pan et al., 2019), REWRITE (Su et al., 2019) and CANARD (Elgohary et al., 2019). The statistics of the datasets are shown in Appendix B.
Baselines We compare the performance of our method with the following baselines: (i) Generation models need to generate rewritten utterances from scratch, including Seq2Seq model L-Gen (Bahdanau et al., 2015), the hybrid pointer generator network L-Ptr-Gen (See et al., 2017), the basic transformer models T-Gen and T-Ptr-Gen (Vaswani et al., 2017), Syntactic (Kumar and Joshi, 2016), PAC (Pan et al., 2019), L-Ptr-λ and T-Ptr-λ (Su et al., 2019). The above models are limited by the speed of generation. (ii) Structure aware models contain RUN(Liu et al., 2020) and SARG (Huang et al., 2021).
For more information about other experimental setups, please see Appendix B.

heading: 3.2 Main Results
text: Table 2 shows the experimental results on RESTORATION-200K. Our proposed approach, MIUR, achieves competitive results compared to all previous State-of-the-Art methods as shown in Table 2. The results indicate MIUR can effectively mine the semantic information between utterances with two types of MLP architecture. Furthermore, we discovered that MIUR places more emphasis on rewriting precision (Pn) metrics. The first MLP architecture captures global semantic associations between context utterances and incomplete utterance, while the second MLP architecture focuses more on significant word embedding information. Our approach effectively combines two different MLPs and provides an effective guideline for the subsequent construction of the joint feature map matrix, leading our approach to concentrate more on essential word information and to pursue higher rewriting precision. Additionally, we achieve comparable Recalln results to the baselines. The experimental results of REWRITE and CANARD also come to the same conclusion, which can be found in Appendix C.

heading: 3.3 Inference Speed
text: Table 3 presents a comparison of the inferential speed of our model with the baselines. All models were implemented in PyTorch and run on a single NVIDIA V100. We can observe that the proposed MIUR achieves the fastest inference speed compared with the SOTA methods. Specifically, MIUR’s speed is 3.14 times faster than that of L-Gen (n_Beam=1). Moreover, Compared with RUN in the second place, MIUR achieves 20% improvement in the inference speed. This enhanced performance can be attributed to the fact that our model employs only a one-layered MLP backbone to capture inter-utterances semantic information, without utilizing other modules. The simplified architecture, thus, contributes to the model’s faster inference speed without compromising the performance.

heading: 3.4 Ablation Study
text: To verify the effectiveness of MLP architecture in our model, we conduct a thorough ablation study in Table 4. Notably, EM and P2 metrics significantly decreased when the model did not use MLP backbone architecture. The results again prove that MLP backbone can effectively mine latent semantic information between utterances and provide more precise guidance for the follow-up edit type classification. In addition, MIUR uses only one type of MLP architecture alone can also lead to performance degradation. Since the first MLP architecture can effectively mine the semantic associations between context utterances and incomplete utterance, and the second MLP architecture increased focus on capturing attention information between utterances. It’s only with the full MLP structure that MIUR can capture semantic information more accurately and to a wider extent.
As mentioned in Section 2.1, we perform an ablation study about using two different padding strategies to ensure consistent sequence length. Table 5 indicates that the model obtains a small performance improvement using copy mechanism, which further increases the semantic interaction between utterances. But this operation limits inference speed. Given a tiny improvement using copy mechanism, our model employs zero padding method.

heading: 3.5 More Discussion for MLP
text: To further investigate whether our proposed MLP backbone can effectively mine the semantic associations between utterances, we visualize the word embeddings composed of the context utterances and the incomplete utterance in Figure 2. The yaxis represents our selection of 40 words consisting of the context utterances and the incomplete utterance. The x-axis represents the features of the first 100 dimensions of our intercepted word embeddings. It is not difficult to notice that word embeddings appear more distinctly characterized by vertical stripes after MLP backbone. Consequently, this further indicates that semantic information between words is more closely related, and our method can effectively learn the semantic relatedness between words after passing through the MLP network we designed.

heading: 4 Conclusion & Future Work
text: In this paper, we propose a simple yet effective IUR method. We utilize one-layer MLP structure to mine the inter-utterance semantic information from different perspectives. This improves the ability to predict the correct token between incomplete utterance and rewritten utterance. Benefiting from
the fact that our model effectively employs MLP to IUR task, allowing our approach to achieve significant results in terms of performance and inference speed. This study represents the first preliminary exploration of the use of MLP on IUR task. In the future, we will investigate on extending our approach to other dialogue areas.
Limitations
One limitation of current token-pair edit matrix based incomplete utterance rewriting models is that they are only able to select tokens that have appeared in the context utterances. Thus, these models, including our own, are unable to generate new words, such as conjunctions and prepositions, to improve metrics such as fluency. However, this can be addressed by incorporating an additional word dictionary as proposed by Liu et al. (2020) to improve fluency for out-of-vocabulary words (OOV). In addition, we will consider combining generative models (GPT (Radford et al., 2019), T5 (Raffel et al., 2020) etc.) to assist in the recovery of the incomplete utterances in the future works.

heading: Acknowledgement
text: This work was funded by National Natural Science Foundation of China (Grant No. 61762069), Key Technology Research Program of Inner Mongolia Autonomous Region (Grant No. 2021GG0165), Key R&D and Achievement Transformation Program of Inner Mongolia Autonomous Region (Grant No. 2022YFHH0077), The Central Government Fund for Promoting Local Scientific and Technological Development (Grant No. 2022ZY0198), Big Data Lab of Inner Mongolia Discipline Inspection and Supervision Committee (Grant No. 21500- 5206043).

heading: A Constructing Supervised Labels
text: We describe here the algorithm for building wordlevel supervised labels. Taking Table 1 as an example, given U as "那是什么" (What is that) and U ′ as "新冠肺炎是什么" (What is COVID-19). Their longest common subsequence (LCS) is "是 什么" (What is). Hence, "那" (that) is marked as [DEL] in U and "新冠肺炎" (COVID-19) is marked as [ADD] in U ′. Correspondingly, the edit type (supervised label) is Substitute.

heading: B Other Experimental Setups
text: Evaluation Following the previous works, we apply BLEUn (Bn) (Papineni et al., 2002), ROUGEn (Rn) (Lin, 2004), EM (exact match), Rewriting Precisionn, Recalln and F-scoren (Pn,Rn,Fn) (Pan et al., 2019) as the automatic evaluation metrics.
Implementation Details We implement our proposed model via pytorch . All experiments are trained on a single NVIDIA Tesla V100. We use Adam (Kingma and Ba, 2015) optimizer and employ grid search to find the best hyperparameters based on the performance on the validation datasets. The learning rate is set to 1e − 5 for all datasets. The best models are selected by early stopping on the validation datasets, and the max epoch is 100.

heading: C Additional Experimental Results
text: Table 7 and Table 8 show the experimental results on REWRITE and CANARD, respectively. Our
Algorithm 1: Construct Supervised Labels
Input: U : the incomplete utterance U ′: the rewritten utterance
Output: L: the supervised label 1 Computing the LCS between U and U ′.
2 for wx ∈ U do 3 if wx /∈ LCS then 4 mark(wx) = [DEL]
5 end
6 end 7 for w′x ∈ U ′ do 8 if w′x /∈ LCS then 9 mark(w′x) = [ADD]

heading: 10 end
text: 11 end 12 The same mark is combined into one span. 13 Comparing U and U ′ at the span level. 14 for (sx, s′x) ∈ (U,U ′) do 15 if sx = [DEL] and s′x = [ADD] then

heading: 16 L = Substitute

heading: 17 else

heading: 18 L = Insert

heading: 19 end
text: 20 end 21 return L
method also achieves competitive results on all scores. The results again demonstrate the effectiveness of our model.

heading: D Effect of BatchNorm
text: To further explore the validity of BatchNorm for our model, we conducted controlled experiments on REWRITE. As shown in Figure 3, Figure 3(a) indicates the loss of training on REWRITE dataset with BN and without. Figure 3(b) shows the EM metrics of REWRITE validation set with BN and without. We can observe that the incorporation of BatchNorm after the construction of the joint feature matrix leads to faster convergence and enhances the model’s ability to learn global semantic information efficiently.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
5
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 1
7 B1. Did you cite the creators of artifacts you used? While the paper itself does not include explicit citations to the creators of the artifacts used, the corresponding Git code repository’s README.md file mentions the appropriate citations and attributions.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The code repository provided follows the Apache-2.0 license, which governs the terms and conditions for using and distributing the code.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will investigate on extending our approach to other dialogue areas. Limitations
One limitation of current token-pair edit matrix based incomplete utterance rewriting models is that they are only able to select tokens that have appeared in the context utterances. Thus, these models, including our own, are unable to generate new words, such as conjunctions and prepositions, to improve metrics such as fluency. However, this can be addressed by incorporating an additional word dictionary as proposed by Liu et al. (2020) to improve fluency for out-of-vocabulary words (OOV). In addition, we will consider combining generative models (GPT (Radford et al. , 2019), T5 (Raffel et al. , 2020) etc. ) to assist in the recovery of the incomplete utterances in the future works