Dialogue-RAG: Enhancing Retrieval for LLMs via Node-Linking Utterance Rewriting
Authors: Qiwei Li, Teng Xiao, Zuchao Li, Ping Wang, Mengjia Shen, hai zhao
Accepted at: acl25

Dialogue-RAG: Enhancing Retrieval for LLMs via Node-Linking Utterance Rewriting

Qiwei Li1†, Teng Xiao1†, Zuchao Li1*, Ping Wang2, Mengjia Shen3, Hai Zhao4
1School of Computer Science, Wuhan University, Wuhan, China,
2School of Information Management, Wuhan University, Wuhan, China,
3Wuhan Second Ship Design and Research Institute, Wuhan, China,
4School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
{qw-line, xiaoxiao, zcli-charlie, wangping}@whu.edu.cn,
shennj13@163.com, zhaohai@cs.sjtu.edu.cn

Abstract

Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) methods have demonstrated significant potential on tasks across multiple domains. However, ellipses and coreferences, as common phenomena in dialogue scenes, pose challenges to LLMs’ understanding and RAG’s retrieval accuracy. The previous works ignore the negative impact of this fuzzy data on RAG system. We explore the capabilities of LLMs and RAG systems in dialogue scenarios and use Incomplete Utterance Rewriting (IUR) to complete the key information in dialogue to enhance retrieval. Besides, we propose a lightweight IUR model for query rewriting. It is an end-to-end framework for node linking and iterative inference, incorporating two newly proposed probing semantic features derived from generative pre-training. This framework treats IUR as a series of link decisions on the input sequence and the incrementally constructed rewriting outputs. To test the performance of RAG system in the model multi-round dialogue scenario, we construct an RAG dialogue dataset on English and Chinese, Dialogue-RAG-MULTI-v1.0. Experiment results show that utterance rewriting can effectively improve the retrieval and generation ability of RAG system in dialogue scenes. Experiments on IUR tasks demonstrate the excellent performance of our lightweight IUR method.

1 Introduction

Human-machine interactions are ubiquitous in today’s international web age. With continuous technological advancements, especially in deep learning, machines have gained the ability to understand human speech and language (Dongbo et al., 2023).

Figure 1: Two examples in human-machine dialogue. The question "How about Portuguese?" in Example (a) omits the noun "food". The pronoun "he" in the second question of Example (b) is a reference to "Jon Hippe".

Recently, the emergence of Large Language Models (LLMs) (DeepSeek-AI, 2024; Yang et al., 2024; GLM et al., 2024; Li et al., 2023c; Yao et al., 2024) and Retrieval Augmented Generation (RAG) (Ram et al., 2023; Asai et al., 2023a; Shi et al., 2023; Gao et al., 2023a; Asai et al., 2023b; Cuconasu et al., 2024) technologies has brought new opportunities for the development of human-computer dialogue.

Retrieval Augmented Generation is a technique that enhances text generation by utilizing information from additional data sources. In the context of LLMs, RAG technology has demonstrated outstanding performance in improving generation tasks. On one hand, models can use retrieved information to assist in generation, enhancing the quality and relevance of the generated text. On the other hand, RAG helps models acquire external knowledge and information, significantly expanding application scenarios of large language models. (Gao et al., 2023b; Gupta et al., 2024)

However, in real-world human-machine interaction scenarios, ellipses and coreferences occur quite frequently. Figure 1 illustrates an example of this situation. In dialogue tasks, the ambiguous information can hinder the ability of LLMs to accurately understand the semantics of user queries. In a RAG system, this ambiguity also impacts the retrieval of key information. The inaccurate retrieval results will lead to a decline in the generation per-
formance of LLMs.

To address this limitation, we propose a retrieval enhancement method for RAG, Dialogue-RAG, based on Incomplete Utterance Rewriting (IUR). It completes utterance using IUR, and realizes accurate RAG retrieval and model generation by more complete query. Besides, a novel lightweight IUR model is designed to complete the sentence information to improve the quality of the search results, and then improve the question-answering ability of the generated model. To evaluate the ability of RAG in the dialogue scenario, we propose an RAG dialogue dataset on English and Chinese, which constructs a question answering pair in the scenario of knowledge dialogue and a knowledge base for retrieving related information. This can effectively test the knowledge retrieval and question answering ability of RAG method in dialogue scenario.

Our main contributions are as follows:

• We explore the dialogue task scenario and design a new retrieval enhancement method for RAG based on IUR. We build an enhanced RAG system based on this approach (Dialogue-RAG) and it can achieve accurate retrieval of knowledge base by rewriting and completing incomplete data. Furthermore, it effectively improve the model’s ability to answer fuzzy questions in real scenarios.

• We design a lightweight IUR model that transforms the IUR task into a digraph parsing task and uses a node linking parser and generative pre-trained language model to realize fast digraph parsing decision-making in the IUR tasks.

• We manually construct a retrieval-augmented generation dialogue dataset on English and Chinese, which contains multiple rounds of question answering pairs and a knowledge base for retrieval. It can not only test the dialogue ability of the model, but also evaluate the effect of RAG method on the enhancement of dialogue ability.

2 Related Work

2.1 Retrieval Augmented Generation

Retrieval-Augmented Generation enhances language models by integrating retrieved text passages, significantly improving performance on knowledge-intensive tasks. Initial work by (Guu et al., 2020) and (Lewis et al., 2020) combined dense passage retrieval with sequence-to-sequence models, showing notable performance gains (Ram et al., 2023). Recent advancements include (Luo et al., 2023), who instruction-tuned models with fixed retrieved passages and pre-trained retrievers and models together. (Jiang et al., 2023) proposed adaptive retrieval during generation, and Toolformer (Schick et al., 2024) trained models to generate API calls. However, these approaches face challenges in efficiency, handling irrelevant context, and attribution accuracy (Mallen et al., 2023; Shi et al., 2023; Liu et al., 2023a). Self-RAG (Asai et al., 2023b) addresses these issues by allowing models to use retrieval on demand for diverse queries to some extent. (Cuconasu et al., 2024) studied the impact of several key factors, like the type, number, and position of documents that should augment the prompt to the LLM. However, the existing benchmarks ignore the negative effects of ellipsis and coreference on knowledge retrieval.

2.2 Incomplete Utterance Rewriting

Early works on Incomplete Utterance Rewriting primarily employ sequence-to-sequence models combined with a copy mechanism to extract relevant information from the context (Su et al., 2019; Elgohary et al., 2019; Kumar and Joshi, 2017). These methods use pointer networks or sequence generation models to extract information and address basic rewriting issues. With the introduction of pre-trained models, Incomplete Utterance Rewriting (IUR) research has significantly advanced. For example, RUN (Liu et al., 2020a) proposes a rewriting matrix based on context and utterance embeddings, which determines whether insertion or replacement operations should be carried out by annotating each token. Additionally, SRL (Xu et al., 2020) pre-identifies the subject, predicate, and object within the context and incorporates these as extra features into the encoder, improving performance. RaST (Hao et al., 2021) models the IUR problem as a sequence labeling task, implementing rewriting by predicting actions (insertion, deletion, and None) and the spans of the context separately. HCT (Jin et al., 2022) further enhances the action predictor from previous work to a rule predictor, optimizing the generation of modified words not explicitly present in the context. Recently, (Li et al., 2023b) improve IUR performance by continued pre-training of the T5 (Raffel et al., 2020) model. Meanwhile, (Du et al., 2023) and (Li et al., 2023a)
further develop fine-grained subtasks and refined model architectures based on the edit matrix. Considering that the context of the dialogue contains similar content that tends to cause confusion and the contents of the different themes are numerous and complex, (Guo et al., 2024) design the Dynamic Context Introduction mechanism to filter out irrelevant contexts to handle the extended multi-turn dialogue. XSS (Peng et al., 2024) designs the Cross Scorer Sharing mechanism to support efficient pair locating for IUR, enabling faster and more accurate processing of extended dialogue contexts.

3 Methods

3.1 Dialogue Retrieval Augmented Generation

To effectively deal with the degradation of retrieval quality caused by incomplete utterance, we design a Dialogue Retrieval Augmented Generation (Dialogue-RAG) method. The overall architecture of the method is shown in Figure 2. We rewrite the input question according to the history context and complete the missing parts. In RAG processing, the completed information is used for corresponding information retrieval and question answering. In this way, more accurate information retrieval and high-quality question answers are achieved.

Specifically, suppose the multi-turn dialogue utterances are \( U = \{u^1, u^2, ..., u^n\} \). We use our new lightweight IUR model rewrite utterance \( u^i \) to a self-contained version \( Ru^i \) with context \( U_c = \{u^1, u^2, ..., u^{i-1}\} \), where \( u^i = \{w_1^i, w_2^i, ..., w_m^i\} \) is the \( i \)-th utterance with \( m \) words in dialogue. We define the procedure as follows:

\[
Ru^i = IUR(\{u^1, u^2, ..., u^{i-1}\}, u^i),
\]

where \( IUR(\cdot) \) is the rewriting process. Details are provided in Subsection 3.2 of this chapter. Based on \( Ru^i \), we match key information through similarity. For knowledge base information \( V = \{v_1, v_2, ..., v_l\} \), \( v_i \) represents the chunk vector in the knowledge base. We use cosine similarity to calculate the similarity \( s_{i,j} \) between \( Ru^i \) and \( v_j \):

\[
s_{i,j} = \frac{\mathbf{Ru}^i \cdot \mathbf{v}_j}{\|\mathbf{Ru}^i\| \|\mathbf{v}_j\|}.
\]

We reorder chunks according to the score \( s_{i,j} \) and select the \( Top - k \) chunks as the matching information \( T_m = \{t_1, t_2, ..., t_k\} \) for \( Ru^i \). The \( t_j \in T_m \) represents the chunk text corresponding to the chunk vector \( v_j \). As shown in Figure 2, matched chunk information \( T_m \) and rewritten utterance \( Ru^i \) will be fused into the prompt template as a context prompt \( P(T_m, Ru^i) \) and it will be used as input to the large language model for the dialogue answer \( a_i \):

\[
a_i = \text{LLM}(P(T_m, Ru^i)),
\]

where \( \text{LLM}(\cdot) \) is the large language model.

3.2 Node Linking Iterative Inference

To meet the retrieval requirements of RAG, we design a lightweight IUR model. Following the practice of previous text editing works (Liu et al., 2020a; Zhang et al., 2022), context utterances \( U_c \) are concatenated into a single word sequence \( S_c \) with a special token [SEP] as the utterance separator. In this part, there are three components to our model: a node linking parser, a contextual similarity feature (CSF) extractor, and a rewrite consistency feature (RCF) extractor. The CSF and RCF feature extractors are GPT model-based probes to enhance the interactive inference in IUR.

Digraph Modeling. For IUR, it mainly uses the determined phrase from the context and inserts it into a certain position of the incomplete utterance. We transform it into a digraph modeling task and regard the beginning and ending words of the phrase span in the context and the words on the left of the insertion position in the incomplete utterance as nodes. The edge between two nodes acts as a link. We define two types of links: span link and insertion link. The first is used to determine the span of the phrase to be inserted in the context, while the second is used to determine the insertion position in the incomplete utterance.

It is worth noting that although insertion can solve the vast majority of rewriting problems, there are still some words in the rewritten sentence that don’t come from the context. Since these words do not depend on context, an additional model such as grammatical error correction can be used for post-processing. However, this phenomenon is not the focus of this paper and is therefore not considered in our digraph modeling transformation.

Digraph modeling can effectively simplify the IUR problem. Since for a context of length \( m \), an incomplete utterance of \( n \), the complexity of the insertion operation is \( O(m \times n) \) and the complexity for determining the insertion span is \( O(m \times m) \), so the overall complexity is \( O(m \times m \times n) \). Digraph
modeling decomposes it into two steps of digraph parsing, determining the insertion point has complexity \( O(m \times n) \), and determining the span has complexity \( O(m \times m) \), so the overall implementation complexity is \( O(m \times n) + O(m \times m) \), which greatly reduces inference difficulty. In the actual implementation, the two-step link inference is finished at one time, so the final complexity is reduced to \( O((m + n) \times (m + n)) \).

Node Linking Parser. Based on digraph modeling, we propose to use a node linking parser to learn the digraphs. The overall structure of the parser is shown on the left side of Figure 2. Formally, for an input sequence \( X = \{w_1, w_2, ..., w_n\} \) with length \( n \) (which is a concatenation of context and incomplete utterances), we insert a special token BOS at the beginning of the sequence. We employ a sequence encoder to convert the input sentence into contextualized vector representations \( H \).

In order to distinguish the tokens in the sequence as the head node and the tail node of the link, we introduce two independent feed-forward neural (FFN) layers to project the representations from the sequence encoder to different spaces respectively:

\[
H^{(head)} = \text{FFN}_{\text{head}}(H),
\]
\[
H^{(tail)} = \text{FFN}_{\text{tail}}(H).
\]

To capture the link relationship between the head node and the tail node, we introduce biaffine attention scorer (Dozat, 2016) to score the relationship between the head-tail node pair.

\[
s(i \xrightarrow{l} j) = H_i^{(head)^\top} W H_j^{(tail)} + b,
\]

where \( i \) and \( j \) indicate the index of the head and tail node in the sequence respectively, \( W \) and \( b \) are learnable parameters, and \( l \) is the relation label of link \( i \rightarrow j \). A multi-class cross-entropy loss is applied to the biaffine score \( s(i \xrightarrow{l} j) \) to guide the model to learn node linking parsing.

\[
\mathcal{L} = \sum_{i,j} XEnt(s(i \xrightarrow{l} j), r(i \rightarrow j)),
\]

where \( r(i \rightarrow j) \) indicates the gold relation label between node \( i \) and \( j \), and NONE is used to indicate that there is no relationship between nodes.

Iterative Inference. The foundation of our proposed two probing features is generative language modeling. Its modeling objective for the input language sequence \( X = \{x_1, x_2, ..., x_L\} \) is to predict each token \( x_i \) using the unidirectional visible token sequence \( X_{<i} \) and maximize the following maximum likelihood so that the model can learn the language generation knowledge in an unsupervised way. GPT (Liu et al., 2023b), which is widely used in natural language generation and other fields, is one of many models that have been pre-trained with generative language modeling, thus becoming the
basis for our probing feature. Formally, the probability of distribution for the next token prediction in GPT predicting is given by:

\[
P_G(y) = \text{GPT}(x_0, ..., x_{i-1}).
\] (7)

where \( y \) represents the next predicted token.

a. Contextual Similarity Feature.

In the IUR inference process, assuming incomplete utterance rewriting depends on the phrase \( u_{k:k+\lambda} = \{w_k^u, w_{k+1}^u, ..., w_{k+\lambda}^u\} \) in the context utterance \( u \). \( u_{k:k+\lambda} \) is a complement to incomplete utterance, and the complement insertion position is \( p \). From the perspective of generative language modeling, the generation context of phrase \( u_{k:k+\lambda} \) in original utterance is \( u_{<k} = \{w_1^u, w_2^u, ..., w_{k-1}^u\} \) while the generation context at the insertion position \( p \) incomplete utterance is \( iu_{<p} \). Since two generation context aims at generating the same phrase \( u_{k:k+\lambda} \), we refer to this case as contextual similarity assumption. That is, by the context \( u_{<k} \) and \( iu_{<p} \) has similarity in the distribution of the next token prediction.

Formally, for the generation context \( u_{<k} \) and \( iu_{<p} \), GPT model output the next token distribution respectively as:

\[
P_G(y)^u = \text{GPT}(w_1^u, w_2^u, ..., w_{k-1}^u),
\]
\[
P_G(y)^{iu} = \text{GPT}(w_1^{iu}, w_2^{iu}, ..., w_{p-1}^{iu}).
\] (8)

According to the contextual similarity assumption, the divergence between the two distributions should be smaller if the token from the \( k \)-th position of the utterance \( u \) can be inserted into the position \( p \) of the incomplete utterance \( iu \). In other words, the distribution divergence measures the extent to which a token at a specific position in the utterance can be inserted into the incomplete utterance in IUR. Thus, the contextual similarity feature (CSF) can be written as:

\[
CSF = D_{KL}(P_G(y)^{iu}||P_G(y)^u)
= \sum P_G(y)^{iu} \log \frac{P_G(y)^{iu}}{P_G(y)^u}.
\] (9)

b. Rewrite Consistency Feature.

Apart from context similarity feature, for the typical scenario of incomplete utterance rewriting, inserting the phrase \( u_{k:k+\lambda} = \{w_k^u, w_{k+1}^u, ..., w_{k+\lambda}^u\} \) started with position \( k \) in the context into the position \( p \) of the incomplete utterance should make the new utterance a reasonable sentence. In other words, the partial sequence of the incomplete utterance \( iu_{<p} \) and the inserted word \( u_k \) will form a new valid sentence fragment. We refer to such phenomenon as rewriting consistency.

Based on this phenomenon, we deduce a rewrite consistency feature (RCF) to enhance the determination of completion words and insertion positions in IUR. Since GPT’s pre-training goal is to predict the next token, we can evaluate the probability of this prediction as the likelihood that the token and its context form a valid segment. In IUR, we adopt the probability value of a given token from GPT prediction distribution as the rewrite consistency score, which reflects the probability that the partial sequence \( iu_{<p} \) and \( u_k \) to form a valid fragment. For all words in the context, a vector composed of scores is calculated accordingly to obtain the rewrite consistency feature for insertion position \( p \). Formally, for position \( p \) in the incomplete utterance and utterance context \( u = \{w_1^u, w_2^u, ..., w_m^u\} \), the rewrite consistency feature is written as:

\[
RCF = [\text{GPT}(w_k^u|w_1^{iu}, w_2^{iu}, ..., w_p^{iu})]_{k=1}^m,
\] (10)

where \([.]_{k=1}^m\) represents a list operation from 1 to \( m \) elements.

c. Feature Integration.

Denote that context length is \( m \) and incomplete utterance length is \( n \). In the CSF calculation, any two token positions in the incomplete utterance and context are probed to obtain a divergence score, then the final CSF feature size is \( m \times n \). Similarly, the RCF feature size is also \( m \times n \). Since in node linking parsing, we concatenate context and incomplete utterance into one sequence. Therefore, we extend the feature to \((m+n) \times (m+n)\) by zero padding to facilitate the use of the feature. The feature integration is achieved by adding the padded features to the biaffine score, i.e.,

\[
s(i \xrightarrow{l} j) = H_i^{(head)\top} W H_j^{(tail)}
+ CSF_{i,j} + RCF_{i,j} + b,
\]

where \( b \) is the bias.

3.3 Dialogue RAG Dataset

To test the performance of LLM and RAG methods in real dialogue scenarios, we manually construct an dialogue dataset on English and Chinese, Dialogue-RAG-MULTI-v1.0. The dataset provides multiple rounds of conversations as well as related document. We crawl web pages to get the documents and converted them into serialized text for retrieval knowledge bases. For the test dataset, we
<table>
  <tr>
    <th>Spilt</th>
    <th>Dialogue Rounds</th>
    <th>Q&A Number</th>
    <th>Document Number</th>
  </tr>
  <tr>
    <td rowspan="3">Train</td>
    <td>Total: 23,996<br>English: 12,005<br>Chinese: 11,991</td>
    <td>Total: 226,173<br>English: 124,852<br>Chinese: 101,321</td>
    <td></td>
  </tr>
  <tr>
    <td>Total: 762<br>English: 391<br>Chinese: 371</td>
    <td>Total: 6,310<br>English: 3,180<br>Chinese: 3,130</td>
    <td>Total: 9,321<br>English: 4,889<br>Chinese: 4,432</td>
  </tr>
  <tr>
    <td>Total: 1,036<br>English: 542<br>Chinese: 494</td>
    <td>Total: 8,216<br>English: 4,321<br>Chinese: 3,895</td>
    <td></td>
  </tr>
</table>

Table 1: Dataset setting.

use the document to annotate the conversation data in the form of manual annotations. For the training and development datasets, annotations were generated using DeepSeek-V3’s API, followed by manual review to remove low-quality data. The conversations are constructed based on the web pages. Dialogue-RAG-MULTI-v1.0 provides subjects with a real complex dialogue scene, which contains special concepts, contextual reference, incomplete utterance and other issues that often occur in daily dialogue process.

In practical application scenarios, we can not only directly test the multi-round dialogue effect of the model, but also evaluate the dialogue ability of the model enhanced by the RAG method. The distribution of the dataset is shown in Table 1. According to the statistics of test dataset, the proportion of ellipses and coreferences in multi-round conversations exceeds 60%, which can effectively test RAG system in dialogue scenarios.

4 Experiments

4.1 Setup

Dataset. We conduct knowledge dialogue task on our proposed Dialogue-RAG-MULTI-v1.0 dataset to test the effect of rewriting in RAG augmented human-machine dialogue tasks. To further explore the impact of utterance rewriting on retrieval, we conduct retrieval comparison on the CoQA (Reddy et al., 2019), TOPIOCQA (Adlakha et al., 2022) and OR-QuAC (Qu et al., 2020) dataset. To evaluate the validity of our IUR component, we conduct experiments on English Task-Oriented Dialogue dataset CANARD (Liu et al., 2020b) and Chinese Open Domain Dialogue dataset REWRITE (Su et al., 2019). Same dataset split as in the original paper has been adopted to ensure reproducibility.

Baselines for RAG. In order to verify the effect of our method, we test the knowledge dialogue task in the condition of the generation without IUR and RAG (w/o IUR), the generation without RAG (w/ IUR), the generation without IUR (RAG w/o IUR), the generation of IUR with few shot based on GPT-4o-mini (RAG w/ GPT-4o-mini few shot IUR), and the generation of IUR with our Node Linking Iterative (Dialogue-RAG). In addition, we also compare the performance with other RAG methods, including GLM-RAG (ZhipuAI, 2024), AAR (Yu et al., 2023), Adaptive-RAG (Jeong et al., 2024), IRCot (Trivedi et al., 2022) and DRAGON+ (Liu et al., 2024).

Baselines for IUR For the IUR task, in the setting without BERT pre-trained language model enhancement, the LSTM encoder is used in the baselines Syntactic, L-Gen, L-Ptr, and L-Ptr-Gen, which are consistent with our work. And vanilla attention-based generator (Bahdanau et al., 2015), pointer network generator (Vinyals et al., 2015) and hybrid pointer generator (See et al., 2017) is employed in L-Gen, L-Ptr, and L-Ptr-Gen, respectively. For DuS (Kumar and Joshi, 2016), a semantic sequence and a syntactic sequence model are combined to generate accurate rewrites. And RUN (Liu et al., 2020b) shaped the problem as the prediction of a word-level edit matrix and proposed a semantic segmentation model (UNet) to predict the edit operations. For the comparison with LLMs, we choose the open source models Qwen2-7B-Instruct (Yang et al., 2024) and ChatGLM3-6B (GLM et al., 2024). In addition, we tested DeepSeek-V3 (DeepSeek-AI, 2024) and GPT-4o-mini (OpenAI, 2024). For the REWRITE dataset, our baseline Transformer-as-encoder versions (T-Gen, T-Ptr and T-Ptr-Gen) are used to analyze the performance difference brought by the encoder architecture. L-Ptr-λ and T-Ptr-λ are variants of a pointer network that use λ to control whether to copy tokens from context or from utterance, instead of deciding to copy or generate as in the traditional pointer network.

Evaluation Metrics. We use BLEU, ROUGE, and BERTScore automatic metrics for RAG enhanced dialogue scene. In the retrieval experiment, we use accuracy, recall and MRR@5 to evaluate the retrieval effect. Besides, we use BLEU, ROUGE, and EM automatic metrics for IUR performance evaluation as done in (Pan et al., 2019)

Model Settings. For the knowledge dialogue task, ChatGLM3-6B (GLM et al., 2024)
<table>
  <tr>
    <th>Method</th>
    <th>B<sub>1</sub></th>
    <th>B<sub>2</sub></th>
    <th>B<sub>3</sub></th>
    <th>B<sub>4</sub></th>
    <th>R<sub>1</sub></th>
    <th>R<sub>2</sub></th>
    <th>R<sub>L</sub></th>
    <th>S<sub>p</sub></th>
    <th>S<sub>r</sub></th>
    <th>S<sub>f</sub></th>
  </tr>
  <tr>
    <td>GLM-RAG</td>
    <td>8.00</td>
    <td>5.33</td>
    <td>4.00</td>
    <td>3.06</td>
    <td>16.83</td>
    <td>8.05</td>
    <td>15.73</td>
    <td>55.74</td>
    <td>66.47</td>
    <td>60.16</td>
  </tr>
  <tr>
    <td>AAR</td>
    <td>8.78</td>
    <td>5.88</td>
    <td>4.43</td>
    <td>3.33</td>
    <td>14.35</td>
    <td>7.29</td>
    <td>13.28</td>
    <td>52.07</td>
    <td>60.78</td>
    <td>55.80</td>
  </tr>
  <tr>
    <td>Adaptive-RAG</td>
    <td>7.33</td>
    <td>4.90</td>
    <td>3.67</td>
    <td>2.76</td>
    <td>15.56</td>
    <td>7.05</td>
    <td>14.62</td>
    <td>56.61</td>
    <td>63.01</td>
    <td>58.51</td>
  </tr>
  <tr>
    <td>IRCot</td>
    <td>5.43</td>
    <td>3.58</td>
    <td>2.68</td>
    <td>1.99</td>
    <td>13.04</td>
    <td>5.65</td>
    <td>12.25</td>
    <td>56.39</td>
    <td>58.34</td>
    <td>55.67</td>
  </tr>
  <tr>
    <td>DRAGON+</td>
    <td>11.91</td>
    <td>8.73</td>
    <td>6.88</td>
    <td>5.41</td>
    <td>21.53</td>
    <td>12.38</td>
    <td>20.07</td>
    <td>56.37</td>
    <td>68.4</td>
    <td>61.46</td>
  </tr>
  <tr>
    <td>w/o IUR</td>
    <td>5.84</td>
    <td>3.31</td>
    <td>2.25</td>
    <td>1.58</td>
    <td>10.34</td>
    <td>3.5</td>
    <td>9.55</td>
    <td>52.82</td>
    <td>57.37</td>
    <td>54.83</td>
  </tr>
  <tr>
    <td>w/ IUR</td>
    <td>6.35</td>
    <td>3.53</td>
    <td>2.36</td>
    <td>1.61</td>
    <td>11.13</td>
    <td>3.64</td>
    <td>10.2</td>
    <td>52.82</td>
    <td>59.75</td>
    <td>55.9</td>
  </tr>
  <tr>
    <td>RAG w/o IUR</td>
    <td>10.29</td>
    <td>7.18</td>
    <td>5.46</td>
    <td>4.14</td>
    <td>19.04</td>
    <td>9.82</td>
    <td>17.91</td>
    <td>57.48</td>
    <td>67.26</td>
    <td>61.62</td>
  </tr>
  <tr>
    <td>RAG w/ GPT-4o-mini few shot IUR</td>
    <td>13.03</td>
    <td>9.59</td>
    <td>7.49</td>
    <td>5.79</td>
    <td>25.86</td>
    <td>14.57</td>
    <td>24.41</td>
    <td>59.82</td>
    <td>73.91</td>
    <td>65.81</td>
  </tr>
  <tr>
    <td><b>Dialogue-RAG (ours)</b></td>
    <td><b>13.11</b></td>
    <td><b>9.69</b></td>
    <td><b>7.63</b></td>
    <td><b>5.95</b></td>
    <td><b>26.09</b></td>
    <td><b>14.72</b></td>
    <td><b>24.6</b></td>
    <td><b>60.33</b></td>
    <td><b>74.09</b></td>
    <td><b>66.21</b></td>
  </tr>
</table>

Table 2: Main results. B, R, and S are BLEU, ROUGE, and BERT SCORE respectively. The cumulative n-gram BLEU score is denote as \( B_n \). For ROUGE metric, it measures the n-gram overlapping (denoted as \( R_n \)) and longest matching (denoted as \( R_L \)) between the rewritten utterances and the golden ones. \( S_p, S_r, \) and \( S_f \) represent the precision, recall, and F1 score of BERT SCORE respectively.

![Three bar charts comparing ACC, Recall, and MRR@5 for CoQA, TOPIOCQA, OR-QuAC with and without IUR](page_1012_1012_627_246.png)

Figure 3: Comparison of results with and without rewrite on test sets across three datasets

is used as our inference model. We test Qwen2.5-7B-Instruct (Team, 2024) and Llama3-8B-Chinese-Chat (Wang et al., 2024) as the base model and show their results in Appendix A. We test BiLSTM and Transformer as the IUR module encoder. The encoder setting can be found in Appendix B. For CSF and RCF feature extraction, we use the official GPT2-base. For the IUR experiment of LLM, we set the number of the LLM few shot samples as 3. For the non-LLM model, we tested it on a GeForce RTX 3090 graphics card. For LLM, we deployed models on two GeForce RTX 3090 GPUs for testing.

4.2 Main Results and Analysis

We test the results of different RAG systems on the incomplete utterance dialogue dataset. As shown in Table 2, we can find that the results become better after IUR (w/ IUR) or RAG (RAG w/o IUR) augmentation, which verifies the positive effect of IUR and RAG method in incomplete utterance dialogues. Besides, We can find the method of using IUR to enhance RAG (Dialogue-RAG) achieves the best results in various indicators, which fully confirms the effect of our method.

To demonstrate the retrieval improvement of IUR, we separately compare the accuracy of correctly finding the dialogue document between direct retrieval (w/o IUR) and retrieval (w/ IUR) after rewriting. The results (w/o IUR, 37.80 vs w/ IUR, 77.54) show a surprising improvement in retrieval accuracy after rewriting incomplete sentences. In addition, we score the responses of the RAG system with GPT-4, and the results (w/o IUR, 4.89 vs w/ IUR, 7.70) also showed the gains brought by the IUR method.

In addition, we also test the impact of rewriting results generated using large language models on RAG. The results show that the results generated by our IUR model have better performance than the results generated by the large language model (RAG w/ GPT-4o-mini few shot IUR). This fully validates the excellent performance of our IUR method.

4.3 Retrieval Comparison

To further explore the retrieval effect of utterance rewriting in the dialogue field, we conduct retrieval
comparison on CoQA, TOPIOCQA and OR-QuAC datasets. The results are shown in Figure 3. The results on three datasets show that retrieval after rewriting is significantly better than direct retrieval. This shows that the utterance rewriting of the ambiguity in the dialogue by the IUR method can significantly improve the retrieval effect, which fully verifies our point.

4.4 Evaluation on IUR Task

To evaluate our IUR approach, we conducted comparative experiments on the CANARD and REWRITE benchmarks. Experiment results are presented in Tables 3 and 4. To show the effect of rewriting, we show the case comparison in Appendix C. Besides, We also show the results on other datasets in Appendix D.

For the CANARD dataset, we can find that our model obtains the best results on BLEU and ROUGE, which illustrates digraph modeling, CSF and RCF features are generally effective for IUR. Besides, we test some LLM baselines on the CANARD benchmark to compare our methods. From Table 3, we can see that LLMs with both zero shot and few shot perform worse than our method. It is important to note that our approach not only performs significantly better than large models. As a lightweight model, its computing speed is also much higher than LLM. This is very important for the retrieval efficiency of the RAG system.

For the REWRITE dataset, comparing the results of LSTM-based with Transformer-based baselines, we find that Transformer has an advantage in most systems, but not in all cases (L-Gen vs T-Gen), which may be due to the absence of optimization for Transformer. It suggests that just changing LSTMs with smaller parameters and training-data to Transformers may not necessarily be the best option, we keep using BiLSTM as the encoder when without BERT. On the REWRITE dataset, our full model still exhibits performance advantages with or without BERT. And since the REWRITE dataset is smaller, to increase the stability of the results, we use stricter EM and \( B_2, B_1, R_2, R_L \) metrics. The reported improvements on these stricter metrics demonstrate the generalization ability of our method.

Additionally, we also test some LLM baselines on the CANARD benchmark to compare our methods. From Table 3, we can see that LLMs with both zero shot and few shot perform worse than our method. Besides, as a lightweight model, our IUR component achieves significantly higher computing speeds. This enhances the retrieval efficiency of RAG.

<table>
  <tr>
    <th>Model</th>
    <th>B_1</th>
    <th>B_2</th>
    <th>B_4</th>
    <th>R_1</th>
    <th>R_2</th>
    <th>R_L</th>
  </tr>
  <tr>
    <th colspan="7">Traditional Model</th>
  </tr>
  <tr>
    <td>L-Ptr</td>
    <td>52.4</td>
    <td>46.7</td>
    <td>37.8</td>
    <td>72.7</td>
    <td>54.9</td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>Pronoun Sub<sup>†</sup></td>
    <td>60.4</td>
    <td>55.3</td>
    <td>47.4</td>
    <td>73.1</td>
    <td>63.7</td>
    <td>73.9</td>
  </tr>
  <tr>
    <td>L-Ptr-Gen</td>
    <td>67.2</td>
    <td>60.3</td>
    <td>50.2</td>
    <td>78.9</td>
    <td>62.9</td>
    <td>74.9</td>
  </tr>
  <tr>
    <td>RUN</td>
    <td>70.5</td>
    <td>61.2</td>
    <td>49.1</td>
    <td>79.1</td>
    <td>61.2</td>
    <td>74.7</td>
  </tr>
  <tr>
    <th colspan="7">Large Language Model w/ Zero shot</th>
  </tr>
  <tr>
    <td>Owen2</td>
    <td>26.5</td>
    <td>17.0</td>
    <td>5.5</td>
    <td>50.3</td>
    <td>29.1</td>
    <td>49.2</td>
  </tr>
  <tr>
    <td>ChatGLM</td>
    <td>33.6</td>
    <td>24.8</td>
    <td>12.2</td>
    <td>58.9</td>
    <td>40.8</td>
    <td>54.8</td>
  </tr>
  <tr>
    <td>Deepseek</td>
    <td>27.1</td>
    <td>18.6</td>
    <td>7.5</td>
    <td>50.4</td>
    <td>31.5</td>
    <td>46.8</td>
  </tr>
  <tr>
    <td>GPT-do-mini</td>
    <td>34.6</td>
    <td>25.9</td>
    <td>18.6</td>
    <td>60.2</td>
    <td>42.5</td>
    <td>56.6</td>
  </tr>
  <tr>
    <th colspan="7">Large Language Model w/ Few shot</th>
  </tr>
  <tr>
    <td>Owen2</td>
    <td>34.6</td>
    <td>25.9</td>
    <td>12.6</td>
    <td>60.2</td>
    <td>42.5</td>
    <td>56.6</td>
  </tr>
  <tr>
    <td>ChatGLM</td>
    <td>36.9</td>
    <td>28.1</td>
    <td>14.7</td>
    <td>62.8</td>
    <td>45.0</td>
    <td>59.3</td>
  </tr>
  <tr>
    <td>Deepseek</td>
    <td>42.6</td>
    <td>34.8</td>
    <td>20.9</td>
    <td>70.4</td>
    <td>55.3</td>
    <td>67.8</td>
  </tr>
  <tr>
    <td>GPT-do-mini</td>
    <td>44.4</td>
    <td>36.7</td>
    <td>22.2</td>
    <td>72.8</td>
    <td>58.0</td>
    <td>70.5</td>
  </tr>
  <tr>
    <th>Ours</th>
    <td>71.5</td>
    <td>62.8</td>
    <td>51.3</td>
    <td>81.0</td>
    <td>63.5</td>
    <td>76.0</td>
  </tr>
</table>

Table 3: Experimental results on CANARD. †: Results from Liu et al. (2020b).

<table>
  <tr>
    <th>Model</th>
    <th>EM</th>
    <th>B_2</th>
    <th>B_4</th>
    <th>R_2</th>
    <th>R_L</th>
  </tr>
  <tr>
    <td>L-Gen</td>
    <td>47.3</td>
    <td>81.2</td>
    <td>73.6</td>
    <td>80.9</td>
    <td>86.3</td>
  </tr>
  <tr>
    <td>L-Ptr-Gen</td>
    <td>50.5</td>
    <td>82.9</td>
    <td>75.4</td>
    <td>83.8</td>
    <td>87.8</td>
  </tr>
  <tr>
    <td>L-Ptr</td>
    <td>51.5</td>
    <td>82.7</td>
    <td>75.5</td>
    <td>84.0</td>
    <td>88.2</td>
  </tr>
  <tr>
    <td>L-Ptr-λ<sup>†</sup></td>
    <td>42.3</td>
    <td>82.9</td>
    <td>73.8</td>
    <td>81.1</td>
    <td>84.1</td>
  </tr>
  <tr>
    <td>T-Gen</td>
    <td>35.4</td>
    <td>72.7</td>
    <td>62.5</td>
    <td>74.5</td>
    <td>82.9</td>
  </tr>
  <tr>
    <td>T-Ptr-Gen</td>
    <td>53.1</td>
    <td>84.4</td>
    <td>77.6</td>
    <td>85.0</td>
    <td>89.1</td>
  </tr>
  <tr>
    <td>T-Ptr</td>
    <td>53.0</td>
    <td>83.9</td>
    <td>77.1</td>
    <td>85.1</td>
    <td>88.7</td>
  </tr>
  <tr>
    <td>T-Ptr-λ</td>
    <td>52.6</td>
    <td>85.6</td>
    <td>78.1</td>
    <td>85.0</td>
    <td>89.0</td>
  </tr>
  <tr>
    <td>RUN</td>
    <td>53.8</td>
    <td>86.1</td>
    <td>79.4</td>
    <td>85.1</td>
    <td>89.5</td>
  </tr>
  <tr>
    <th>Ours</th>
    <td>57.6</td>
    <td>86.7</td>
    <td>79.8</td>
    <td>85.3</td>
    <td>90.8</td>
  </tr>
  <tr>
    <td>T-Ptr-λ + BERT</td>
    <td>57.5</td>
    <td>86.5</td>
    <td>79.9</td>
    <td>86.9</td>
    <td>90.5</td>
  </tr>
  <tr>
    <td>RUN + BERT</td>
    <td>66.4</td>
    <td>91.4</td>
    <td>86.2</td>
    <td>90.4</td>
    <td>93.5</td>
  </tr>
  <tr>
    <th>Ours + BERT</th>
    <td>69.4</td>
    <td>91.5</td>
    <td>86.9</td>
    <td>91.7</td>
    <td>94.7</td>
  </tr>
</table>

Table 4: Experimental results on REWRITE. †: Reproduced from the code released by Su et al. (2019).

4.5 Ablation Study

To illustrate the enhancement of CSF and RCF features on digraph modeling, we conduct an ablation study on the REWRITE dataset, and the results are shown in Table 5 and 6. We not only report the performance of IUR on the REWRITE dataset, but we also report the unique metrics of digraph modeling, parsing precision (PP), parsing recall (PR), parsing f-score (PF), and link triplet precision (TP), triplet recall (TR) and triplet f-score (TF). First, when the CSF or RCF feature is removed, the IUR and parsing performance both drop, indicating that these features can effectively help digraph decision-making. Among them, the CSF and RCF
<table>
  <tr>
    <th></th>
    <th>EM</th>
    <th>B_2</th>
    <th>B_1</th>
    <th>R_2</th>
    <th>R_L</th>
  </tr>
  <tr>
    <td>RUN + BERT</td>
    <td>66.4</td>
    <td>91.4</td>
    <td>86.2</td>
    <td>90.4</td>
    <td>93.5</td>
  </tr>
  <tr>
    <th colspan="6">Full Model</th>
  </tr>
  <tr>
    <td></td>
    <td>69.4</td>
    <td>91.5</td>
    <td>86.9</td>
    <td>91.7</td>
    <td>94.7</td>
  </tr>
  <tr>
    <td>w/o CSF</td>
    <td>69.0</td>
    <td>90.9</td>
    <td>86.3</td>
    <td>90.8</td>
    <td>94.1</td>
  </tr>
  <tr>
    <td>w/o RCF</td>
    <td>69.2</td>
    <td>90.8</td>
    <td>86.4</td>
    <td>90.9</td>
    <td>94.3</td>
  </tr>
  <tr>
    <td>w/o CSF, RCF</td>
    <td>68.6</td>
    <td>90.9</td>
    <td>86.3</td>
    <td>90.8</td>
    <td>94.1</td>
  </tr>
</table>

Table 5: IUR performance on REWRITE test set.

<table>
  <tr>
    <th></th>
    <th>PP</th>
    <th>PR</th>
    <th>PF</th>
    <th>TP</th>
    <th>TR</th>
    <th>TF</th>
  </tr>
  <tr>
    <th colspan="7">Full Model</th>
  </tr>
  <tr>
    <td></td>
    <td>82.6</td>
    <td>76.8</td>
    <td>79.6</td>
    <td>76.9</td>
    <td>70.1</td>
    <td>73.3</td>
  </tr>
  <tr>
    <td>w/o CSF</td>
    <td>82.0</td>
    <td>76.8</td>
    <td>79.3</td>
    <td>76.2</td>
    <td>69.3</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>w/o RCF</td>
    <td>81.7</td>
    <td>77.4</td>
    <td>79.5</td>
    <td>75.7</td>
    <td>70.6</td>
    <td>73.1</td>
  </tr>
  <tr>
    <td>w/o CSF, RCF</td>
    <td>81.0</td>
    <td>77.3</td>
    <td>79.1</td>
    <td>74.8</td>
    <td>70.3</td>
    <td>72.5</td>
  </tr>
</table>

Table 6: Parsing performance on REWRITE test set.

![Bar chart comparing rewriting speed between RUN, Ours w/o CSF, RCF, and Ours models](page_1012_120_388_312.png)

Figure 4: Comparison of rewriting speed (sentences per second) on REWRITE test dataset.

features have the greatest impact on the EM metric, while the BLEU and ROUGE metrics are influenced slightly. Second, when CSF and RCF are completely removed, the results of digraph modeling still have a performance advantage compared to RUN, which shows the advantage of digraph modeling that simplifies the complexity without complex model designs. To further verify the advantages of our method, we conduct a case analysis in Appendix E.

4.6 Rewriting Speed

To illustrate the advantages of digraph modeling in IUR, we compare the rewriting speed between the RUN model, our base model, and our full model on the REWRITE test dataset in Figure 4. In order to minimize the influence of irrelevant factors, we run the three systems on the same machine for 5 times and report the average speed. Comparing the speed of RUN with our base model, it shows that modeling the IUR as a digraph structure is very efficient. Further use of our proposed CSF and RCF features reduces this efficiency advantage, but still achieves more than 3 times speedup. In addition, the speed of model w/ BERT is reduced compared with that w/o BERT, indicating that BERT encoding also occupies an important time overhead. In addition, the speed of LLM generation is about 20-30 tokens/s. We test the sentence generation speed of Qwen2-7B and ChatGLM3-6B on the same dataset and the results are 2.72 and 2.73 sentence per second, respectively. LLMs are far slower than our proposed method and not suitable for rewriting in RAG.

5 Conclusion

Ellipses and coreferences in dialogues limit the retrieval capabilities of RAG systems. We propose a RAG method based on IUR enhancement. This method uses the Node Linking IUR model to rewrite queries, enabling better retrieval. In addition, we design a knowledge conversation dataset to evaluate performance of RAG in real dialogue scenarios. The experimental results show that our approach achieves better results when dealing with incomplete discourse in dialogs. In addition, our experimental results on the IUR task show that our IUR model greatly outperforms baseline and related systems, including large language models.

Limitation

In this study, we investigate the effects of utterance rewriting methods on the enhancement of retrieval generation. Our research is centered on enhancing the generation capabilities of LLMs within RAG systems by optimizing retrieval effects. While the incorporation of IUR significantly enhances the quality of both retrieval and generation compared to standard retrieval methods, our exploration of the impact of retrieval results on generation remains incomplete. During the experiment, we find that not all queries positively influence the generation process of large models. In fact, some incorrect queries will mislead the large model generation. Moving forward, we will diligently examine how retrieval results impact models within RAG systems. By considering the necessity of retrieval, the relevance of retrieval outcomes, and other factors, we will further delve into the influence and optimization strategies for retrieval for LLMs and RAG systems.
References

Vaibhav Adlakha, Shehzad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. 2022. TopiOCQA: Open-domain conversational question answering with topic switching. volume 10, pages 468–483.

Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023a. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41–46.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023b. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719–729.

DeepSeek-AI. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954.

Ma Dongbo, Sami Miniaoui, Li Fen, Sara A Althubiti, and Theyab R Alsenani. 2023. Intelligent chatbot interaction system capable for sentimental analysis using hybrid machine learning algorithms. Information Processing & Management, 60(5):103440.

T Dozat. 2016. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734.

Haowei Du, Dinghao Zhang, Chen Li, Yang Li, and Dongyan Zhao. 2023. Multi-granularity information interaction framework for incomplete utterance rewriting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2576–2581.

Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5918–5924.

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023a. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023b. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiada Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuanzao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793.

Xinnan Guo, Qian Zhu, Qihui Shi, Xuan Lin, Liubin Wang, DaqianLi DaqianLi, and Yongrui Chen. 2024. Context-aware tracking and dynamic introduction for incomplete utterance rewriting in extended multi-turn dialogues. In Findings of the Association for Computational Linguistics ACL 2024, pages 2138–2148.

Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 2024. A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions. arXiv preprint arXiv:2410.12837.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Jie Hao, Linfeng Song, Liwei Wang, Kun Xu, Zhaopeng Tu, and Dong Yu. 2021. Rast: Domain-robust dialogue rewriting as sequence tagging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4913–4924.

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong-Cheol Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7036–7050. Association for Computational Linguistics.

Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992.
Lisa Jin, Linfeng Song, Lifeng Jin, Dong Yu, and Daniel Gildea. 2022. Hierarchical context tagging for utterance rewriting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10849–10857.

Vineet Kumar and Sachindra Joshi. 2016. Non-sentential question resolution using sequence to sequence learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2022–2031.

Vineet Kumar and Sachindra Joshi. 2017. Incomplete follow-up question resolution using retrieval based sequence to sequence learning. In Proceedings of the 40th international acm sigir conference on research and development in information retrieval, pages 705–714.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Jiang Li, Xiangdong Su, Xinlan Ma, and Guanglai Gao. 2023a. How well apply simple mlp to incomplete utterance rewriting? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1567–1576.

Zitong Li, Jiawei Li, Haifeng Tang, Kenny Zhu, and Ruolan Yang. 2023b. Incomplete utterance rewriting by a two-phase locate-and-fill regime. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2731–2745.

Zuchao Li, Shitou Zhang, Hai Zhao, Yifei Yang, and Dongjie Yang. 2023c. Batgpt: A bidirectional autoregressive talker from generative pre-trained transformer. arXiv preprint arXiv:2307.00360.

Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023a. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001–7025.

Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020a. Incomplete utterance rewriting as semantic segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2846–2857.

Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020b. Incomplete utterance rewriting as semantic segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2846–2857, Online. Association for Computational Linguistics.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023b. Gpt understands, too. AI Open.

Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Chatqa: Surpassing gpt-4 on conversational qa and rag. Advances in Neural Information Processing Systems, 37:15416–15459.

Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. 2023. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822.

OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/.

Zhufeng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving open-domain dialogue systems via multi-turn incomplete utterance restoration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1824–1833, Hong Kong, China. Association for Computational Linguistics.

Letian Peng, Zuchao Li, and Hai Zhao. 2024. Fast and accurate incomplete utterance rewriting. IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020. Open-Retrieval Conversational Question Answering. In SIGIR.

Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. GECOR: An end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4547–4557, Hong Kong, China. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36.

Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.

Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance rewriter. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 22–31.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2692–2700.

Shenzhi Wang, Yaowei Zheng, Guoyin Wang, Shiji Song, and Gao Huang. 2024. Llama3-8b-chinese-chat (revision 6622a23).

Kun Xu, Haochen Tan, Linfeng Song, Han Wu, Haisong Zhang, Linqi Song, and Dong Yu. 2020. Semantic role labeling guided multi-turn dialogue rewriter. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6632–6639.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Yao Yao, Zuchao Li, and Hai Zhao. 2024. Sirllm: Streaming infinite retentive llm. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2611–2624.

Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023. Augmentation-adapted retriever improves generalization of language models as generic plug-in. In The 61st Annual Meeting Of The Association For Computational Linguistics.

Yong Zhang, Zhitao Li, Jianzong Wang, Ning Cheng, and Jing Xiao. 2022. Self-attention for incomplete utterance rewriting. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8047–8051. IEEE.

ZhipuAI. 2024. Glmrag. Accessed: 2024-9-10.
A  RAG Experiments on Other Base Model

The experiment results on Qwen2.5-7B-Instruct (Team, 2024) and Llama3-8B-Chinese-Chat (Wang et al., 2024) are shown in Table 7. We are surprised to find that the IUR results on Qwen and Llama models are not ideal compared with ChatGLM3-6B (GLM et al., 2024). We check and find that this is due to the problem of rewriting the previous reference. Although this helps the model understand the question better, it can also lead to inconsistent responses, especially when the question answering of our dataset relies more on external data. The rewritten question is also incorrectly answered due to the lack of external knowledge, and the rewriting breaks the original dialogue coherence, so it is not effective. However, it can be found that the RAG system with IUR still has better results, which benefits from the enhancement of retrieval in the RAG process by IUR.

B  Encoder Setting

In BiLSTM sequence encoder setting, we use randomly initialized word embedding with dimension 100, character embedding with dimension 50, and set the character convolutional hidden size to 100. The embedding dropout is set to 0.2. Three BiLSTM layers are stacked with hidden size 600 and dropout 0.33. The hidden size of FFN for head and tail node projection is set to 600. While for the Transformer encoder setting, we fix the parameters of the bottom Transformer layers and only finetune the top 4 layers. We use a mean pooling strategy to deal with the inconsistency between the word-level parsing and subword-level representations. The hidden size of FFN is kept the same as in the BiLSTM encoder setting.

Our models are trained using Adam optimizer for up to 200 epochs, with early stopping based on development dataset performance. The learning rate is set to 1e-3 and 5e-5 for the BiLSTM encoder and Transformer encoder settings respectively. For English, we use the official BERT-large-uncased, and for Chinese, we use BERT-base-chinese.

C  Case Study of Dialogue-RAG

To further explore the enhancement effect of IUR on RAG, we conduct a case study. Figure 5 illustrates a dialogue sample in our Dialogue-RAG-MULTI-v1.0 dataset and Figure 6 shows its related document. Figure 7 and Figure 8, respectively, show the query results and responses enhanced without or with IUR. Through comparison, it can be found that the IUR enhanced question carry more critical information. This enhances the retrieval accuracy, which in turn enhances the RAG system’s response accuracy.

D  Supplementary Dataset Experiments for IUR Task

We conducted additional comparative experiments on English Task-Oriented Dialogue dataset MULTI (Pan et al., 2019) and Chinese Open Domain Dialogue dataset TASK (Quan et al., 2019).

{
    "lang": "en",
    "dialogue": [
        {
            "user": "What is the Athar Jamad Mosque known for?",
            "assistant": "It is one of the oldest and biggest mosques in Coimbatore."
        },
        {
            "user": "Who built the mosque?",
            "assistant": "A perfume merchant named Athar built it."
        },
        {
            "user": "How long did the construction take?",
            "assistant": "It took 44 years to complete."
        },
        {
            "user": "What architectural materials were used?",
            "assistant": "The structure is made of limestone and mortar."
        },
        {
            "user": "What special events attract crowds there?",
            "assistant": "Ramadan and Bakrid attract many visitors."
        },
        {
            "user": "Is there anything significant beside the mosque?",
            "assistant": "Yes, there is a dargah for Pir Jamesha Walliulah nearby."
        }
    ]
}

Figure 5: Sample Dialogue.

{
    "doc_id": "7998",
    "doc_content": "Athar Jamad Mosque in Tamil Nadu, IndiaAthar Jamad Masjid (also known as 'The Big Mosque') is located at Oppanakara Street within the Coimbatore district of Tamil Nadu in India. It is one of the oldest and biggest masjid in Coimbatore. Its history and construction involve a famous merchant named Athar, who migrated from Mecca in 1860. The mosque was constructed over a period of nearly 44 years, beginning in 1860 and finishing in 1904. The structure is made up of limestone and mortar and polished with egg white. The facade is covered with cusped arches surrounded by domes. The mosque has a large courtyard, a small garden, a pond, and ablution pond in the southeastern corner and a small library on the eastern side. There's also a kitchen that prepares 'Yerboor kanji' (rice soup) in the fasting period during Ramadan. However, the mosque entered the list of National Monuments in 1966 under the Indian National Trust for Art and Cultural Heritage (INTACH), the two minars with domed roofs on the northern and southern sides are 85 feet high. This pair of silver domes stand out in the skyline of the city. The mosque is visited by thousands of people, especially the visitors during Friday prayers in Dargah. The mosque is built beside the tomb of Pir Jamesha Walliulah, a Sufi Walliulah who died in the 1850s. His tomb, which is now a dargah in the middle of the mosque, is visited by many people. The mosque is blessed by the visitors inside the dargah with amulets tied around their necks to ward off evil spirits. Following the Jamaat comprises the descendants of the 52 families from Tirunelveli that moved to Coimbatore in 1850. According to Jamaat secretary A.B. Basavaraj, 1355 men and women make up the Jamehsaheb Committee. The committee manages the mosque, as well as Jamesha Walliulah dargah on Big Bazaar Road, Jugal Pir dargah on Trichy road, and the Cemetery Masjid beside Coimbatore Junction. The committee runs three schools in the area which serve 1000 students including Ramadan and Bakrid, crowds flock to the mosque and the dargah beside it. References:ininin"
}

Figure 6: Sample Document.
<table>
  <tr>
    <th>Method</th>
    <th>B_1</th>
    <th>B_2</th>
    <th>B_3</th>
    <th>B_4</th>
    <th>R_1</th>
    <th>R_2</th>
    <th>R_L</th>
    <th>S_p</th>
    <th>S_r</th>
    <th>S_f</th>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5</td>
    <td>w/o IUR</td>
    <td>9.04</td>
    <td>5.54</td>
    <td>3.90</td>
    <td>2.77</td>
    <td>14.97</td>
    <td>5.79</td>
    <td>13.96</td>
    <td>58.43</td>
    <td>63.53</td>
    <td>60.69</td>
  </tr>
  <tr>
    <td>w/ IUR</td>
    <td>9.32</td>
    <td>5.49</td>
    <td>3.72</td>
    <td>2.51</td>
    <td>15.43</td>
    <td>5.69</td>
    <td>14.27</td>
    <td>57.83</td>
    <td>65.86</td>
    <td>61.44</td>
  </tr>
  <tr>
    <td>RAG w/o IUR</td>
    <td>10.61</td>
    <td>7.45</td>
    <td>5.73</td>
    <td>4.39</td>
    <td>20.75</td>
    <td>10.96</td>
    <td>19.54</td>
    <td>59.22</td>
    <td>70.02</td>
    <td>63.96</td>
  </tr>
  <tr>
    <td><b>Dialogue-RAG (ours)</b></td>
    <td><b>14.20</b></td>
    <td><b>10.81</b></td>
    <td><b>8.62</b></td>
    <td><b>6.76</b></td>
    <td><b>29.27</b></td>
    <td><b>17.07</b></td>
    <td><b>27.8</b></td>
    <td><b>62.39</b></td>
    <td><b>77.39</b></td>
    <td><b>68.9</b></td>
  </tr>
  <tr>
    <td rowspan="3">LLaMA3</td>
    <td>w/o IUR</td>
    <td>7.79</td>
    <td>4.71</td>
    <td>3.29</td>
    <td>2.32</td>
    <td>13.11</td>
    <td>5.05</td>
    <td>12.11</td>
    <td>54.7</td>
    <td>61.75</td>
    <td>57.82</td>
  </tr>
  <tr>
    <td>w/ IUR</td>
    <td>7.74</td>
    <td>4.55</td>
    <td>3.08</td>
    <td>2.06</td>
    <td>13.54</td>
    <td>4.95</td>
    <td>12.4</td>
    <td>54.4</td>
    <td>63.77</td>
    <td>58.56</td>
  </tr>
  <tr>
    <td>RAG w/o IUR</td>
    <td>9.76</td>
    <td>6.92</td>
    <td>5.37</td>
    <td>4.14</td>
    <td>19.10</td>
    <td>10.15</td>
    <td>18.07</td>
    <td>56.58</td>
    <td>68.47</td>
    <td>61.69</td>
  </tr>
  <tr>
    <td><b>Dialogue-RAG (ours)</b></td>
    <td><b>13.00</b></td>
    <td><b>9.80</b></td>
    <td><b>7.78</b></td>
    <td><b>6.09</b></td>
    <td><b>26.82</b></td>
    <td><b>15.59</b></td>
    <td><b>25.44</b></td>
    <td><b>60.28</b></td>
    <td><b>76.02</b></td>
    <td><b>66.99</b></td>
  </tr>
</table>

Table 7: RAG experiments on Qwen2.5 and LLaMA3. B, R, and S are BLEU, ROUGE, and BERT SCORE respectively. The cumulative n-gram BLEU score is denote as \( B_n \). For ROUGE metric, it measures the n-gram overlapping (denoted as \( R_n \)) and longest matching (denoted as \( R_L \)) between the rewritten utterances and the golden ones. \( S_p, S_r, \) and \( S_f \) represent the precision, recall, and F1 score of BERT SCORE respectively.

{
  "question": "How long did the construction take?",
  "ground truth": "It took 44 years to complete.",
  "retrieval results": [
    {
      "chunk id": "1894", "doc id": "5462", "content": "The work was suspended several times due to violation of the contract clauses by the customer. As of 2024, the bridge is not completed."
    },
    {
      "chunk id": "2397", "doc id": "6828", "content": "On March 6 & 10, 2011, installation of the 653 ton ea. sector gates was completed."
    },
    {
      "chunk id": "1761", "doc id": "5109", "content": "The first such conditions were met in 1964. Now Sadar Muncipal Assembly in 1970, the Sadar Municipal Assembly decides on the construction of the bridge in December 1973. The architects of the bridge were academics Nikola Hajdin, Gojko Nenadic and Predrag Zelicic.Work began on 22 September 1976, when the first part on the main bridge. Dams were built during the construction. The composition of the experts was focused on three details: The construction of the bars for pillar 17, where there were traces of land slides; the assembly of the steel structure, in pieces, weighing 200 tonnes, without scaffolding, with the help of the Derrick cranes; and the assembly of the steel ropes, which was original."
    }
  ],
  "model response": "The construction of the bridge took approximately 48 months."
}

Figure 7: The case of RAG result without IUR.

{
  "question": "How long did the construction take?",
  "ground truth": "It took 44 years to complete.",
  "retrieval results": [
    {
      "chunk id": "2491", "doc id": "7788", "content": "Athar Jamad Mosque/Mosque in Tamil Nadu, India is a mosque located in the town of Coimbatore, Tamil Nadu, India. It is situated near the Manakkur Street within the Coimbatore district of Tamil Nadu in India. It is one of the oldest and biggest mosque in Coimbatore.inHistory and construction,inA merchant named Athar, who migrated from Thirunelveli, built the mosque in 1356 AD. The mosque was constructed in 1356 AD and was finished in 1904. The structure is made up of limestone and mortar and polished with egg white.",
      "chunk id": "3054", "doc id": "8689", "content": "The analysis of the bricks during the restoration works revealed that the mosque was built using bricks. The walls and the surviving walls were made of brick and stone. Moreover, the surveys show that during the conversion into a mosque in 1356 the arched and the wall of the Mihrab had to be rebuilt. At the same time, the mosque was also renovated and the minaret was added."
    },
    {
      "chunk id": "4877", "doc id": "4877", "content": "Construction of the mosque took four years. It is 10,000 square meters in area. It has two minarets, each 35 meters high, and a dome of aluminium composite. The calligraphy on the rim of the dome was written by Semih Yüksel. The large windows of the mosque have been replaced by glass. The mosque was designed by Tayfur Erdogmus. Decorative motifs are derived from Seljuk art. The large, asymmetrical chandelier has waterlily-shaped glass globes made by Nahide Büyükkamyak. 'Yüksel' means 'to rise'. The mosque should be visited at night, like rain,' and the women's section is designed especially to allow a clear view of the chandeliers."
    }
  ],
  "model response": "The construction of the Athar Jamad Mosque in Coimbatore, Tamil Nadu, India took 44 years, from 1889 to 1934."
}

Figure 8: The case of RAG result with IUR.

D.1 Results on Multi Dataset

In Table 8, we report the IUR performance on the MULTI dataset and list five baselines for comparison: Syntactic, L-Gen, L-Ptr-Gen, PAC, and RUN.

Our method formulates IUR as a digraph parsing problem and proposes CSF and RCf features to strengthen the parsing process, thus achieving better IUR performance than the previous best system, improving on \( F_1, F_2, \) and \( F_3 \) metrics, 2.4, 1.9, 2.0, respectively. For the final generated rewrite utterance, we get an improvement in BLEU and ROUGE metrics, reaching +0.8 \( B_1 \), +0.8 \( B_2 \), +0.5 \( R_1 \), and +1.3 \( R_2 \). On one hand, these improvements are due to digraph modeling and node linking parser, which can seek the insertion span and the insertion position more easily than UNet, due to only single-step digraph derivation is required, rather than enumerating all the words in the span as in RUN. On the other hand, the features derived from our generative PrLM can effectively help remove unreasonable digraph links, reducing the search space for insertion decisions.

In the case of using BERT PrLM, PAC (Pan et al., 2019) proposes a “pick-and-combine” model that demonstrates strong competitiveness. We also achieved good improvements within this setting, with \( F_1+2.1, F_2+2.0, F_3+1.3 \) scores respectively, and 0.8, 1.2, 0.6, 1.2 gains on \( B_1, B_2, R_1, R_2 \) compared to RUN. This further illustrates the advantages of our digraph modeling compared to formalization as semantic segmentation. In addition, the improvements on \( B_1, B_2, R_1, \) and \( R_2 \) show that our method makes the determination of context span and insertion point more accurate.

D.2 Results on Task Dataset

Experiment results on the TASK benchmarks are presented in Table 9. We have additionally introduced two systems for comparison, GECOR and Pronoun Sub, of which GECOR 1 adopts the copy mechanism, while GECOR 2 adopts the gated copy mechanism. For a fair comparison, experiments
<table>
  <tr>
    <th>Model</th>
    <th>\( P_1 \)</th>
    <th>\( R_1 \)</th>
    <th>\( F_1 \)</th>
    <th>\( P_2 \)</th>
    <th>\( R_2 \)</th>
    <th>\( F_2 \)</th>
    <th>\( P_3 \)</th>
    <th>\( R_3 \)</th>
    <th>\( F_3 \)</th>
    <th>B_1</th>
    <th>B_2</th>
    <th>R_1</th>
    <th>R_2</th>
  </tr>
  <tr>
    <td>DuS †</td>
    <td>67.4</td>
    <td>37.2</td>
    <td>47.9</td>
    <td>53.9</td>
    <td>30.3</td>
    <td>38.8</td>
    <td>45.3</td>
    <td>25.3</td>
    <td>32.5</td>
    <td>84.1</td>
    <td>81.2</td>
    <td>89.3</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td>L-Gen †</td>
    <td>65.5</td>
    <td>40.8</td>
    <td>50.3</td>
    <td>52.2</td>
    <td>32.6</td>
    <td>40.1</td>
    <td>43.6</td>
    <td>27.0</td>
    <td>33.4</td>
    <td>84.9</td>
    <td>81.7</td>
    <td>88.8</td>
    <td>80.3</td>
  </tr>
  <tr>
    <td>L-Ptr-Gen †</td>
    <td>66.6</td>
    <td>40.4</td>
    <td>50.3</td>
    <td>54.0</td>
    <td>33.1</td>
    <td>41.1</td>
    <td>45.9</td>
    <td>28.1</td>
    <td>34.9</td>
    <td>84.7</td>
    <td>81.7</td>
    <td>89.0</td>
    <td>80.9</td>
  </tr>
  <tr>
    <td>RUN (Liu et al., 2020b)</td>
    <td>66.9</td>
    <td>54.9</td>
    <td>60.3</td>
    <td>53.0</td>
    <td>43.4</td>
    <td>47.7</td>
    <td>43.8</td>
    <td>35.7</td>
    <td>39.3</td>
    <td>91.1</td>
    <td>88.0</td>
    <td>91.0</td>
    <td>83.3</td>
  </tr>
  <tr>
    <th>Ours</th>
    <th>67.3</th>
    <th>58.7</th>
    <th><b>62.7</b></th>
    <th>53.7</th>
    <th>46.1</th>
    <th><b>49.6</b></th>
    <th>44.9</th>
    <th>38.3</th>
    <th><b>41.3</b></th>
    <th><b>91.9</b></th>
    <th><b>88.8</b></th>
    <th><b>91.5</b></th>
    <th><b>84.6</b></th>
  </tr>
  <tr>
    <td>PAC †</td>
    <td>70.5</td>
    <td>58.1</td>
    <td>63.7</td>
    <td>55.4</td>
    <td>45.1</td>
    <td>49.7</td>
    <td>45.2</td>
    <td>36.6</td>
    <td>40.4</td>
    <td>89.9</td>
    <td>86.3</td>
    <td>91.6</td>
    <td>82.8</td>
  </tr>
  <tr>
    <td>RUN + BERT</td>
    <td>73.2</td>
    <td>64.6</td>
    <td>68.6</td>
    <td>59.5</td>
    <td>53.0</td>
    <td>56.0</td>
    <td>50.7</td>
    <td>45.1</td>
    <td>47.7</td>
    <td>92.3</td>
    <td>89.6</td>
    <td>92.4</td>
    <td>85.1</td>
  </tr>
  <tr>
    <th>Ours + BERT</th>
    <th>74.5</th>
    <th>67.4</th>
    <th><b>70.7</b></th>
    <th>60.8</th>
    <th>55.4</th>
    <th><b>58.0</b></th>
    <th>51.2</th>
    <th>46.4</th>
    <th><b>48.7</b></th>
    <th><b>93.1</b></th>
    <th><b>90.8</b></th>
    <th><b>93.0</b></th>
    <th><b>86.3</b></th>
  </tr>
</table>

Table 8: Experimental results on the MULTI dataset. †: Results from Pan et al. (2019). The bolded results in a column indicate a statistically significant improvement against all the baselines (\( p < 0.05 \)).

<table>
  <tr>
    <th>Model</th>
    <th>EM</th>
    <th>B_4</th>
    <th>F_1</th>
  </tr>
  <tr>
    <td>L-Ptr-Gen †</td>
    <td>50.4</td>
    <td>74.1</td>
    <td>44.1</td>
  </tr>
  <tr>
    <td>GECOR 1 †</td>
    <td>68.5</td>
    <td>83.9</td>
    <td>66.1</td>
  </tr>
  <tr>
    <td>GECOR 2 †</td>
    <td>66.2</td>
    <td>83.0</td>
    <td>66.2</td>
  </tr>
  <tr>
    <td>RUN (Liu et al., 2020b)</td>
    <td>69.2</td>
    <td>85.6</td>
    <td>70.6</td>
  </tr>
  <tr>
    <th>Ours</th>
    <th>71.2</th>
    <th>86.7</th>
    <th>72.8</th>
  </tr>
</table>

Table 9: Experimental results on TASK. †: Results from Quan et al. (2019).

are not augmented with BERT. From the results, our model obtains the best results on EM, BLEU, and ROUGE. In conclusion, on four datasets with different languages and different domains, these improvements illustrate digraph modeling, CSF and RCF features are generally effective for IUR.

E Case Study of Rewrite

Table 10 shows a specific case-based comparison of performance between our model and baseline RUN. To rewrite the final utterance Could you give me the phone number and postcode ?, our model can locate the insertion relationship clearly, and select the appropriate spans one restaurant listing for North American food and the only restaurant serving north American food for action. In contrast, RUN resolves the reference to north American food wrongly from the previous context. Though north American food forms a proper supplementary for the sentence, it is meaningless due to the user asks for restaurant instead of north American food. The possible reason is that the semantic segmentation and region searching algorithms are misled by the inference error and decode a grammatically correct but inappropriate span for insertion. The error from CNNs prediction in RUN is passed to the decoding algorithm, resulting in such inconsistency with the fact, which makes the region wrongly attended and weakens the model performance. Thus, it demonstrates that our model shows better robustness and is free from disturbing factors compared with region-based RUN.

In addition, comparing the model outputs with and without CSF and RCF features, although both our models found the correct reference, i.e. American food restaurant, the one without CSF and RCF features chooses Utterance 4, while w/ CSF,RCF finds Utterance 5. Since CSF and RCF bring constraints from contextual similarity and rewriting consistency, the only restaurant serving American food in Utterance 5 is more consistent with the requirement. This also illustrates the effectiveness of our proposed CSF and RCF for IUR. However, in the final generation from surface tokens, there is an additional only compared to a gold reference, which is a common problem in extractive IUR modeling, but it does not have much effect on the actual role of IUR.
Utterance 1: I am looking for a Danish restaurant .
Utterance 2: There are no danish restaurants listed . may i direct you toward another restaurant ?
Utterance 3: How about American food ?
Utterance 4: There is one restaurant listing for north American food , gourmet burger kitchen in centre part of town .
Utterance 5: Is there any north American food in centre part of town else ?
Utterance 6: That is the only restaurant serving north American food .
Utterance 7: Could you give me the phone number and postcode ?

Gold: Could you give me the phone number and postcode of the restaurant serving north American food?

RUN: Could you give me the phone number and postcode of north American food ?

Ours: Could you give me the phone number and postcode of one restaurant listing for north American food?

Ours w/ CSF;RCF: Could you give me the phone number and postcode of the only restaurant serving north American food?

Table 10: Case study for comparison between our model and baselines.

Reviewer paper summaries: 1. This paper explores the capabilities of LLMs and RAG systems in dialogue scenarios, and uses Incomplete Utterance Rewriting (IUR) to complete the key information in dialogue to enhance retrieval and model generation. 
2. The authors further propose an innovative end-to-end framework for node linking and iterative inference, incorporating two newly proposed probing semantic features derived from generative pre-training. This framework treats IUR as a series of link decisions on the input sequence and the incrementally constructed rewriting outputs. 
3. The authors  manually construct a RAG dialogue dataset in English and Chinese, which contains multiple rounds of question-answering pairs and a knowledge base for retrieval. It can not only test the dialogue ability of the model but also evaluate the effect of the RAG method on the enhancement of dialogue ability.

This paper uses IUR to complete utterances, which are then used as more complete queries for accurate RAG. Additionally, they design a lightweight IUR model that transforms the task into a digraph parsing task and manually construct a retrieval-augmented dialogue dataset in English and Chinese.
Summary of strengths: 1. Motivation is clear, the ellipses and coreferences, as common phenomena in dialogue scenes, bring a challenge to the understanding of LLMs and retrieval accuracy in RAG. However, previous works ignore the negative impact of this fuzzy data on RAG system.

2. The method is sound, including the designed innovative end-to-end framework for node linking and iterative inference. The designed framework treats IUR as a series of link decisions on the input sequence and the incrementally constructed rewriting outputs.  Meanwhile, the constructed RAG dialogue dataset will help the community address the challenges in modern RAG systems.

3. Experiments are comprehensive.

1. The work considers the ellipsis in utterances, making the research motivation meaningful.
    
2. The main experiments demonstrate the effectiveness of the proposed method.
Summary of weaknesses: 1. Limited Models: only the ChatGLM3-6B is used as the inference model.

2. Evaluation Metrics: The authors use BLEU, ROUGE, and BERTScore automatic metrics for the enhanced dialogue scene. Besides, the authors use BLEU, ROUGE, and EM automatic metrics for IUR. Tsing GPT-4 evaluation to measure the dialogue scene and IUR will be interesting.

1. Line 048-054: Lacks reference support.
    
2. The paper states that ellipses and coreferences are considered ambiguous information.
    
    According to Figure 1, ellipses specifically refer to the omission of key terms, which causes ambiguity, but this ambiguity can be resolved through context.
    
    What about coreferences?
    
3. The paper proposes using IUR to complete utterances, making queries clearer for more accurate RAG. However, the motivation and solution don't align, as IUR clarifies queries but doesn’t introduce a new RAG method.
    
4. The paper needs improvement in its writing. The motivation, connection, and logic of the three contributions in the introduction are scattered. (Except for IUR, which aims for complete queries and has a clear research motivation.) This makes it difficult to understand the method.
    
    - The method mentioned in the abstract, "we propose an innovative end-to-end framework for node linking and iterative inference, incorporating two newly proposed probing semantic features derived from generative pre-training," feels abrupt and its connection to the research problem is unclear. This is also not explained in the introduction.
        
5. Line 882: ChatGLM is not a RAG method. Why is it used as the only RAG baseline in Table 2?
    
6. What’s the impact of using IUR to supplement queries? A case study is needed. Also, how large is the performance gap between RAG with and without IUR? A case study is required to better understand the work’s significance.
    
7. Is there any statistic on the proportion and types of incomplete utterances in the dataset?
Comments, suggestions and typos: As shown in Summary Of Weaknesses.

see Weaknesses
Overall assessment scores: 3.5

2.5
Metareview score: 4
Metareview: A kind of query rewriting method for information retrieval, specifically focusing on ellipsis and coreference in conversations. Assessment of the method as a component in a RAG system on a new bilingual dataset of conversations finds that it improves results on several reference based metrics.
Summary of reasons to publish: The new dataset will be useful for future work on dialogue agents.

The proposed method is new and interesting, combining the conversation context with node linking and iterative inference to incrementally complete utterances.

Experiments clearly show strong results with the proposed method backed up by ablation studies.