Interpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embedding (2024)

Abstract: Conversational dense retrieval has shown to be effective in conversational search. However, a major limitation of conversational dense retrieval is their lack of interpretability, hindering intuitive understanding of model behaviors for targeted improvements. This paper presents CONVINV , a simple yet effective approach to shed light on interpretable conversational dense retrieval models. CONVINV transforms opaque conversational session embeddings into explicitly interpretable text while faithfully maintaining their original retrieval performance as much as possible. Such transformation is achieved by training a recently proposed Vec2Text model (Morris et al., 2023) based on the ad-hoc query encoder, leveraging the fact that the session and query embeddings share the same space in existing conversational dense retrieval. To further enhance interpretability, we propose to incorporate external interpretable query rewrites into the transformation process. Extensive evaluations on three conversational search benchmarks demonstrate that CONVINV can yield more interpretable text and faithfully preserve original retrieval performance than baselines. Our work connects opaque session embeddings with transparent query rewriting, paving the way toward trustworthy conversational search. Our code is available at this repository.

heading: 1 Introduction
text: With the rapid development of language modeling, conversational search has emerged as a novel search paradigm and is garnering more and more attention. Different from the traditional ad-hoc search paradigm characterized by keyword-based queries and “ten-blue” links (Yu et al., 2020), conversational search empowers users to interact with the search engine through multi-turn natural language conversations to seek information, which
*Corresponding author.
brings a more intuitive and efficient search experience (Mao et al., 2022b; Gao et al., 2022; Zhu et al., 2023).
In conversational search, the system input is a multi-turn natural language conversation, which may have many linguistic problems such as omissions, co-references, and ambiguities (Radlinski and Craswell, 2017), posing great challenges for accurately grasping the user’s real information needs. Recently, conversational dense retrieval (CDR) (Yu et al., 2021; Lin et al., 2021; Kim and Kim, 2022; Mao et al., 2022a; Qian and Dou, 2022; Mo et al., 2023b; Chen et al., 2024), which directly encodes the whole conversational search session and the passages into a unified embedding space to perform matching, has shown to be a promising method to solve this complex search task. Compared to another type of method: conversational query rewriting (CQR) (Lin et al., 2020; Vakulenko et al., 2021a; Wu et al., 2022; Mo et al., 2023a), which is a two-step method that first reformulates the search session into a decontextualized query rewrite and subsequently inputs this rewrite into existing ad-
2879
hoc search models for search, the end-to-end CDR models can be directly optimized towards better search effectiveness (Yu et al., 2021) and is more efficient as it avoids the extra latency caused by the rewriting step.
However, a notable drawback of conversational dense retrieval is that it inherently lacks interpretability (Mao et al., 2023d). By encoding conversations into dense vector embeddings rather than readable text, it becomes opaque how these CDR models comprehend search intent. The absence of interpretability becomes a severe obstacle for developers to comprehend the reasons behind the search results, hindering effective and targeted enhancements to the bad cases of the models (Mao et al., 2023d,a). Moreover, the absence of interpretability poses challenges in identifying and addressing potential biases or errors within the models, which could lead to unfair or misleading search results without the possibility of timely correction.
In this paper, we present CONVINV : a simple and effective approach aiming to shed light on the opacity problem of conversational dense retrieval. CONVINV demystifies the opaque conversational session embeddings by transforming them into explicitly interpretable text while faithfully maintaining their retrieval performance as much as possible. This transformation allows us to intuitively decipher the characteristics of behaviors of different conversational dense retrieval models.
Figure 1 provides an overview of CONVINV. Specifically, our approach is based on the recently proposed Vec2Text (Morris et al., 2023), which is a powerful method that can invert any text embedding into its original text given the corresponding text encoder. However, inverting the session embedding into the original session is meaningless as it brings no interpretability. We adapt Vec2Text to suit our interpretable inversion of conversational session embedding by taking specific advantage of how the conversational session encoders are trained: the session encoder starts from an ad-hoc query encoder and the passage encoder is frozen during the training. This makes the session and query embeddings finally share the same embedding space for retrieval. Therefore, we propose to train a Vec2Text model based on the ad-hoc query encoder to transform the session embedding so that the transformed text is different from the original session, but also maintains a similar retrieval performance when encoding it with the ad-hoc query encoder. To further enhance the interpretability
of the transformed text, we directly incorporate well-interpretable external query rewrites into the Vec2Text transformation process, effectively guiding it to yield more interpretable text.
We conduct extensive evaluations on three conversational search benchmarks. Compared to baselines, the proposed CONVINV can transform conversational session embeddings into more interpretable text as well as faithfully restore the original retrieval performance of the session embeddings.
In summary, the contributions of our work are: (1) We introduce a simple and effective approach CONVINV to shed light on the interpretability of conversational dense retrieval models by transforming opaque conversational session embeddings into interpretable text as well as faithfully maintain their original retrieval performance.
(2) We propose to incorporate the query rewrites into the transformation process to effectively enhance the interpretability of the transformed text.
(3) Our work connects opaque session embeddings with transparent query rewriting, paving the way toward trustworthy conversational search.

heading: 2 Related Work

heading: 2.1 Conversational Search
text: Currently, conversational search primarily relies on two main methods: conversational query rewriting (CQR) and conversational dense retrieval (CDR). CQR (Yu et al., 2020; Wu et al., 2022; Kumar and Callan, 2020; Voskarides et al., 2020; Lin et al., 2020; Mao et al., 2023b; Liu et al., 2021; Vakulenko et al., 2021a,b; Mao et al., 2023c; Mo et al., 2023a) transforms the whole session into a contextindependent query. The generated query rewrites can directly perform ad-hoc retrieval. In contrast, CDR (Yu et al., 2021; Mo et al., 2024; Krasakis et al., 2022; Mao et al., 2022a; Mo et al., 2023b; Mao et al., 2023d, 2022b; Dai et al., 2022; Hai et al., 2023; Mao et al., 2024) aims to train a session encoder that is capable of encoding the conversational context into a high-dimensional space for conducting dense retrieval. However, the session embedding encoded by the conversational query encoder lacks interpretability, hindering developers from comprehending the retrieval results.

heading: 2.2 Interpretable information retrieval
text: The interpretability issues have increasingly garnered attention within the domain of information retrieval. (Ram et al., 2023) proposed to inter-
pret the session embeddings from dual encoders by mapping them into the lexical space of the model. (Mao et al., 2023d) proposed to augment the SPLADE model by incorporating multi-level denoising approaches, which can produce denoised and interpretable lexical session representations.
To explore the intricate interplay between embedded representations and their textual counterparts, a substantial body of research has focused on the task of inverting embeddings to coherent text. Representing the embedding of sentences as the initial token, (Li et al., 2023) trained a powerful decoder model to decode the entire sequence. (Morris et al., 2023) endeavored to produce text whose embedding closely approximates the given embedding. They achieved this by using the difference between hypothesis embeddings and actual embeddings.

heading: 3 Methodology
text: In this work, we present CONVINV, a new approach designed to demystify conversational session embeddings. Our approach focuses on transforming these opaque conversational session embeddings into explicitly interpretable text while maintaining their retrieval performance as much as possible. CONVINV aims to bridge the gap between the mysterious nature of dense embeddings and the necessity for clear, understandable insights in conversational search intent analysis.

heading: 3.1 Preliminaries

heading: 3.1.1 Conversational dense retrieval
text: Formally, conversational search involves a series of turns {(qi, ai)}ni=1, where the users express their information needs at i-th turn through qi, and the system returns a relevant response ai. This paper focuses on the conversational retrieval task, where the goal of conversational search models is to retrieve relevant passages p for the current query qi, considering its historical context Hi = {(qj , aj)}i−1j=1. The idea of conversational dense retrieval is to jointly map the current query qi along with the historical context Hi and passages into a unified embedding space, and use the similarity between the session embedding and the passage embedding as the retrieval score:
si = Es(qi, Hi), p = Ep(p), (1)
r = cos(si,p), (2)
where Es and Ep are the session and passage encoders, respectively. cos is the cosine similarity
used to compute the retrieval score r.

heading: 3.1.2 Task formulation
text: The encoded conversational session embedding si, while effective, is inherently mysterious and lacks interpretability. Our goal is to transform the session embedding si into an explicit, interpretable text q̂i while faithfully maintaining the original retrieval effectiveness of the session embedding in q̂i.

heading: 3.2 Our Approach
text: To achieve this transformation from session embeddings to interpretable text, we propose a simple yet effective approach, called CONVINV , which is built upon the Vec2Text model (Morris et al., 2023) with tailored adjustments for the interpretation of conversational dense retrieval. Specifically, our approach has two important steps: (1) Training a Vec2Text model based on the ad-hoc query encoder. (2) Enhancing interpretation with rewriting. Figure 2 shows an illustration of our approach.

heading: 3.2.1 Training Vec2Text based on Ad-hoc Query Encoder
text: Vec2Text (Morris et al., 2023) is a recently proposed method for transforming embeddings into text. Given any text encoder E and a large collection of texts T = {ti} where ti is a text, a Vec2Text model ϕ is trained based on a large number of (embedding, text) pairs (i.e., ⟨E(ti), ti⟩) to learn to invert any text embedding E(ti) into a text t ′ i, where E(t ′ i) is very similar to E(ti). As reported in their original paper, cos(E(t ′ i), E(ti)) can reach up to 0.99. Motivated by the remarkable effectiveness of Vec2Text, we adapt it to suit our interpretable inversion of conversational session embedding by leveraging a specific training characteristic of conversational session encoders: Shared Embedding Space for Retrieval. Shared embedding space for retrieval. For the training of conversational dense retrievers, it is common to initialize the conversational session encoder and the passage encoder from a pre-trained ad-hoc retriever, and only fine-tune the session encoder while freezing the passage encoder for facilitating the training (Yu et al., 2021; Lin et al., 2021; Mao et al., 2022a; Mo et al., 2023b). Therefore, we may assume that the session encoder and the adhoc query encoder share the same embedding space for retrieval as they share the same passage encoder. This characteristic is ideal for us to achieve more interpretable session embedding inversion as well
as maintain its original retrieval effectiveness. Interpretable query generation. For a session encoder Es fine-tuned from an ad-hoc query encoder Eq, we train a Vec2Text model ϕq based on Eq but not based on Es. Then, for a session embedding si = Es(qi, Hi), we obtain its transformed text q̂i = ϕq(si) through ϕq. Specifically, Vec2Text includes two models: the inversion model and the correction model, and the generation process of Vec2Text includes two steps: (1) The initial inversion step, where an inversion model first inverts the embedding into an initial inverted text tinv. (2) The correction step, where a correction model then progressively refines this initial inverted text tinv to be more accurate. Figure 2 shows an illustration of the whole generation process of Vec2Text. The detailed introduction of our Vec2Text model training is provided in Appendix A.1.
Since Es and Eq share the same retrieval embedding space, the transformed query embedding Eq(q̂i) is supposed to be highly similar to the original session embedding si and thus keep similar retrieval performance.

heading: 3.2.2 Interpretability Enhancement with Conversational Query Rewriting
text: While the transformed text q̂i can attain retrieval performance comparable to that of the original session embedding si when encoded by the ad-hoc query encoder Eq, there is no assurance that q̂i will form a coherent and interpretable sentence for human understanding.
We propose a simple method to leverage external query rewrites to enhance the interpretability.
Specifically, we first employ a conversational query rewriting model R (for example, the T5QR (Lin et al., 2020) model) to transform the conversational search session {qi, Hi} into a standalone query rewrite q∗i = R(qi, Hi). Then, in the generation process of Vec2Text, we discard the initial inversion process and directly use the query rewrite q∗i as the initial inverted text tinv.
The rewriting model R, trained on a vast dataset of human-crafted rewrites, ensures that the resultant query rewrite is coherent and understandable compared to the original inverted text produced by VecText’s inversion model. The new inverted text q∗i , serving as an improved starting point for the session embedding transformation, can help lead the whole generation process towards a more interpretable direction, and thus enhance the interpretability of the final transformed text q̂i.

heading: 4 Experimental Settings
text: This section presents our basic experimental settings. See Appendix A.2 for full details.

heading: 4.1 Datasets
text: We use four public conversational search datasets: QReCC (Anantha et al., 2021), TREC CAsT19 (Dalton et al., 2020b), TREC CAsT-20 (Dalton et al., 2020a), and TREC CAsT-21 (Dalton et al., 2021). The QReCC dataset consists of 13.6K conversations, with an average of 6 turns per conversation. While the three CAsT datasets (19, 20, 21) only comprise 50, 25, and 26 conversations, respectively, but with more detailed relevance labeling. All four datasets provide human rewrites
for each turn. Following existing works (Mao et al., 2023d; Mo et al., 2023a), we train CDR models on the QReCC dataset and conduct evaluations on the three CAsT datasets.

heading: 4.2 Conversational Dense Retrieval Models
text: Currently, there are mainly two paradigms to train conversational session encoders. The first is proposed by Yu et al. (2021) which employs an ad hoc query encoder as the teacher and learns the student session encoder by mimicking the teacher embeddings originating from human queries. The second is to use the classical ranking loss function (Karpukhin et al., 2020; Lin et al., 2021) to maximize the distance between the session and its positive passages and minimize the distance between the session and negative passages.
Our evaluation is based on both types of CDR models. We name the first type KD-Retriever and the second type Conv-Retriever, where Retriever can be replaced with any base ad-hoc retriever. Specifically, we mainly experiment with a popular ad-hoc retriever, i.e., GTR (Ni et al., 2022), and we investigate the universality of our method to different ad-hoc retrievers in Section 5.3.

heading: 4.3 Baselines
text: Our main goal is to demonstrate the interpretability and preserved retrieval performance of the transformed text generated by our CONVINV , compared to the original session embeddings of KD-GTR and Conv-GTR. To the best of our knowledge, there is no existing method that is completely suitable for our task, i.e., interpreting conversational session embeddings (see the full task definition in Section 3.1.2). Therefore, we propose a straightforward but strong baseline called UniCRR. Figure 3 illustrates UniCRR. Specifically, we unify the ses-
sion encoder and the query rewriter in an encoderdecoder architecture and adopt multi-task learning to simultaneously train both. As such, the rewrite generated from the decoder part can interpret the session embedding generated from the encoder part to some extent.
In addition to the original KD-GTR, ConvGTR, and our proposed UniCRR, we also use the following conversational search baselines mainly for the comparisons of retrieval performance: (1) T5QR (Lin et al., 2020): A conversational query rewriter based on T5 (Raffel et al., 2020), trained using human-generated rewrites. (2) ConvGQR (Mo et al., 2023a): A framework for query reformulation that integrates query rewriting with generative query expansion. (3) LeCoRE (Mao et al., 2023d): A conversational lexical retrieval model extending from the SPLADE model with two well-matched multi-level denoising approaches.

heading: 4.4 Evaluation Metrics
text: Retrieval and inversion evaluation. Following existing works (Mo et al., 2023a; Mao et al., 2023d) and the official settings of the CAsT datasets (Dalton et al., 2020a), we choose MRR, NDCG@3, and Recall@100 to evaluate the retrieval performance. We use two metrics to quantify the fidelity of the embedding inversion: (1) The absolute difference in the retrieval performances between using the session embeddings and the transformed text. (2) Following Vec2Text (Morris et al., 2023), we also calculate the cosine similarity between the session embeddings and the transformed text embeddings. Interpretability evaluation. We conduct a human evaluation for the interpretability of the transformed text from three aspects: (1) Clarity: evaluating the clarity of text expression and identifying the presence of ambiguity or vague expressions; (2) Coherence: examining the logical structure of the text; (3) Completeness: determining the extent to which the text comprehensively covers all historical information. Five information retrieval researchers are employed to assign scores ranging from 1 to 5. A larger score indicates better performance.

heading: 4.5 Implementations
text: For CONVINV , we train Vec2Text models on the large-scale MSMARCO (Nguyen et al., 2016) query and passage collections based on different ad-hoc query encoders. The inversion model is trained for 50 epochs with a batch size of 128 and
the correction model is trained for 100 epochs with a batch size of 200 with 1e-3 learning rate. The maximum sequence length is set to 48. By default, we use the rewrites generated by T5QR to perform rewriting enhancement.
We train the conversational dense retrieval models on the QReCC dataset. The session encoder is initialized from an ad-hoc query encoder and the passage encoder is frozen during training. The input of the session encoder is the concatenation of all historical turns and the current query following existing works (Mao et al., 2023d; Mo et al., 2023a). For KD-Retriever, we follow Yu et al. (2021) using the Mean Squared Error (MSE) loss function to perform knowledge distillation. For Conv-Retriever, we use the contrastive ranking loss function with 48 batch size. The maximum input lengths of the session encoder and the passage encoder are set to 512 and 384, respectively. We generally train 2 epochs with 5e-5 learning rate for CDR models.

heading: 5 Experimental Results

heading: 5.1 Retrieval and Inversion Evaluation
text: Note that our work does not aim to achieve absolutely higher retrieval performance, but rather to faithfully restore the retrieval performance of the original session embeddings, so the main competitor of our CONVINV is only UniCRR. The retrieval performance comparisons on three CAsT datasets are shown in Table 1 and the similarity is shown in Table 3. We find:
(1) Compared to UniCRR, CONVINV achieves superior embedding restoration. For example, for KD-GTR, the average absolute differences
for CONVINV are 0.87 (MRR), 1.5 (NDCG@3), and 1.43 (Recall@100), and the average absolute differences for UniCRR are 9.53 (MRR), 6.3 (NDCG@3), and 9.4 (Recall@100). This indicates that the transformed texts generated by CONVINV are closer to the original session embeddings. This aligns with the restoration similarity, which is shown in Table 3. The superior reconstruction performance of ConvInv compared to UniCRR may stem from the fact that UniCRR fails to establish a direct correlation between session embeddings during both the training and inference phases.
(2) We surprisingly notice that the transformed text generated by CONVINV can sometimes even yield slightly better retrieval performance. For example, on the CAsT-21 dataset, we observe 2.7% NDCG@3 relative gains over the original session embedding, respectively. This discovery could potentially pave the way for enhancing retrieval efficacy and interpretability through the collaborative optimization of CQR and CDR.
Ablation study for rewriting enhancement. We propose using external query rewrites generated by T5QR to improve the interpretability of transformed text, which matches the original session embedding’s retrieval performance but may lack coherence and understandability. Building on this proposition, we compare three types of transformed text to investigate the effect of rewriting enhancement: (1) using T5QR rewrites for the rewriting enhancement, which is the default CONVINV. (2) TX-Human: using human rewrites for the rewriting enhancement. (3) TX-Inversion: not performing rewriting enhancement (i.e., just using the text generated by the inversion model for the correction step). The ablation results of the retrieval performance of transformed text are shown in Table 2. We observe that the utilization of rewriting enhancement brings the retrieval performance closer to the original. Using rewriting enhancement generally leads to stronger overall retrieval performance compared to not.

heading: 5.2 Interpretability Evaluation
text: We manually evaluate the interpretability of three types of transformed text generated by CONVINV. Evaluation results are shown in Figure 4 and we have the following observations:
(1) Using query rewrites as the initial inverted text improves the interpretability of the transformed text of KD-GTR and Conv-GTR across the CAsT19, CAsT-20, and CAsT-21 datasets. This improve-
ment can be attributed to the introduction of the rewrite as the initial inverted text, which essentially offers the corrector model a more informative and clear starting point. These notable enhancements underscore the necessity of our rewritingenhancement approach in improving text interpretability.
(2) For both KD-GTR and Conv-GTR, the human evaluation scores of transformed text on CAsT-19 are higher, whether using rewritingenhancement or not, compared to CAsT-20 and CAsT-21. This observation may be attributed to the absence of response information in the CAsT-19 dataset, which exclusively contains query content. Consequently, the session embedding on CAsT-19 is relatively simple, lacking the complexity introduced by response data.
(3) The lower human evaluation scores of transformed text for Conv-GTR compared to KD-GTR on three datasets may be due to the implications of contrastive learning. This method often introduces additional noise. Therefore, Conv-GTR’s session embedding might be more prone to interference, potentially leading to its less effective performance in generating transformed text.
We provide a concrete example of the transformed texts in Table 4. More case studies are in Appendix A.4. We find that the transformed text CONVINV not only exhibits high interpretability,
fully capturing the user’s query intent about “in Washington D.C.”, but also maintains the closest proximity of retrieval performance to the original session embedding. We notice that it includes an additional clue “(like the Smithsonian Museum)” in the query, which may just be additional knowledge reflected in the mysterious session embedding that can help retrieve passages about famous attractions in Washington D.C.

heading: 5.3 Experiments with Different Retrievers
text: We investigate the universality of our CONVINV by changing the base ad-hoc retriever of the CDR models. Specifically, we experiment with another two popular ad-hoc retrievers: ANCE (Xiong et al., 2021) and BGE (Xiao et al., 2023). Results are shown in Table 5. We find that:
(1) Regardless of the selected ad-hoc retriever, both retrieval similarity and text similarity metrics are observed to be high. To illustrate, on the CAsT19 dataset, the average absolute differences for KDANCE on CAsT-19 dataset are 0.0 (MRR), 0.1 (NDCG@3), and 0.1 (Recall@100), and the cosine similarity is up to 99.9%.
(2) Across both CAsT-19 and CAsT-21 datasets, there is a sustained consistency between similarity scores and human evaluations, indicating that textual similarity is a reliable indicator of quality as perceived by human judges. However, this does not encapsulate all the factors considered in human evaluations, especially as similarity scores remain robust while human evaluations show a decline from CAsT-19 to CAsT-21. Although there is a noted decrease in human evaluation scores across all methods when moving from CAsT-19 to CAsT21, the similarity scores remain high or even show marginal improvement.

heading: 6 Conclusion
text: In this paper, we present a novel approach CONVINV to shed light on the interpretability of conversational dense retrieval. By experimenting with two typical conversational dense retrieval models on three conversational search benchmarks, we demonstrate the effectiveness of our approach in providing interpretable text as well as faithfully restoring the original retrieval performance of ses-
sion embeddings. Our work not only enhances interpretability in conversational dense retrieval but also lays a groundwork for future research toward trustworthy conversational search.
Limitations
Our work provides a simple but effective solution to enhance the interpretability of conversational dense retrieval models, bridging the gap between opaque session embeddings and transparent query rewriting. However, the necessity to train distinct Vec2Text models based on various retrievers demands a significant time investment. Additionally, for session embeddings trained using contrastive learning, the transformed text fails to achieve sufficiently high similarity to the original session embedding, suggesting an incomplete decoding of the session embedding. Besides, some of the transformed texts may not exhibit retrieval performance as effective as the original session embeddings. Some more sophisticated conversational dense retrievers have not been investigated.

heading: Acknowledgments
text: This work was supported by National Key R&D Program of China No. 2022ZD0120103, National Natural Science Foundation of China No. 62272467, the fund for building world-class universities (disciplines) of Renmin University of China, and Public Computing Cloud, Renmin University of China. The work was partially done at the Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE.

heading: A Appendix
text: A.1 Vec2Text
Due to the necessity of transforming session embeddings into explicit and interpretable text, we integrate the Vec2Text model into our architecture. The utilization of Vec2Text (Morris et al., 2023) is driven by its capability to effectively invert the full text represented in dense text embeddings, aligning
with our goal to provide interpretability of session embeddings in conversational dense retrieval.
The Vec2Text model aims for the complete inversion of input text from its embedding; it leverages the difference between a hypothesis embedding and a ground-truth embedding to make discrete adjustments to the text hypothesis. Specifically, the Vec2Text model begins by proposing an initial hypothesis and subsequently refines this hypothesis through iterative corrections. The goal is to progressively bring the hypothesis’s embedding êt closer to the target embedding e. The Vec2Text comprises two models: the inversion model and the corrector model. Firstly, the inversion model endeavors to invert encoder ϕ by learning a distribution of texts given embeddings p (x | e, θ). The training objective for the inversion model is to find θ using maximum likelihood estimation:
θ = argmax θ̂
Ex∼D [p (x | ϕ (x) ; θ)]
On the basis of the simple learned inversion hypothesis x0, the corrector model iteratively refines this hypothesis via marginalizing over intermediate hypotheses:
p ( x(t+1) | e ) = ∑
x(t)
p ( x(t) | e ) p ( x(t+1) | e, x(t), ê(t) )
where ê(t) = ϕ ( x(t) ) .
A.2 More Detailed Experimental Settings A.2.1 Details of Datasets The statistical data for each dataset are presented in Table 6 and a more detailed description is provided as follows:
QReCC is a large dataset designed for the study of conversational search. Every query is accompanied by an answer and a human-generated rewrite. QReCC includes a total of 13,598 dialogues featuring 79,952 queries. Of these, 9.3K conversations originate from QuAC questions; 80 from TREC CAsT; and 4.4K from NQ. Additionally, 9% of the questions within QReCC lack corresponding answers.
CAsT-19, CAsT-20, and CAsT-21 are three widely used conversational search datasets released by TREC Conversational Assistance Track (CAsT). For CAsT-19, relevance assessments are available for 173 queries within 20 test conversations. For CAsT-20, the majority of queries are accompanied by relevance judgments. For CAsT-21, there are
Context: (CAsT-19 Session 79) q1: What is taught in sociology? q2: What is the main contribution of Auguste Comte? q3: What is the role of positivism in it? q4: What is Herbert Spencer known for? q5: How is his work related to Comte? Current Query(35.2): What is the functionalist theory? CONVINV (46.9): what is comte’s functionalist theory in philosophy? TX-Human(46.9): what is comte’s functionalist theory in philosophy? TX-Inversion(20.7): What is the functionalist theory? Human Rewrite(38.3): What is the functionalist theory in sociology?
Table 7: An additional case illustrating the distinction in utilizing rewriting enhancement for transformed text. The numbers in parentheses indicate the retrieval performance NDCG@3 of the transformed text. Notably, the number in parentheses under Current Query represents the retrieval results of the original session embedding, not that of the current query statement.
relevance judgments for 157 queries within 18 test conversations. CAsT-19 and CAsT-20 share the same corpus, whereas CAsT-21 employs a different one.
A.2.2 Implementation Details
During the training process, we conduct the training experiments of the Vec2Text model on four Nvidia A100 40G GPUs. We use bf16 precision and AdamW optimizer with 0.001 as the initial learning rate. The strategy to adjust the learning rate is constant with warm-up. We choose T5 (Raffel et al., 2020) as the backbone model. The number of times to repeat embedding along the T5 input sequence length is set to 16.
During the inference process, the sequence beam width and the invert num steps are set to 10 and 30, respectively. The maximum input length and the maximum response length are set to 512 and 100, respectively. The dense retrieval is performed using Faiss (Johnson et al., 2021).
A.3 Examples of Human Evaluation
Examples of the three metrics for human evaluation are shown in Table 8.
A.4 Supplement of Case Study In this section, We provide an additional case in Table 7 for analysis. The transformed text not only includes the keyword of the original query "functionalist theory", but also enriches it with additional information "comte" and "philosophy", thus yielding a retrieval performance that surpasses that of the human rewrite.
A.5 Experiments with Different Retrievers Investigations of Based on Different Ad-hoc Retrievers on CAsT-19, CAsT-20, and CAsT-21 datasets are shown in Table 9, Table 10 and Table 11, separately.

heading: Future Work
text: Our work not only enhances interpretability in conversational dense retrieval but also lays a groundwork for future research toward trustworthy conversational search. Limitations
Our work provides a simple but effective solution to enhance the interpretability of conversational dense retrieval models, bridging the gap between opaque session embeddings and transparent query rewriting. However, the necessity to train distinct Vec2Text models based on various retrievers demands a significant time investment. Additionally, for session embeddings trained using contrastive learning, the transformed text fails to achieve sufficiently high similarity to the original session embedding, suggesting an incomplete decoding of the session embedding. Besides, some of the transformed texts may not exhibit retrieval performance as effective as the original session embeddings. Some more sophisticated conversational dense retrievers have not been investigated