CFE2: Counterfactual Editing for Search Result Explanation

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-20 23:01:50

Abstract: Search Result Explanation (SeRE) aims to improve search sessions' effectiveness and efficiency by helping users interpret documents' relevance. Existing works mostly focus on factual explanation, i.e. to find/generate supporting evidence about documents' relevance to search queries. However, research in cognitive sciences has shown that human explanations are contrastive i.e. people explain an observed event using some counterfactual events; such explanations reduce cognitive load and provide actionable insights.
Though already proven effective in machine learning and NLP communities, there lacks a strict formulation on how counterfactual explanations should be defined and structured, in the context of web search. In this paper, we first discuss the possible formulation of counterfactual explanations in the IR context. Next, we formulate a suite of desiderata for counterfactual explanation in SeRE task and corresponding automatic metrics. With this desiderata, we propose a method named **C**ounter**F**actual **E**diting for Search Research **E**xplanation (**CFE2**). CFE2 provides pairwise counterfactual explanations for document pairs within a search engine result page. 
Our experiments on five public search datasets demonstrate that CFE2 can significantly outperform baselines in both automatic metrics and human evaluations.

Main paper: 
 introduction: 
 Search systems such as those used within search engines and product search have played a central role in the way people acquire information. Albeit effective, recent advancements in neural retrieval methods [18] provides little human-interpretable evidence of the underlying reasoning they use to determine relevance [41, 42] . Providing explanations on Search Engine Result Page (SERP) to explain documents' relevance has been shown to improve search sessions' efficiency and users' trust towards the system [25, 33] .In line with the general work of explainable AI literature [26, 51] , these explanations are factual in nature. The goal of these factual explanations is to find/generate supporting evidence for model's decisions, i.e. to answer the "Why P" question. Most existing works currently utilize factual explanation for the task of Search Result Explanation (SeRE), i.e. to find/generate supporting evidence to explain documents' relevance w.r.t. the search query. Examples of factual explanations are snippet-based [4, 49] methods and NLGbased methods [32] . However, research in cognitive science has shown that human explanations are contrastive [26] , i.e. people explain an observed event using some counterfactual events/contrast cases. 1 Instead of answering the "Why P" question, counterfactual explanation answers the "Why P rather than Q" question. For example, Alice gets rejected in credit card application and the counterfactual explanation is, were her credit history to be one year longer, the application would have been approved. Such explanations prune the space of causal factors to reduce the users' cognitive load [13, 19] and provide actionable suggestions [51] . Although counterfactual explanations have proven effective in other communities [36, 51] , there lacks a well-established formulation of counterfactual explanation for search result explanation task. Moreover, it is unclear what is the desiderata of such explanations and corresponding evaluation metrics properly evaluate such explanations.Seeing this gap, we aim to investigate the impact of counterfactual explanations for the task of SeRE. Within the scope of this work, we discuss the following research questions: RQ1. Possible formulations of counterfactual explanation problem in the context of web search.In Section 3, we motivate the problem by reviewing counterfactual explainability literature from psychology and cognitive science communities, and discuss the possible formulation of counterfactual explanations in the web search setting. We propose the problem formulation of using counterfactual queries as explanations, as such explanations are directly related to the user activity, thus enabling users to interact with and to interpret the system via potential query reformulation in the rest of the search session.With this formulation, we study pairwise counterfactual explanation, where counterfactual query/explanation is provided as explanation to the (initial query, top-ranked document, lower-ranked counterfactual document) triplet, i.e. were the initial query to be changed to the counterfactual query, the counterfactual document would be ranked higher than the initially higher-ranked document. Providing such explanation not only explains the initial pairwise relevance relation, but also enables actionability where users can interpret the system, and potentially refine the search result with the counterfactual query.Explanations: What are the desired properties of counterfactual explanations for the task of SeRE? How do we evaluate the quality of machine-generated counterfactuals?In Section 4 we formulate the desiderata for counterfactual explanations in the context of web search. With this desiderata, we propose a suite of automatic evaluation metrics to comprehensively evaluate the quality of counterfactual explanations in the IR context. RQ3. Method for Counterfactual Explanations: How do we design an algorithm to provide pairwise counterfactual explanations for document pairs within the current SERP?In Section 5 we propose a model named CounterFactual Editing for Search Result Explanation (CFE2 ), overviewed in Figure 1 . Given (initial query, top-ranked document, lower-ranked counterfactual document), CFE2 edits the initial query into a new query such that the previously lower-ranked document is now ranked higher by the system. This edited counterfactual query serves as a counterfactual explanation to the documents' pairwise relevance relation w.r.t. the initial query. CFE2 outperforms baselines in automatic metrics and human evaluations on public IR datasets (Section 7).The proposed CFE2 has additional strengths: (1) CFE2 works in a black-box/model-agnostic setting, and therefore does not depend on specific retrieval model choices. (2) CFE2 can provide counterfactuals that are of minimal modifications compared to the initial query 𝑞, meeting the definition of counterfactual explainability. (3) CFE2 uses off-the-shelf transformer-based word prediction model as backbone but is also easily extensible by finetuning on the target corpus with minimum extra computational effort. (4) CFE2 is lightweight, meeting the low-latency requirement of SeRE task. 
related work: 
 We mainly introduce two lines of related works and leave a longer discussion to the final version.Explainable AI and search. Existing explainable AI methods can be broadly divided into two categories: model-intrinsic methods, where the decision model is interpretable by design; and modelagnostic methods where explanations are generated from a specific explanation model that is different from the decision model [1, 22] . Model-intrinsic explainable systems often introduce extra model complexity and challenge the search/retrieval systems' low latency design principle [21, 22, 47] . In this work, we opt to study modelagnostic explanation for search/retrieval, where we assume no knowledge w.r.t. architecture and/or parameters of the underlying blackbox retrieval model. In this work, the proposed explanation method CFE2 does not utilize the knowledge from the underlying retrieval model, i.e. architecture and/or parameters, therefore falls into the category of model-agnostic explanations; and the explanation methods that utilize the decision model's knowledge, e.g. gradient attribution methods, are not directly comparable.Existing explainable search works mainly focus on answering questions such as (1) why is this document relevant to the query (pointwise) i.e. to explain document's relevance, (2) why is this document ranked higher than the other (pairwise)? i.e. to explain the ranking. or (3) why is this set of documents returned (listwise)? Works on pointwise explanation broadly falls into three categories: (a) Snippet-based methods provide snippets (sentences or sentence fragments) as explanations [4, 46, 49, 62] and have been provided in mainstream search engine applications (Google, Bing, etc.) [52] . (b) NLG methods propose to generate natural language explanations separately to explain documents' relevance. GenEx [32] utilizes an encoder-decoder structure to generate terse explanations in addition to extractive snippets. (c) Perturbation-based methods perturb the input (query and document) to create saliency maps to explain documents' relevance. Verma and Ganguly [50] and Singh and Anand [41] adapt LIME [34] to generate word-based explanations to explain one single document's relevance w.r.t. the search query. Very few works are centered on pairwise explanations. Singh and Anand [41] propose EXS to explain pairwise ranking preference. To the best of our knowledge, LiEGe [63] is the only work that aims to jointly explain the entire list of retrieved documents.In this work, we also aim to explain a neural ranker's pairwise preference. Instead of generating/editing counterfactual documents like prior works [5, 36, 41] , we focus on providing counterfactual explanations from the query perspective, i.e. Document 𝑑 ′ would be ranked higher than 𝑑 if you modify your query 𝑞 to 𝑞 ′ . Thus, our method can provide user with ways to understand model's reasoning and take actions such as reformulating query to improve search session's effectiveness. As suggested by previous works in web search [2, 14, 27, 37] , it is more relevant to suggest alternate queries that are informative, actionable and can assist users in satisfying their information needs.Factual and counterfactual explanation. Existing factual explanation [43, 51] methods mostly aim to find/generate supporting evidence to explain model predictions. All works discussed were examples of factual explanations. Some representative methods include input feature attribution methods [40, 55] and natural language explanations [58] . From the causal inference perspective, counterfactual explanation doesn't directly answer the "why" part of prediction [51] , but instead answers the what-if question, i.e. what would the model predict if the current input is changed? As discussed by Miller [26] and Lipton [19] , human explanations are contrastive from a cognitive science perspective; people explain an observed event using some counterfactual events. Multiple works from NLP community [13, 36, 60] have discussed the formulation and usage of counterfactual explanations.Contrastive editing. Ross et al. [36] proposed Minimum Contrastive Editing (MiCE) method to explain NLP models. They prepend classification labels to input text and apply a binary search method to find minimum modification to input text to alter the prediction of the NLP classifier. Our method shares similar design choices but is different from the following perspective: (a) our method is designed for ranking task compared to the classification task; (b) MiCE operates on longer text input and uses a span infilling approach for editing while our method operates on shorter queries and uses a different token prediction approach. (c) MiCE uses gradient attribution method to select important tokens to mask, which requires full knowledge w.r.t. the prediction model. In contrast, our method utilizes a separate Masker model M to select important tokens thus operates in a blackbox setting and can serve as a plug-in explanation method to existing search/retrieval models. 
motivation and problem statement: 
 Motivation of counterfactual explanation. We draw inspiration from prior works on counterfactual explanations in the psychology and social science literature [11, 19, 26] . Hilton [11] poses an important insight that one does not explain events per se, but that one explains why the puzzling event occurred in the target cases but not in some counterfactual contrast cases. Denote 𝑃 as fact, and 𝑄 as the counterfactual case, Lipton [19] and Miller [26] argue that the explainer does not need to consider all causes of an event, but should focus on those causes that are relevant to the counterfactual case. For instance, instead of explaining "Why P?", it is more effective to explain "Why P rather than Q?". This approach prunes the space of causal factors to reduce the user's cognitive load. Additionally, it provides useful actionable suggestions, allowing the user to better interpret the system by further interacting with the system to refine the predictions. In the context of web search, counterfactual explanations are useful as the user can easily interact with the system by reformulating queries and refining the search results, eventually improving the effectiveness and efficiency of their information-seeking sessions [24, 39, 56] .Formulation of counterfactual explanation in web search. To the best of our knowledge, there has not been a strict formulation for defining and structuring counterfactual explanations in the context of web search. We consider a vanilla search system setting, where the user issues query 𝑞, and the system returns a list of individual document 𝑑 ∈ D, D denotes the document collection. We denote search system's input as (𝑞, 𝑑) pair, and the system predicts a relevance score rel.(𝑞, 𝑑). The counterfactual event can be of two formulations:(1) The user would have issued a different query, i.e. counterfactual query, and subsequently, the system would have retrieved a different list of documents, i.e. counterfactual documents. (2) A currently top-ranked document, would have been ranked to a lower position by the system, if the document were to be changed in particular ways, i.e. a counterfactual document.In brief, counterfactuality in Formulation ( 1) is from the counterfactual query, while in Formulation (2) it is from the counterfactual document. In machine learning systems, to be tangible to end users, the provided explanations should relate to the user's own activity, and be scrutable, actionable, and concise [3, 9, 10, 48] . With this setup, the user can leverage the explanation to prune the action space, and further interpret/interact with the complex system. Since a search engine user can issue alternative queries but cannot change the document(s) in the collection, we propose that Formulation (1) is more suitable for web search setting. With this formulation, the user can leverage the counterfactual explanation to interpret the black-box web search system, to reformulate queries to fulfill their information needs [7, 39] , potentially improving the effectiveness and efficiency of the search session [24, 56] .Pairwise counterfactual explanation. Given an initial query and a list of documents, we can further provide counterfactual explanations for:(1) The whole list of documents.(2) A pairwise relation between the initially top-ranked document and one counterfactual document that was initially at a lower Perplexity Calculator, computed by the mean perplexity of a sequence; a smaller value indicates the sequence is "fluent" from natural language perspective rank position, but would be ranked at a higher position were the counterfactual query to be issued.These two settings align with the listwise and the pairwise explanation paradigm of explainable search [1] . Since a minor perturbation to the initial query may result in a completely different list of documents being retrieved, which is intractable as document collection size increases, we exclusively study pairwise counterfactual explanations in the scope of this work. Specifically, we study the pairwise relevance relations within the same current search engine result page, e.g. one document at an initially higher rank position (more relevant), i.e. original document, and one document at a lower rank position (less relevant), i.e. counterfactual document.Problem definition of counterfactual explanation for web search. We show a summary of notations in Table 2 This query 𝑞 ′ is counterfactual to the initial query 𝑞, and serves as a counterfactual explanation to the initial triplet (𝑞, 𝑑, 𝑑 ′ ). 2 This counterfactual explanation provides explainability of why 𝑑 is ranked higher than 𝑑 ′ by the system. In addition, it also providing actionability (e.g. query reformulation) where the user can leverage the counterfactual query 𝑞 ′ to refine the ranklist 𝜋 (𝑞), or to retrieve more documents similar to 𝑑 ′ . 
evaluation principles and metrics: 
 Previously in Section 3, we have discussed the possible formulations of counterfactual cases in the context of web search, and formalized the definition of counterfactual explanation for web search. However, there has not been a well-structured framework to evaluate counterfactual explanations in this problem setting. In this section, we first discuss desiderata for counterfactual explanations (Section 4.1), then design corresponding automatic evaluation metrics in Section 4.2.Prior studies [12, 19, 26, 51] have established a comprehensive list of desiderata for explainable systems. In this work, we account for these important factors, and adapt them to the specific context of counterfactual explanations for search systems. An ideal counterfactual explanation method should meet the following desiderata:• Max #Flips: The counterfactual explanation method should be effective in achieving its objective 3 , i.e. it ensures that generated counterfactual query could flip the ranking order of the initially higher-ranked document with the counterfactual document. This effectiveness should hold true for all or most of the random document pairs in the collection. • Closeness: An ideal counterfactual explanation method should generate counterfactual queries that are semantically close to the initial query. This closeness can make the explanations more intelligible and more actionable for end users. • Fluency: Web search queries are often short and concise. The counterfactual query and hence the explanation should have similar fluency to the initial query. Over-complex or poorly phrased explanations are deemed less useful to users [26] . On the other hand, over-simplified counterfactual queries may be incoherent, grammatically incorrect, and may fail the purpose to serve as meaningful explanations [36, 57] . • Low Latency: The counterfactual query generation system should provide explanations in real-time [21, 47] . This ensures the counterfactual queries are interactive and actionable for users.We operationalize the desiderata into automatic metrics to evaluate the effectiveness of any proposed or existing methods:• FlipRate: the proportion of the edited counterfactual queries that satisfy the objective i.e. rel. (𝑞 ′ , 𝑑 ′ ) > rel. (𝑞 ′ , 𝑑). This metric captures the factor Max #Flip. • CosSim: Cosine Similarity computed between the vector representation of 𝑞 and 𝑞 ′ used by search model S. It is bound to [0, 1] where higher CosSim indicates higher similarity (Closeness). • BERTScore-F1: BERTScore [66] evaluates the semantic similarity between two sequences using contextualized representations from a transformer-based model. It is bound to [0, 1] where a higher value indicates higher semantic similarity (optimizes for Closeness); we use BERTScore-F1 to balance precision and recall. • RelFluency: we first use a pre-trained language model GPT-2 [31] to compute the mean perplexity of the generated counterfactual and the initial query, respectively; denote 𝑞 (𝑖 ) as the 𝑖-th token in query 𝑞, and 𝑃 LM (𝑞 (𝑖 ) ) as the language model's probability of predicting token 𝑞 (𝑖 ) given the context of (𝑖 -1) tokens:EQUATIONHere ppl. (•) denotes the mean perplexity of a text sequence. RelFluency=1.0 is an ideal case indicating the counterfactual query is of the same fluency as the initial query [36] ; while an extreme RelFluency score means the counterfactual query deviates from the initial query. 4• Runtime: We also measure the average wallclock time per edit (captures Latency). Additionally, later results from human evaluation (Section 7.2) indicate these metrics are coherent with human understanding. 
the proposed cfe2: 
 Based on desiderata in Section 4.1, we propose an approach to generating counterfactual queries as explanations, named CounterFactual Editing for Search Result Explanation (CFE2 ). CFE2 operates by iteratively editing the initial query by dropping or replacing tokens to obtain the counterfactual query. It comprises three components: Search Model S, Masker Model M and the Editor Model E. A sample editing workflow is as follows:(1) The Search Model S returns a SERP where 𝑑 is top-1 ranked document.(2) The Masker M assigns each token in 𝑞 an importance score.(3) For a counterfactual document 𝑑 ′ ∈ (𝜋 𝑞 \𝑑) (other documents returned by SERP), Editor E will edit the initial query 𝑞 to generate 𝑞 ′ . An overview of our framework and a sample editing workflow is presented in Figure 1 . A detailed algorithm is in Algorithm 1.A key advantage of our counterfactual explanation framework is its model-agnostic nature. It operates independently of the underlying search algorithm, allowing for flexibility in choosing any model. In this paper, we conduct all our experiments using CL-DRD [64] , a state-of-the-art single vector dense retrieval model. For query 𝑞, S assigns each document in D a relevance score and constructs a ranklist 𝜋 (𝑞) accordingly. The SERP for which we provide explanations consists of both ranklist 𝜋 (𝑞) and query 𝑞.Masker model M needs to identify important tokens in 𝑞. Then these tokens can be replaced with tokens that alter the meaning of the query, such that the counterfactual document can be ranked higher over the initially higher-ranked document. To do this, we leverage a method inspired by ColBERT [15] , a dense retrieval model originally designed for neural ad-hoc retrieval.Given a query 𝑞 = 𝑞 (1) 𝑞 (2) . . . 𝑞 (𝑙 ) of 𝑙 tokens and a document 𝑑 = 𝑑 (1) 𝑑 (2) . . . 𝑑 (𝑛) of 𝑛 tokens, we can get a bag of contextualized token embeddings for each token in 𝑞 and 𝑑 from output of BERT [8] , this we denote by v. For each token 𝑞 (𝑖 ) in 𝑞, we estimate its importance score 𝑟 𝑖 by computing as follows:𝑟 𝑖 = max 𝑗 ∈ |𝑛 | v 𝑞 (𝑖 ) • v T 𝑑 ( 𝑗 ) (2)where v 𝑞 (𝑖 ) and v 𝑑 ( 𝑗 ) denote the contextualized token embedding for 𝑖-th token in 𝑞 and 𝑗-th token in 𝑑, respectively; max denotes a max pooling operation over all tokens in 𝑑. Here the pooling end if 17: end while 18: return Null if not 𝑞 ′ else 𝑞 ′ operation catches the most relevant document token to the query token 𝑞 𝑖 ; and different 𝑟 𝑖 indicates the contribution of each query token to the overall ranking score of (𝑞, 𝑑) pair, as computed by rel.(𝑞, 𝑑) = 𝑙 𝑖=1 𝑟 𝑖 . Here we use 𝑟 𝑖 as an approximation of token level importance of the query 𝑞.Given the input triplet (𝑞, 𝑑, 𝑑 ′ ), the task of counterfactual query editing is to edit initial query 𝑞 to a counterfactual query 𝑞 ′ . We formulate counterfactual query editing task as a generation task for a predefined number of 𝑛 tokens. Specifically, we try to find a counterfactual query 𝑞 ′ = 𝑞 ′ (1) , 𝑞 ′ (2) , . . . 𝑞 ′ (𝑙 ) from initial query 𝑞 = 𝑞 (1) , 𝑞 (2) , . . . 𝑞 (𝑙 ) . First, we mask a few important tokens in 𝑞 as measured by 𝑟 𝑖 , which is computed by the masker model. Next, we run word prediction algorithm (also known as Masked Language Modeling task) to predict the masked tokens [MASK] in an autoregressive manner. Since we may have multiple [MASK]s to predict and the candidate space is the whole vocabulary V, it is infeasible to track all possible edits and rate them. Therefore, we leverage Beam Search to prune the search space. In addition, given that we want to ensure Closeness (Section 4.1), we formulate the algorithm as an iterative process, i.e. we start from only one [MASK] token, and gradually increase the number of [MASK]s until we find an edit that achieves the objective of returning document 𝑑 ′ at a higher relevance score than the initially higher ranked document 𝑑. We describe the complete editing algorithm to Algorithm 1. Our method can be divided into three steps:• Step 1: Decoding (line 3-10): We start with replacing top-𝑛 important tokens as identified by M with [MASK] to get the masked query 𝑞 M ; Then, we prepend 𝑞 M to the counterfactual document 𝑑 ′ and use the concatenated sequence as input to Editor model E. E will then run pre-specified transformer model to predict the masked tokens in an autoregressive manner. To be more specific, at the start of timestep 𝑡, the beam B maintains a beam of 𝑏 possible edits 𝑞 E 𝑡 -1 from the previous timestep 𝑡 -1. For each existing 𝑞 E 𝑡 -1 , the probability of token 𝑖 being predicted at current timestep 𝑡 is computed by normalizing the logits over token 𝑖 with a softmax functionEQUATIONwhere the denominator is computed over the whole vocabulary V. Then for each existing possible edit 𝑞 E 𝑡 -1 in B, we add one of the corresponding 𝑏 most probable tokens, resulting in 𝑏 × 𝑏 new possible edits. For these new possible edits, we compute the new probabilitiesEQUATIONWe sort the possible edits at the current timestep by the newly computed probabilities from Equation ( 4 token and perform step 1-2. If there are no edits in B that can flip the results, we mask one more token and keep on repeating step 1-2 until an available edit is found or there are no more tokens from 𝑞 that can be masked.Our editing algorithm's design reflects the desiderata: (1) we start the search for counterfactuals by modifying one token at a time to meet Closeness; (2) we stop the search when a counterfactual can flip the result such that the principle of Max #Flips is satisfied; (3) among the candidates that can flip the result, we choose the counterfactual query with lowest perplexity. This choice ensures fluent counterfactual queries. (4) We use word prediction model as the backbone for Editor E. In our implementation, we use a lightweight DistilBERTForMaskedLM (66M), which has a fully connected layer on top of a DistilBERT for word prediction. This choice meets the Low Latency principle. Notably, it can also be replaced with other word prediction algorithms (e.g. BERTForMaskedLM, etc.). In addition, it is known that large-scale transformer-based language models can benefit from further finetuning on target domain corpus [8, 35] . Thus we also finetune DistilBERTForMaskedLM on the target corpus. To distinguish from the one using off-the-shelf LM, we name our methods as CFE2 and CFE2+ , respectively. 
experiments setup: 
 Datasets. We use the well-established MS MARCO passage ranking dataset [29] together with three information retrieval datasets of different domains from BEIR collection [44] : NQ [16] , FiQA [23] , SciFact [54] and FEVER [45] . A summary of dataset statistics is For each query in our testset, we first use search model S to retrieve top-5 passages from passage collections; then we construct the test triplets (𝑞, 𝑑, 𝑑 ′ ) using top-1 passage as 𝑑 and the rest four passages as 𝑑 ′ , leading to 20.4k, 53.3k, 4.6k, 1.8k and 1.2k test triplets for MS MARCO, FEVER, FiQA, NQ and SciFact respectively. CFE2 does not rely on relevance judgments for training so we do not make use of the train pairs for our method.Compared methods. We conduct experiments with the following methods that are relatively closer to our problem setting:• MaxFlip: we split the counterfactual document 𝑑 ′ into multiple sentences, and select the sentence with the lowest perplexity from the sentences that can flip the pairwise relevance from S. • Summary: we use a Seq2Seq model finetuned on summarization datasets to summarize the counterfactual document as the counterfactual query.• GenEx [32] : GenEx is originally designed to generate terse abstractive explanations given (𝑞, 𝑑) pair and is trained using (𝑞, 𝑑, 𝑒) triplets where 𝑒 is golden reference explanations. We train GenEx with only (𝑞, 𝑑) pairs from trainset as we do not have access to gold references 𝑒 from the dataset. • Mask-only: this is an ablation of the proposed method without the Editing phase, i.e. we only replace the [MASK] token(s) from the masked query with [PAD] iteratively according to the token weights computed by ColBERT model, until the counterfactual query can flip the pairwise relevance from S. • CFE2 : proposed method with off-the-shelf word prediction model DistilBERTForMaskedLM. • CFE2+ : DistilBERTForMaskedLM is finetuned on the corpus of the target dataset. We construct the training corpus using the passages from target collections and train the model with standard masked language model objective.Implementation details. We use t5-base finetuned on XSUM dataset [28] for Summary baseline, with 1e-4 learning rate. We replicate the structure of GenEx and disable the components that does not fit our task, and correspondingly train with the same hyperparameters reported by the original paper. For Editing w/ FT, i.e. CFE2 , we use a standard 0.15 mask ratio, 128 block size, 2e-5 learning rate and train 3 epochs on each dataset's corpus. In the decoding stage of CFE2 , we set beam size 𝑏 to 10 to balance performance and computational complexity. The choice of beam size's effect is studied in Section 7.3 and results reported in Figure 3 . We use the official checkpoint of CL-DRD and ColBERT for the Search model S and Masker Model M. For evaluation, we use gpt2-large [31] to [20] to compute BERTScore-F1, as recommended by prior works [36, 66] . For fair comparison of latency, all of experiments are conducted on a single AWS instance p3.2xlarge with 61 GiB of memory and a single V100 GPU with 16 GiB VRAM. Our code implementation will be made public once this paper gets accepted.Evaluations. We conduct evaluation with the suite of automatic evaluation metrics discussed in Section 4.2. In addition, we include human evaluation, details of which will be elaborated in Section 7.2. We leave qualitative studies in Appendix A. 
results and analysis: 
 We discuss results for automatic evaluation metrics in Section 7.1 and human evaluation results in Section 7.2. We include more ablation studies in Section 7.3.CFE2 can improve Max #Flip. We can observe from Table 4 that on all datasets, CFE2 attains better FlipRate compared to baselines. For example, CFE2 can reach 0.999 FlipRate on FEVER and NQ, and 0.998 FlipRate on MS MARCO compared to the strongest baseline MaxFlip's 0.997, 0.996, 0.986. Interestingly, we find that after finetuning on the target corpus, the performance of CFE2+ actually downgrades by 0.001 on NQ. A potential reason is that finetuning makes the model pick up the specific writing patterns in the target collections and makes it hard to distinguish between hard negatives at the top ranks of SERP, leading to a slight degradation in FlipRate.CFE2 can improve closeness. We study the CosSim and BERTScore-F1 metric from Table 4 . We notice that on all datasets, CFE2 edits achieve significantly higher semantic similarity to the initial queries than the baselines. For example, on FEVER dataset, CFE2 reaches 0.944 CosSim and 0.949 BERTScore-F1 compared to the best baseline GenEx's 0.894, 0.865. We also discover that finetuning degrades similarity. For example, CFE2+ has a downgrade of 0.005 on both CosSim and BERTScore-F1 on FEVER dataset. We hypothesize the reason might be the similarity models, i.e. RoBERTa-base for BERTScore and CL-DRD are trained on a slightly different data distribution compared to FEVER. Given the fact that CFE2 operates by iteratively Masking → Predicting, one can easily see that the generated edits will be of similar length as that of the initial query. Moreover, since we are replacing the masked tokens using algorithms that are trained on the corpus, the meaning and lexical/syntactic structure will also be similar to the initial queries. Therefore we do not include lexical similarity metrics such as BLEU [30] , ROGUE [17] and Lexical Tree Similarity [65] for fair comparison with baselines. However, we do include lexical similarity in our human evaluation (Section 7.2). CFE2 can generate edits of similar fluency to the initial queries.Recall our desiderata that the counterfactual query should be fluent (low perplexity) by itself and should have similar perplexity to the initial query (details in Equation ( 1)). From Table 4 we can observe that CFE2 can generate edits that are close to 1.0 in Fluency. CFE2 finetune reaches 0.972 fluency on FEVER dataset compared to best baseline GenEx's 0.756. CFE2+ consistently improves in terms of Fluency on all datasets. This is expected as finetuning makes word prediction models predict more coherent tokens, leading to the low perplexity of the sequences. Between baselines, we notice that MaxFlip consistently outputs absolutely fluent counterfactuals (low perplexity). This is because it directly selects the lowest perplexity sentence from counterfactual document that can flip the pairwise relevance, leading to low perplexity results. CFE2 are faster in wall clock time, compared to other generationbased methods. Ideally, explanations on SERP should have low computation complexity to pair with the low latency search system, and to also enable interactivity and actionability. We notice on all datasets, Mask-only and MaxFlip consistently achieve lower latency. This is expected as they do not rely on the editing/generation process and require significantly less computing. Compared to the generation-based baselines, i.e. Summary and GenEx, CFE2 achieves significantly faster inference time, which suggests the efficacy of the proposed editing-based approach.We additionally include human evaluation to (1) validate the consistency between the proposed suite of automatic evaluation metrics (Section 4.2) and ( 2) examine the quality of counterfactual queries generated by CFE2 .Human evaluation setup. Annotators are recruited through MTurk; each annotator was given 50 (𝑞, 𝑑, 𝑑 ′ ) triplets randomly sampled from SciFact dataset to simulate challenging web search queries. For each triples, annotators were asked multiple questions comparing the queries generated by the two methods i.e. CFE2+ and closest baseline method GenEx. They were asked to rate each of the queries compared to the initial queries on the following metrics: (a) fluency i.e. query being natural, grammatically correct and likely to be generated by a human [38] , (b) clarity-how easy it is to understand the intent behind the query [37] , (c) closeness-semantic, i.e. how semantically (i.e. in meaning) is the query close to the initial query, and (d) closeness-syntactic, i.e. how syntactically (i.e. in syntax) is the query close to the initial query. We show a sample screen in Figure 2 . For each triplet, we collected at least three qualified responses, and the Fleiss' Kappa agreement rate for four metrics ranges from 0.08 (slight agreement) to 0.22 (fair agreement).Human evaluation results. We report the human evaluation in Table 5 . We can observe that among all four metrics, CFE2 edits significantly outperform edits from GenEx, and the improvement is statistically significant. These results are in alignment with automatic metrics from Table 4 and validate that the proposed suite of automatic metrics (Section 4.2) are indeed a good reflection of the generated counterfactual queries' quality. Effect of beam size. In Figure 3 , we study the sensitivity of the performance due to the change in beam size. We observe that as beam size increases from 5 to 10, the fluency quickly converges to 1. In addition, CosSim, BERTScore-F1 and FlipRate only achieve minor improvement. Yet, the runtime increases as beam size increases, since each word prediction takes 𝑏 ×𝑏 forward passes. Thus, we use 𝑏 = 10 for all our experiments. The figure shows that different beam sizes do not affect end metrics significantly, showing the robustness of the framework to the hyperparameters.Effect of rank positions on CFE2 . In Table 6 we show a comparison of editing performance on different rank positions of the counterfactual documents. For all rank positions, we observe no significant difference in terms of FlipRate. In fact, CFE2 only fails to flip the pairwise relation in 1 out of 1148 instances. In terms of CosSim and BERTScore-F1 , we observe a trend that similarity decreases along ranks, although the difference is minimal. This pattern is anticipated since the original ranking algorithm CL-DRD relies on Maximum Inner Product Search (MIPS) to construct the ranklists, thus the lower ranked documents are less similar to the query, leading to less similar counterfactual queries. We also observe that Fluency slightly degrades along ranks. One possible interpretation is that the lower-ranked documents are less similar to the original query, leading to less similar counterfactual queries in Fluency (higher perplexity compared to the original query). Effect of different word prediction models. One concern that readers may have is that our editor model and search model both use DistilBERT as the backbone, and this may challenge the modelagnostic claim of the proposed methodology. In addition, we also would like to know the effect of different word prediction models on the editing performance. We also experimented with off-theshelf BERTForMaskedLM and report its performance in Table 7 . We can observe that BERT only achieves minor improvement compared to DistilBERT in terms of CosSim, BERTScore-F1 and Fluency, and have the same performance in terms of FlipRate on FiQA dataset; the same pattern holds for the rest three datasets as well.In contrast, we observe CFE2 based on BERT is significantly slower than DistilBERT, e.g. it takes 42.8% more time per edit on FiQA dataset, as bert-base-uncased has 110M parameters compared to distilbert-base-uncased's 66M. Therefore, we conclude that CFE2 with DistilBERT better meets our low latency desiderata. 
conclusion and future work: 
 In this work we study the effectiveness of counterfactual explanations for search engine result explanation task. First, we discuss the possible formulations of counterfactual explanation in the context of web search. Next, we discuss desiderata for counterfactual explanations in search system and operationalize them with appropriate metrics. Ultimately, we propose CFE2 , a framework to automatically generate counterfactual queries by editing the initial queries. The proposed method achieves significant improvement compared to baselines on both proposed evaluation metrics and human evaluation. For future work, we hope to verify the effectiveness of the CFE2 for the end use case of query suggestion/reformulation in online experiments. 
 
 Appendix: Although the definitions of 'Counterfactual' and 'Contrastive' have minor difference in literature[26,36,53], they can be seen as equivalent in our problem formulation of using counterfactual query as explanation, as a counterfactual query will naturally lead to a different search result, i.e. a contrast case.For the rest of the paper, we use counterfactual query and counterfactual explanation interchangeably.also referred to as soundness[19,26]We should note that perplexity metric ppl. (𝑞) in Equation (1a) tends to get lower values for longer sequences because of the denominator |𝑞 |, this is also reflected in our experimental results in Table4, RelFluency<0 may be due to the methods tend to provide long and tedious counterfactual explanations. 


Decision: This is a meta-review. This paper proposes a method for search result explanation based on counterfactual query editing. The reviews are mixed, but three out of the four reviewers would like to see this paper accepted for ICTIR. The main strengths of the paper are the motivation and idea, the good problem definition, and the broad evaluation including a user study. Weaknesses are the low agreement between human judges, the evaluation metrics used (e.g. the MaxFlip metric is directly related to the proposed method), and the utility of the work in a practical search setting. In addition, examples would help understand the intuition of the method and again, the utility of it in practice. Reviewer 7JXf provides a relevant discussion of this issue that I recommend the authors to incorporate in their revision. After discussion with the four reviewers, my standpoint is that this paper has value to the community because: (a) explainability is underexplored in search engines compared to recommender systems; (b) the limitations in terms of practical utility that is pointed out by the reviewers is not a major concern in this stage of the work, especially not for ICTIR; (c) the idea of counterfactual generation is interesting and deserves further discussion and exploration during the conference and in follow-up work.

Reviews: review: main_review: main_review: Why this specific list of desiderata (in Section 4.1) and not more/less/others? Some more motivation would be nice.

Human evaluation focuses on the validity of the automatic measures and the quality of queries. I would have expected also to ask users about their perceived usefulness of such explanations.

Nit: typo in the heading of Section 7.2
main_review: main_review: This is an interesting work that outlines a nice new direction for explaining search results. It convincingly motivates the counterfactual approach taken. The problem definition and proposed direction are the most compelling parts of the paper to this reviewer. While the proposed method performs well in the evaluation, there are several issues that limit how exciting this is (see objective part). The paper itself is easy to follow and I would expect it to be of interest to the community despite its limitations, though it would be much better to acknowledge the evaluation limitations in the paper.
main_review: main_review: This is an important problem, and the work here could be useful in many places. It would be much stronger though if there were some explicit, well-argued connection between what we want an explanation to do and the properties being measured here.

Since there are humans providing data, it would be nice to see a short note about IRB approval (or whatever process applies locally).
main_review: main_review: Here are some small points that I hope the authors will consider, though none of them is a deal-killer:

* In the introduction, the RQ's are not actual questions. It's a little confusing.
* At the start of section 2, what do you mean by "the final version"?!?
* When defining the problem, there should be no comma after problem: "We formally define our problem as follows:"
* In Algorithm 1, $n$ appears to be a count. But in equation two it is listed as $|n|$ which suggests that it is a set or a number that might be negative. 
* In 7.2's header, Evaluation is misspelled as Evaluaiton