SlowGuess/NAACL_2022_OCR huggingface dataset:
Diego Ortiz Jose G Moreno Gilles Hubert
Karen Pinel-Sauvagnat Lynda Tamine
Universite Paul Sabatier
IRIT UMR 5505 CNRS
first.last@irit.fr
Recent years have witnessed a growing interest towards learning distributed query representations that are able to capture search intent semantics. Most existing approaches learn query embeddings using relevance supervision making them suited only to document ranking tasks. Besides, they generally consider either user's query reformulations or system's rankings whereas previous findings show that user's query behavior and knowledge change depending on the system's results, intertwine and affect each other during the completion of a search task. In this paper, we explore the value of multi-view learning for generic and unsupervised session-aware query representation learning. First, single-view query embeddings are obtained in separate spaces from query reformulations and document ranking representations using transformers. Then, we investigate the use of linear (CCA) and non linear (UMAP) multi-view learning methods, to align those spaces with the aim of revealing similarity traits in the multi-view shared space. Experimental evaluation is carried out in a query classification and session-based retrieval downstream tasks using respectively the KDD and TREC session datasets. The results show that multiview learning is an effective and controllable approach for unsupervised learning of generic query representations and can reflect search behavior patterns.
Understanding user's search intent is central in information retrieval (IR). Modeling user's intent inevitably requires to capture search context. Search history is arguably the most salient facet of context that has been widely captured and used in previous work (Teevan et al., 2005; Dehghani et al., 2017; Aloteibi and Clark, 2020; Zhou et al., 2020). It mainly includes the following: (1) the previous user's queries, generally recorded into physical sessions (also called time-based sessions (Lucchese
et al., 2011)) or task-based sessions (also called missions (Hagen et al., 2013)); (2) the retrieved documents that the user subsequently selects (e.g., based on clicks), among those retrieved by the IR system in response to her queries. Mining user's search intent from search history is challenging because of phenomena such as vocabulary mismatch between the query and documents, ambiguity issues since two queries even with slight lexical variations may underline different intents (Steiner, 2019; Sanderson, 2008), and topic change in user's search behavior which is particularly prominent while completing complex search tasks (e.g., exploratory and multi-step tasks (Hassan Awadallah et al., 2014; He and Yilmaz, 2017). To address these challenges, recent years have witnessed a growing interest in learning query representations to capture hidden syntactic and semantic relationships (Zamani and Croft, 2016; Grbovic et al., 2016; Bing et al., 2018; Zhang et al., 2019; Zhou et al., 2020). However, learning context-aware query embeddings faces two main issues: (1) user's query formulations included in the search sessions bring word contexts that do not extensively occur at the training phase in web search data (Keller and Lapata, 2003); (2) queries do not exhibit a clear structure as sentences. In most of previous work, query embeddings are learned based on search session contexts modeled from relevant or pseudorelevant documents returned by the system (Zamani and Croft, 2016, 2017; Zhang et al., 2019). These methods are suited to supervised relevance ranking tasks with sufficient training data. Other methods learn distributed query representations based on user's query reformulations in the search session (Grbovic et al., 2016; Sen et al., 2018; Zhou et al., 2020). These methods are rather unsupervised and applicable to a wide range of downstream language processing tasks making them generic.
In this work, we explore the unsupervised problem of learning generic distributed query representa
Findings of the Association for Computational Linguistics: NAACL 2022, pages 304 - 315
tions, able to support a wide range of downstream search tasks. As outlined recently, unsupervised representation learning for IR has not received much attention yet (Lin, 2021). This paper attempts to fill this gap by following a query oriented fashion. Specifically, we argue that by considering only one facet of the search session (i.e., documents vs. query reformulations) as done in Sen et al. (2018); Grbovic et al. (2016); Zamani and Croft (2016); Zhang et al. (2019); Zhou et al. (2020), or by considering them both but without relating the semantics underlying between the user's search intent and the system's document results (Bing et al., 2018), we lose valuable mutual information about the interactive intentions (Xie, 2002) that could act as a soft supervision during the search task. Based on previous findings (Eickhoff et al., 2014; Liu et al., 2019a) showing how user's query behavior and knowledge change from system's results during the search session, we propose a framework for Session-aware Query rRepresentation learning based on multi-View Learning (SaQuEvI L).
SaQuEViL is a two-step architecture that consists of two single-view query encoders, namely user-view and system-view query encoders, and a multi-view query encoder. Each single-view query encoder is based on a bidirectional transformer (Vaswani et al., 2017) at the session level. By investigating the use of unsupervised multi-view based learning algorithms, namely Cross-modal Factor Analysis (CFA) and Uniform Manifold Approximation and Projection (UMAP), the multi-view encoder takes as input the two single-view query embeddings related to the same query and provides a multi-view query representation. The underlying objective functions aim to maximize the alignment of features between both views which leans to reveal the underlying manifold. In the multi-view embedding space, similar queries formulated in the context of similar tasks have spatially close representations.
Our key contributions are: 1) we model generic session-aware query representation as an unsupervised multi-view learning task using a two-step framework architecture, SaQuEViL; 2) we experimentally show the effectiveness of multi-view based representations in query classification and session-retrieval as downstream tasks; 3) we conduct quantitative and qualitative analyses showing the potential of SaQuEViL in understanding user's search behavior.
2.1 Distributed query representation
A common problem in IR is that queries –the pivotal parts of a retrieval process– are under-specified which is prone to the vocabulary mismatch and thereby, the poor performance of search-related tasks. Recently, much attention has been paid to learning distributed query representations. Previous work following this approach can be organized based on the facet of query context and type of supervision used to learn the distributed representations. In the first line of work, both query context and supervision include user's relevance signals on documents (Zamani and Croft, 2016, 2017; Zhang et al., 2019). The underlying assumption is that the more queries share the same relevant or pseudorelevant documents among those selected by the retrieval system, the more they have semantically close intent leading to similar embeddings in the latent representation space. Using a probabilistic framework, Zamani and Croft (2016) propose to learn relevance-based query representations based on the embeddings of the query words. Then, the closeness between the probability distribution of the query representation, based on similarity metrics of word embeddings, and the query language model is maximised. Zhang et al. (2019) propose the GEN Encoder which learns distributed representations of queries in two stages. The first stage captures user's intent based on document clicks by using the assumption that queries with similar clicks underline similar intent. The second stage denoises the representations and enhances their generalizability by leveraging human paraphrase labeling in a multi-task learning setting. The second line of work relies on query context held by the search history through query reformulations recorded into physical sessions (Grbovic et al., 2016) or task-based sessions (Mehrotra and Yilmaz, 2017; Sen et al., 2018). Query embeddings are learned based on the assumption that lexically similar queries formulated in similar search sessions across users are semantically related leading to close representations in the embedding space. Mehrotra and Yilmaz (2017) propose task-aware query embeddings by applying the skip-gram model on sequences of queries belonging to the same task-based session. These query representations learned in an unsupervised manner are expected to be generic, thought their evaluation has been limited to specific downstream tasks such as query expansion in sponsored
search (Grbovic et al., 2016) and search task extraction (Sen et al., 2018). A recent line of work uses context built up on query reformulation in a session and documents (Bing et al., 2018; Zhou et al., 2020). For instance, Bing et al. (2018) model a unified graph information where vertices consist of queries in the session, clicked documents and corresponding websites; and edges reflect undifferentiated semantic relationships. The authors propose a supervised model based on an objective function that aims at optimizing, over session data, the log-likelihood of reaching a leaf (i.e., query, URL) in the corresponding Huffman tree.
In contrast to most all the aforementioned works that model query representation as supervised text representation learning based on the core idea of "query sentence", we model query representation as multi-view learning of manifold underlying queries and document results based on the core idea of interactive intentions (Xie, 2002) that provide soft supervision during the search session.
2.2 Session-aware query reformulation
Session-aware query reformulation is involved in retrieval-based interactive systems, including dynamic IR systems (Yang et al., 2016), multi-turn Question Answering (QA) (Mensio et al., 2018), and dialogue systems (Cui et al., 2019). Several works studied the connections between search sessions, intentions in query reformulation, and search behavior (Lu et al., 2017; Liu et al., 2019b; Tamine et al., 2020). Among the major findings, we particularly mention the following: (1) query reformulation patterns can be observed in search sessions providing insights on the search process characteristics such as underlying search task stage (Tamine et al., 2020; Eickhoff et al., 2014) and success (Odijk et al., 2015); (2) during the session search, system's results often lead to a change in both user's knowledge and the complexity of subsequent queries (Eickhoff et al., 2014; Liu et al., 2019a); (3) user search process runs into sequential phases, specialization, and intent shift. As user's search intents are gradually satisfied based on system's results, their subsequent queries lean to topically shift (Chen et al., 2021).
The main findings that have been drawn from the literature review strengthen our motivation toward learning single-view query embeddings that capture hidden session-related patterns from the two perspectives of user's sequence of query reformu
lations in the one hand and system's results in the other hand, and then identify mutual information that can reveal similarities across users' search intents.
3.1 Multi-view representation learning
Multi-view representation learning (Li et al., 2019) aims to recover a meaningful latent representation of a target object using data provided by one or multiple sources. The views correspond to measurement modalities from such different sources, such as text and images of the same scene (Hwang and Grauman, 2012) but may also be multiple information from the same source such as document text and hyperlinks (Bickel and Scheffer, 2004). Potential applications of multi-view learning include cross-modal retrieval (Hwang and Grauman, 2012; Li et al., 2003) and machine translation (Faruqui and Dyer, 2014). SOTA methods for multi-view feature learning are the Canonical Correlation Analysis (CCA) (Dhillon et al., 2011) and Cross-modal Factor Analysis (CFA) (Li et al., 2003) whose primary goal is to maximize the correlations of features among multiple different views. These methods generally admit global solutions and ignore the non-linearities of multi-view data (Viinikanoja et al., 2010). Unlikely, k-neighbor based manifold learning methods such as Laplacian Eigenmaps (Belkin and Niyogi, 2003), IsoMap (Tenenbaum et al., 2000), and Uniform Manifold Approximation and Projection (UMAP) (McInnes et al., 2018) recover non-linear dependencies between views. The core of these methods relies upon optimization over a graphical representation of different data sets that are characterized by the same underlying manifold where edges in the graph are computed to preserve the topological structure of this manifold. This optimization yields a shared low-dimensional space where the latent representations of semantically similar data are spatially close to one another. Recently, several proposed methods for multi-view representation learning are based on deep neural networks. For instance, Deep CCA aims to learn complex nonlinear transformations of two views in a shared space (Andrew et al., 2013).
3.2 Definitions and notations
We introduce here some key definitions. Note that we refer the term of embedding to either the user-view query vector or system-view query vector and
refer the term of representation as the final multiview query vector.
Definition 1. Search session. In the literature review, there are two main definitions of search sessions: (1) a physical session (Hagen et al., 2013) is a set of consecutive queries automatically delimited using a time-out threshold on user's activities; (2) a task-based session which targets an atomic information need through a set of queries that are possibly neither consecutive nor within the same time-based session. SaQuEViL can be readily applied to both definitions of search sessions.
Formally, let  $S$  be the set of users' search sessions. A user's search session  $S \subset S$  consists of: (1) all on-session user's queries  $q_{1,S}, q_{2,S}, \ldots, q_{k,S}$  ordered by time where each query  $q_{m,S}$ , consists of  $K_m$  words  $q_{m,S} = \{w_{m1}, w_{m2}, \ldots, w_{mK_m}\}$ ; (2) the sets of  $N$  top documents returned by the retrieval system as an answer to each query  $q_{m,S}$ , denoted as  $\mathcal{D}_{m,S}^N$ .
Definition 2. User-view query embedding. Each on-session query  $q_{m,S}$  is embedded as a  $d_1$ -dimensional user-view query embedding, denoted as  $\mathbf{q}_{m,S}^u \in \mathbb{R}^{d_1}$ , that captures the user's formulation of his search intent.  $\mathbf{q}_{m,S}^u$  is encoded based on its formulation  $\{w_{m1}, w_{m2}, \ldots, w_{mK_m}\}$  as well as all the formulations of the previous queries in the session  $\{q_{m-1,S}, q_{m-2,S}, \ldots, q_{1,S}\}$ .
Definition 3. System-view query embedding. Each on-session query  $q_{m,S}$  is embedded as a  $d_2$ -dimensional system-view query embedding, denoted as  $\mathbf{q}_{m,S}^s \in \mathbb{R}^{d_2}$ , that captures the system's understanding of the user's search intent.  $\mathbf{q}_{m,S}^s$  is encoded based on document results obtained from the concatenation of the query  $q_{m,S}$  along with previous queries in the session.
4 Session-Aware Query Representation By Multi-View Learning
Let  $S = \{S_{1},\ldots ,S_{K}\}$  be a set of sessions such as  $S_{i} = \{q_{1,i},q_{2,i},\dots,q_{ki,i}\}$ , including a total of  $n$  on-session queries  $q_{m,i}$  with  $n = (\sum ki)_{i = 1}^{i = K}$ . The objective of SaQuEViL is twofold: (1) encoding  $\Sigma^1\in \mathbb{R}^{n\times d_1}$  (resp.  $\Sigma^2\in \mathbb{R}^{n\times d_2}$ ) the vector space embedding and user-view query embeddings  $\mathbf{q}_{m,i}^{u}$  (resp. system-view query embeddings  $\mathbf{q}_{m,i}^{s}$ ); (2) learn a multi-view latent space  $\Sigma^{*}\in \mathbb{R}^{n\times d}$  (with  $d\leq min(d_1,d_2)$ ) and query representations  $\hat{\mathbf{q}}_{m,i}\in \Sigma^{*}$  by jointly achieving pairwise align-
ments between the user-view embedding  $\mathbf{q}_{m,i}^{u}$  and system-view embedding  $\mathbf{q}_{m,i}^{s}$  and recovering an optimal alignment of manifolds over all the query representations  $\hat{\mathbf{q}}_{m,i}$ . Final representations are picked to match the downstream task, either when document matching is required or session-aware query is required.
The two key assumptions of multi-view learning are satisfied (Blum and Mitchell, 1998; Foster et al., 2008): (1) each of the user-view and system-view are independent conditionally to the sessions; and (2) the two single views provide a redundant estimate of the session.
4.2 Multi-view query representation learning
4.2.1 Framework overview
Figure 1 presents an overview of the SaQuE-ViL framework. For encoding the single-view query embeddings  $\mathbf{q}_{m,i}^{u}$ ,  $\mathbf{q}_{m,i}^{s}$ , we opted for BERT (Devlin et al., 2019) as transformer embedding and followed the standard CLS encoding strategy (BERTCLS). So,  $\mathbf{q}_{m,i}^{u}$  (resp.  $\mathbf{q}_{m,i}^{s}$ ) is obtained by applying  $\Gamma(\mathrm{BERT}_{CLS}([CLS]q_{m,i}))$  (resp.  $\otimes_{j=1}^{j=N}\mathrm{BERT}_{CLS}([CLS]head(d_{m,i}^{j}))$ ), where  $\otimes$  is a vector concatenation operator,  $head(\cdot)$  is a function that returns the title and first tokens of a given document, and  $\Gamma$  is an expansion function such as broadcast used to match the dimensions.
Following, we detail the key principles of multiview query representation learning  $\hat{\mathbf{q}}_{m,i}$  using linear (CFA (Dhillon et al., 2011)) and non-linear (UMAP (McInnes et al., 2018)) methods.
4.2.2 CFA-based representation learning
Given the two mean centered matrices  $\mathcal{Q}^u\in$ $\mathbb{R}^{d_1\times n}$  and  $\mathcal{Q}^s\in \mathbb{R}^{d_2\times n}$ , where columns refer respectively to the user-view embeddings  $\mathbf{q}_{m,i}^u$  and system-view embedding  $\mathbf{q}_{m,i}^s$ , CFA learns two linear and orthogonal transformations  $A\in \mathbb{R}^{d_1\times d}$  and  $B\in \mathbb{R}^{d_2\times d}$  such that the distance between  $A^{\intercal}\mathcal{Q}^u$  and  $B^{\intercal}\mathcal{Q}^s$  is minimized. The CFA objective is:
$$
A ^ {*}, B ^ {*} = \operatorname {a r g m i n} _ {A, B} \left(\left\| A ^ {\top} Q ^ {u} - B ^ {\top} Q ^ {s} \right\| _ {F}\right) \tag {1}
$$
where  $A^{\intercal}A = I$  and  $B^{\intercal}B = I$  and  $\| \cdot \| _F$  is the Frobenius norm. The solution of Equation (1) is obtained through the Singular Value Decomposition (SVD) of  $Z = (\mathcal{Q}^u)^{\intercal}\mathcal{Q}^s$ , such as  $Z = S_{z}V_{z}D_{z}$  and  $A^{*} = S_{z},B^{*} = D_{z}$  (Krzanowski, 1988). Thus, we obtain the multi-view query representations  $\mathbf{q}^u$  and  $\mathbf{q}^s$  as the rows of the user-view or
Figure 1: Overview of the SaQuEViL framework.
system-view transformations  $\hat{\mathcal{Q}}^u = (\mathcal{Q}^u)^\intercal A^*$  and  $\hat{\mathcal{Q}}^s = (\mathcal{Q}^s)^\intercal B^*$  respectively.
4.2.3 UMAP-based representation learning
Let  $G(\mathcal{V}, \xi)$  be the graph where the vertices  $\mathcal{V}$  correspond to queries  $q_{m} \in \cup (S_{i})_{i=1}^{K}$  and  $\xi$  the edges that reflect a weighted neighborhood relationship  $\mathbf{q}_{m} \sim \mathbf{q}_{m'}$  defined in matrix  $\mathcal{W}$  such as  $\mathcal{W}(m, m') > 0$  if  $\mathbf{q}_{m}, \mathbf{q}_{m'}$  are neighbors. The two key differences between SOTA graph-based manifold learning algorithms (Belkin and Niyogi, 2003; Tenenbaum et al., 2000) lie in the construction of the k-neighbor edges  $\xi$  and the choice of the weights  $\mathcal{W}(i, j)$ . Specifically, in the multi-view setting of the UMAP method, for each query  $q_{m}$ , there are two induced local graphs: (1) the user graph  $G_{m}^{u}(\mathcal{V}^{u}, \xi_{m}^{u})$  where  $\mathcal{V}^{u}$  is the set of k-nearest neighbors of  $\mathbf{q}_{m}^{u}$  denoted as  $Fset^{u}(q_{m})$  and  $\xi_{m}$  is the set of outgoing edges directed from  $q_{m}$  to its set k-nearest neighbors  $q_{mj}^{u}$  thereby inducing the similarity relationship  $\mathbf{q}_{m}^{u} \sim \mathbf{q}_{mj}^{u}$  defined in matrix  $\mathcal{W}^{u}(n, n)$ ; (2) the system graph  $G_{m}^{s}(\mathcal{V}^{s}, \xi_{m}^{s})$  where  $\mathcal{V}^{s}$  is the set of k-nearest neighbors of  $\mathbf{q}_{m}^{s}$  denoted as  $Fset^{s}(q_{m})$  and  $\xi_{m}^{s}$  is the set of outgoing edges directed from  $q_{m}$  to its set k-nearest neighbors  $\mathbf{q}_{mj}^{s}$  thereby inducing the similarity relationship  $\mathbf{q}_{m}^{s} \sim \mathbf{q}_{mj}^{s}$  defined in matrix  $\mathcal{W}^{s}(n, n)$ . Pairwise alignment between the user-view and system-view of query  $q_{m}$  is ensured by building the graph  $G(\mathcal{V}, \xi)$  as a graph intersection between user graph  $G_{m}^{u}(\mathcal{V}^{u}, \xi_{m}^{u})$  and system graph  $G_{m}^{s}(\mathcal{V}^{s}, \xi_{m}^{s})$  for each query  $q_{m} \in \cup (S_{i})_{i=1}^{K}$ . This intersection builds the weighting matrix  $\mathcal{W}(n, n)$  based on the weighting matrices  $\mathcal{W}^{u}$  and  $\mathcal{W}^{s}$ . Spectral optimization of the multi-view query representations
is then achieved by functions  $\mathbf{f}:\mathcal{V}\mapsto \mathbb{R}$  that recover the optimal alignment of manifolds underlying queries  $q_{m}\in \cup (S_{i})_{i = 1}^{K}$  through the minimization of a cost on graph  $G(\mathcal{V},\xi)$ , defined as (McInnes et al., 2018):
$$
\mathcal {L} (f) = \sum_ {S \in \mathcal {S}; q _ {m}, q _ {m} ^ {\prime} \in S} \frac {1}{2} \left(f _ {m} - f _ {m} ^ {\prime}\right) ^ {2} \mathcal {W} \left(m, m ^ {\prime}\right) \tag {2}
$$
subject to scale and translation constraints  $f^T f = 1$  and  $f^T e = 0$ .
The optimization process of UMAP is detailed in Belkin and Niyogi (2003); McInnes et al. (2018).
We address the following research questions:
RQ1) How does the SaQuEViL framework perform in query classification and session-based retrieval as downstream tasks?
RQ2) To what extent the SaQuEViL embedding space preserves the similarities of each of the single-view embedding spaces?
RQ3) Can we use SaQuEViL framework to understand user's search behavior?
5.1 Downstream tasks
5.1.1 Query classification
The goal of query classification consists in assigning an incoming query the most appropriate topic labels (categories). Labels are pre-defined and search-related data are available to train each label.
Data. As previously done by Zamani and Croft (2016); Zamani et al. (2017) to evaluate query em
bedding performances, we used the KDD 2005 dataset (Li et al., 2005). The dataset consists of 800 queries recorded from MSN search log. The dataset also includes 43 categories -that act as candidate task-based sessions- labeled by human assessors. Accordingly, we assume that the set of queries belonging to each target category  $c$  represents a session  $S_{c}$ . To solely measure the quality of the query representations and ensure comparability across query representations, we opted for the classification strategy proposed in Zamani and Croft (2016); Zamani et al. (2017). We first compute the probability of each category (session)  $p(Sc_{i} / q) = \frac{\delta(\vec{Sc}_{i},\vec{q})}{\sum_{j}\delta(\vec{Sc}_{j},\vec{q})}$  where  $\vec{q}$  is a query vector,  $\vec{Sc}_i$  is the centroid vector of category (session)  $Sc_{i}$ .  $\vec{Sc}_i$  is computed by averaging the query vectors  $q_{ki}$  of queries  $q_{ki}$  belonging to session  $Sc_{i}$ . Then we select the  $N$  top sessions with the highest probabilities as the more likely ones to be assigned to query  $q$ .
Evaluation metrics. We consider the evaluation metrics used in the KDD challenge (Li et al., 2005), Recall and F1 measures, and carefully followed their description to implement our evaluation script. Statistical tests are performed using two-tailed paired t-test. We depict a significant increase for  $p < 0.05$  as  $^*$ .