SlowGuess/NAACL_2022_OCR huggingface dataset:
Baselines and scenarios. We reported traditional SOTA pre-trained embeddings as query encoders GloVe (Pennington et al., 2014), Word2vec (Mikolov et al., 2013) and BERT (Devlin et al., 2019), as well as RPE (Zamani and Croft, 2016; Zamani et al., 2017), a SOTA relevance-based query representation model. To show the impact of user-view and system-view alignment, we also compared our multi-view CFA-based and UMAP-based query representations  $\hat{\mathbf{q}}$  to the representation vector obtained by concatenation of  $\mathbf{q}^u$  and  $\mathbf{q}^s$  vectors. The latter scenario is denoted w/o Align.
Training and inference. We performed a 5-fold cross-validation over the queries and used the document rankings provided by the ClueWeb12<sup>1</sup> corpus to learn the SaQuEViL multi-view query representations. The ClueWeb12 corpus was indexed using the respective default configuration of Anserini<sup>2</sup> while the retrieval was done using the default configuration of Pyserini<sup>3</sup> search. With respect to Fig-
ure 1, projected  $\hat{\mathbf{q}}$  vectors are averaged in order to obtain a unique vector per query. The number of labels assigned to each query was tuned on the training set from 1 to 5.
5.1.2 Session-based retrieval
The goal of session-based retrieval consists in evaluating document rankings over user sessions rather than isolated queries (Carterette et al., 2016).
Data. We use the TREC 2014 session track (Carterette et al., 2016) which provides the following: (1) 1,257 full sessions among which 1,021 of these have at least one reformulation. On average there are 4.33 queries per session, among which the final query in the session is referred to as the current query; (2) the ranked list of documents for each past query; and (3) human annotations about type of search for 54 sessions; the latter are labeled using 4 categories of user search behavior w.r.t. the classification designed by Li and Belkin (2008): known-item, interpretive, known-subject, and exploratory.
It is worth noting that we did not use the users' clicks in our experiments since they are considered as weak supervision. The corpus used is the ClueWeb12 collection. The relevance of a document was judged for the results of the current query but judgment is based on the whole session.
Evaluation metrics. We use the TREC session track's official metrics. These are: nDCG@10, ERR@10, nERR@10, and PC@10. All runs are evaluated using the official evaluation script<sup>4</sup>.
Baselines and scenarios. We used classical baselines including Current and Aggregated query. The latter is a concatenation of all the session's queries as suggested in Van Gysel et al. (2016).
Training and inference. In contrary to query classification, projected  $\hat{\mathbf{q}}$  vectors are not aggregated as each is used for document ranking. We first compute a neural score by calculating the cosine similarity between the session vector  $\sum_{j=1}^{m-1} \hat{\mathbf{q}}_{j,S}$  and the document vector  $\hat{\mathbf{q}}_{m,S}^{d_{m,S}^l}$  in the SaQuEViL space. Then we obtain the final score used for document ranking by linearly combining the neural score with the BM25 score as commonly done in neural IR (MacAvaney, 2020).
1https://lemurproject.org/clueweb12. php/
$^{2}$ https://github.com/castorini/anserini
<sup>3</sup>https://github.com/castorini/pyserini
4https://trec.nist.gov/data/ session2014.html
Model	Precision	F1
GloVe	0.3643 (+22.0%)	0.3912 (+28.3%)
Word2vec	0.3712 (+19.7%)	0.4008 (+25.2%)
BERT	0.4143 (+7.2%)	0.4537 (+10.6%)
RPE	0.3961 (+12.2%)	0.4294 (+16.9%)
SaQuEViL
w/o Align	0.4274	0.4827*
CFA	0.4443*	0.5020*
UMAP	0.4246	0.4802*
Table 1: Performance of SaQuEViL query representations and baselines (GloVe (Pennington et al., 2014), Word2vec (Mikolov et al., 2013), BERT (Devlin et al., 2019), and RPE (Zamani and Croft, 2017)) in query classification. The improvements over each baseline of our best scenario, SaQuEViL CFA, are reported in brackets. The highest values are highlighted in bold. Improvement significance w.r.t. BERT is indicated by the superscript \*'.
6.1 RQ1: Effectiveness evaluation of SaQuEViL in downstream tasks
6.1.1 Query classification
Table 1 presents the performance results in terms of Precision and F1. Note that one strong baseline is obtained by encoding the query with BERT (0.4143) which clearly outperforms a supervised alternative (0.3961), e.g., RPE which is trained on relevance signals (Zamani and Croft, 2017). It can be explained as RPE do not use contextualized embeddings as BERT. We can interestingly see that SaQuEViL, even trained without supervision, outperforms (0.443) both unsupervised (GloVe, Word2vec, and BERT) and supervised encoders (RPE model). This result clearly indicates the value of the alignment to identify relevant mutual information between user's view through query reformulation and system's view through document rankings to enhance the query representation. We can also see that even without alignment, SaQuEViL (0.4274) outperforms BERT (0.4143) indicating that each view information is helpful on this task. Finally, our best scenario corresponds to the SaQuEViL CFA setup that achieves a minimum improvement of  $7\%$  in terms of Precision and F1 w.r.t. reported baselines. This result leads us to consider that linear dependencies are revealed from session-based query reformulations and corresponding documents.
Model	NDCG@10	ERR@10	nERR@10	PC@10
Current	0.1659	0.1639	0.2332	0.3190
Aggregated	0.1834	0.1952	0.2645	0.3460
SaQuEvI L
w/o Align	0.1841	0.2021	0.2749	0.3340
CFA	0.1843	0.1950	0.2646	0.3473
UMAP	0.1835	0.1951	0.2644	0.3450
Table 2: Performance of SaQuEViL query representations and baselines (Aggregated (Van Gysel et al., 2016)) in session-based retrieval. Best results are highlighted in bold.
6.1.2 Session-based retrieval
Table 2 presents the performance scores of SaQuEViL scenarios and baselines in the session-based retrieval downstream task. As expected, including session information outperforms (0.1834) the use of the single query (0.1659) in terms of NDCG@10, but also for all the other metrics. Moreover, we can notice that SaQuEViL slightly improves the Aggregated (Van Gysel et al., 2016) results but none scenario shows a clear wining. SaQuEViL w/o Align setup outperforms in terms of ERR@10 (0.2021) and nERR@10 (0.2749) but SaQuEViL CFA obtains the best scores for NDCG@10 (0.1843) and PC@10 (0.3473). Nevertheless, the improvements for the session-based retrieval downstream task are modest<sup>5</sup>. We can also notice that CFA and UMAP methods exhibit the same performance trend.
6.2 RQ2: Analysis of the SaQuEViL multi-View embedding space
Our main objective here is to analyse to what extent the SaQuEvI L framework builds a shared embedding space that preserves the structure of the single-view spaces. Grounded with the results obtained above (Section 6.2), we achieve this goal by analyzing the discrepancies between the singleview spaces and the shared space obtained with SaQuEvI L using the query representations learned in query classification. For each target query  $q$  ,we consider the k-neighbors of  $\hat{\mathbf{q}}$  in the SaQuEvI L shared space as the gold standard and the plurality vote of the k-neighbors in each of the single-view spaces, namely,  $\mathbf{q}^u$  and  $\mathbf{q}^s$  ,as the prediction. We used the cosine similarity to find neighbors and then compute Precision, Recall and F-measure metrics under a multi-label setup, where each query
Note that stronger results on the TREC session 2014 dataset are reported by Aloteibi and Clark (2020), but we only focused on an extrinsic use of SaQuEViL and integration to task specific models is out of the scope of the paper.
Figure 2: F1 performances when comparing SaQuEviL CFA (left) / UMAP (right) multi-view space and the concatenation of both views embedding. Number of neighbors and ranked documents are in log scale. Better in color as brighter color indicates higher values.
Model	First k queries into the session
1	3	6	9	all
Aggregated	0.373	0.573	0.553	0.535	0.535
SaQuEvI L
w/o Align	0.462	0.571	0.607	0.589	0.589
CFA	0.516	0.569	0.625	0.625	0.625
UMAP	0.498	0.571	0.625	0.589	0.589
Table 3: F1-micro performances of SaQuEVI L and baseline (Aggregated (Van Gysel et al., 2016)) encoders in search type classification using TREC session 2014. Highest values of F1-measure are highlighted in bold.
identifier is considered as a target class. In particular, we analyze the impact of two key parameters of the SaQuEViL framework: number of neighbors  $(k)$  and number of top documents  $(N)$  used to learn the query representations. Results for different values of  $k$  (1, 2, 4, 8, and 16) and  $N$  (2, 4, 8, 16, 32, and 64) in log scale are presented in Figure 2. Three main conclusions can be grasped from Figure 2: (1) increasing the number of neighbors increases the similarity between the spaces until 8-16 neighbors then it stabilizes for both methods (CFA and UMAP) in terms of F1; (2) adding extra documents impacts in the same way, e.g., positive at early increments and then stabilizes, but for the two multi-view learning methods; (3) a higher preservation of original similarities in SaQuEViL spaces correlates with higher performances on the downstream task as SaQuEViL CFA obtains a maximum score close to 0.20 of F1 while UMAP is 0.06 points behind  $(0.14\mathrm{of~F1})^6$ . These results might shed light on possible controllable room of improvements of a wide range of downstream tasks including, but not limited to session-based retrieval.
6.3 RQ3: Search behavior understanding
Our aim here is to understand in what extent the SaQuEViL representation space helps understanding behaviors in user session. To do so, we used the type of search annotations provided in the TREC session 2014 dataset (known-item, interpretive, known-subject, and exploratory). A standard 5-cross fold setup with k-nearest neighbor classifier is used to draw the intrinsic capabilities of the encoders to distinguish user search behavior types. Average results of F1-micro across the 5 folds are presented in Table 3. To perform the classification at the test stage, we used as context the first  $k$  queries of sessions (columns 1, 3, 6 and 9) as well as the full session (column all). As can be seen from Table 3, SaQuEViL CFA encoder (0.625) clearly outperforms the proposed alternatives, the BERT encoder for the Aggregated queries (0.535) and the SaQuEViL w/o Align (0.589) when considering the full session. Looking at the impact of context length  $(k)$  in the classification, we can note that the Aggregated query representation starts with a low performance (0.373) and, when up to 3 queries are used in the session, it achieves the maximal performance (0.573). However, the SaQuEViL w/o Align encoder starts in a higher performance (0.462) and achieves the maximal performance when up to 6 queries are used from the session (0.607). In both cases, the performance drops when the size of the session increases. This also points an advantage of SaQuEViL CFA encoder as it shows a more stable performance (0.516 to 0.625) regardless the number of used queries.
To further our analysis, we plot in Figure 3 distributions of distances between adjacent query pairs for each session w.r.t. corresponding search type and by using different query encoders: GloVe, SaQuE
Note that this correlation must have an upper limit lower than 1.0 (F1) as exactly similar spaces may lay on similar performances to our strategy w/o align in downstream tasks.
(c)
Figure 3: Distribution of cosine similarities for (a) GloVe, (b) SaQuEvL-w/o Align, and (c) SaQuEvL CFA between adjacent queries per session categorized by search type known-item, interpretive, known-subject, and exploratory.
ViL w/o Align, and SaQuEViL CFA<sup>7</sup>. We can see that the distribution of CFA encoder significantly differs from the other encoders. Interestingly, we note that CFA better separates the four search types and gradually differentiates the trends of query similarities based on the two dimensions of search namely "goal-quality" and "product" of the search. Indeed, the curves with more spread query similarity values (0.87-0.99) correspond to interpretive and exploratory sessions which reflect non-factual task products with either specific or amorphous goals leading to issue semantically different queries along the sessions. Unlikely, the curves with high density of narrow and relatively high similarity values (0.93-0.99) reflect factual search as characterized in known-subject and known-item search.
The paper presented SaQuEViL, a framework that learns query representations that reflect users' intents within a session-based search. By relying on the key finding that system's results affects user's query behavior and knowledge, we advocate the use of unsupervised multi-view learning to capture manifolds in a shared distributed representation space. Through experimental evaluation in two downstream tasks, we show the effectiveness of SaQuEViL over supervised and unsupervised pretrained encoders, though improvements are limited in session-based retrieval that inherently requires relevance supervision. A series of experiments and qualitative analyses also show the potential of SaQuEViL to control the representation space through key parameters that directly influence performance of downstream tasks and additionally, to
clearly separate user behavior patterns in search sessions. We believe that this work opens avenues of research in the design of unsupervised distributed representations able to support search tasks, which has not received much attention yet.
This work was supported by the Agence National de la Recherche (ANR), through project CoST (https://www.irit.fr/COST/), code ANR-18-CE23-0016.
Saad Aloteibi and Stephen Clark. 2020. Learning to personalize for web search sessions. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 15-24. ACM.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. ICML'13, page III-1247-III-1255. JMLR.org.
Mikhail Belkin and Partha Niyogi. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373-1396.
Steffen Bickel and Tobias Scheffer. 2004. Multi-view clustering. In Proceedings of the Fourth IEEE International Conference on Data Mining, ICDM '04, page 19-26, USA. IEEE Computer Society.
Lidong Bing, Zheng-Yu Niu, Piji Li, Wai Lam, and Haifeng Wang. 2018. Learning a unified embedding space of web search from large-scale query log. Knowledge-Based Systems, 150:38 - 48.
Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT'98, page 92-100, New York, NY, USA. Association for Computing Machinery.
7UMAP exhibits the same distribution trend than CFA and has not been presented for limited space.
Ben Carterette, Paul Clough, Mark Hall, Evangelos Kanoulas, and Mark Sanderson. 2016. Evaluating retrieval over sessions: The trec session track 2011-2014. SIGIR '16, page 685-688, New York, NY, USA. Association for Computing Machinery.
Jia Chen, Jiaxin Mao, Yiqun Liu, Fan Zhang, Min Zhang, and Shaoping Ma. 2021. Towards a better understanding of query reformulation behavior in web search. In Proceedings of the Web Conference 2021, pages 743-755.
Chen Cui, Wenjie Wang, Xuemeng Song, Minlie Huang, Xin-Shun Xu, and Liqiang Nie. 2019. User attention-guided multimodal dialog systems. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, page 445-454, New York, NY, USA. Association for Computing Machinery.
Mostafa Dehghani, Sascha Rothe, Enrique Alfonseca, and Pascal Fleury. 2017. Learning to attend, copy, and generate for session-based query suggestion. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, pages 1747-1756.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. 2011. Multi-view learning of word embeddings via cca. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS'11, page 199-207, Red Hook, NY, USA. Curran Associates Inc.
Carsten Eickhoff, Jaime Teevan, Ryen White, and Susan Dumais. 2014. Lessons from the journey: A query log analysis of within-session learning. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining, WSDM '14, page 223-232, New York, NY, USA. Association for Computing Machinery.
Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462-471, Gothenburg, Sweden. Association for Computational Linguistics.
Dean P. Foster, Rie Johnson, and Tong Zhang. 2008. Multi-view dimensionality reduction via canonical correlation analysis. Technical report.
Mihajlo Grbovic, Nemanja Djuric, Vladan Radosavljevic, Fabrizio Silvestri, Ricardo Baeza-Yates, Andrew
Feng, Erik Ordentlich, Lee Yang, and Gavin Owens. 2016. Scalable semantic matching of queries to ads in sponsored search advertising. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, page 375-384, New York, NY, USA. Association for Computing Machinery.
Matthias Hagen, Jakob Gomoll, Anna Beyer, and Benno Stein. 2013. From search session detection to search mission detection. In Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, OAIR '13, pages 85-92.
Ahmed Hassan Awadallah, Ryen W. White, Patrick Pantel, Susan T. Dumais, and Yi-Min Wang. 2014. Supporting complex search tasks. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, pages 829-838.
Jiyin He and Emine Yilmaz. 2017. User behaviour and task characteristics: A field study of daily information behaviour. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR '17, pages 67-76.
Sung Ju Hwang and Kristen Grauman. 2012. Learning the relative importance of objects from tagged images for retrieval and cross-modal search. Int. J. Comput. Vision, 100(2):134-153.
Frank Keller and Mirella Lapata. 2003. Using the web to obtain frequencies for unseen bigrams. Comput. Linguist., 29(3):459-484.
W. J. Krzanowski. 1988. Principles of Multivariate Analysis: A User's Perspective. Oxford University Press, Inc., USA.
Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K. Sethi. 2003. Multimedia content processing through cross-modal association. In Proceedings of the Eleventh ACM International Conference on Multimedia, MULTIMEDIA '03, page 604-611, New York, NY, USA. Association for Computing Machinery.
Ying Li, Zijian Zheng, and Honghua (Kathy) Dai. 2005. Kdd cup-2005 report: Facing a great challenge. SIGKDD Explor. Newsl., 7(2):91-99.
Yingming Li, Ming Yang, and Zhongfei Zhang. 2019. A survey of multi-view representation learning. IEEE Transactions on Knowledge and Data Engineering, 31(10):1863-1883.
Yuelin Li and Nicholas J. Belkin. 2008. A faceted approach to conceptualizing tasks in information seeking. Information Processing and Management, 44(6):1822-1837. Adaptive Information Retrieval.
Jimmy Lin. 2021. A proposed conceptual framework for a representational approach to information retrieval. SIGIR Forum.
Hanrui Liu, Chang Liu, and Nicholas J. Belkin. 2019a. Investigation of users' knowledge change process in learning-related search tasks. Proceedings of the Association for Information Science and Technology, 56.
Jiqun Liu, Matthew Mitsui, Nicholas J. Belkin, and Chirag Shah. 2019b. Task, information seeking intentions, and user behavior: Toward a multi-level understanding of web search. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIIR '19, page 123-132, New York, NY, USA. Association for Computing Machinery.
Kun Lu, Soohyung Joo, Taehun Lee, and Rong Hu. 2017. Factors that influence query reformulations and search performance in health information retrieval: A multilevel modeling approach. J. Assoc. Inf. Sci. Technol., 68(8):1886-1898.
Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2011. Identifying task-based sessions in search engine query logs. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, pages 277-286.
Sean MacAvaney. 2020. OpenNIR: A complete neural ad-hoc ranking pipeline. In WSDM 2020.
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction.
Rishabh Mehrotra and Emine Yilmaz. 2017. Task embeddings: Learning query embeddings using task context. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, page 2199-2202, New York, NY, USA. Association for Computing Machinery.
Martino Mensio, Giuseppe Rizzo, and Maurizio Morisio. 2018. Multi-turn qa: A rnn contextual approach to intent classification for goal-oriented systems. WWW '18, page 1075-1080, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Daan Odijk, Ryen W. White, Ahmed Hassan Awadallah, and Susan T. Dumais. 2015. Struggling and success in web search. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM '15, page 1551-1560, New York, NY, USA. Association for Computing Machinery.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In *EMNLP*, volume 14, pages 1532–1543.