OpenRaiser/Intern-Atlas huggingface dataset:
Robust Models of Mouse Movement on Dynamic Web Search Results Pages
Authors: Fernando Diaz, Ryen White, Georg Buscher, Daniel J. Liebling
Venue: Conference CIKM 2013
Year: 2013

Understanding how users examine result pages across a broad range of information needs is critical for search engine design. Cursor movements can be used to estimate visual attention on search engine results page (SERP) components, including traditional snippets, aggregated results, and advertisements. However, these signals can only be leveraged for SERPs where cursor tracking was enabled, limiting their utility for informing the design of new SERPs. In this work, we develop robust, log-based mouse movement models capable of estimating searcher attention on novel SERP arrangements. These models can help improve SERP design by anticipating searchers' engagement patterns given a proposed arrangement. We demonstrate the efficacy of our method using a large set of mouse-tracking data collected from two independent commercial search engines.
Struggling or exploring?: Disambiguating long search sessions
Authors: Ahmed Hassan Awadallah, Ryen W. White, Susan T. Dumais, Yi-Min Wang
Venue: Conference WSDM 2014
Year: 2014

Web searchers often exhibit directed search behaviors such as navigating to a particular Website. However, in many circumstances they exhibit different behaviors that involve issuing many queries and visiting many results. In such cases, it is not clear whether the user's rationale is to intentionally explore the results or whether they are struggling to find the information they seek. Being able to disambiguate between these types of long search sessions is important for search engines both in performing retrospective analysis to understand search success, and in developing real-time support to assist searchers. The difficulty of this challenge is amplified since many of the characteristics of exploration (e.g., multiple queries, long duration) are also observed in sessions where people are struggling. In this paper, we analyze struggling and exploring behavior in Web search using log data from a commercial search engine. We first compare and contrast search behaviors along a number dimensions, including query dynamics during the session. We then build classifiers that can accurately distinguish between exploring and struggling sessions using behavioral and topical features. Finally, we show that by considering the struggling/exploring prediction we can more accurately predict search satisfaction.
One query, many clicks: Analysis of queries with multiple clicks by the same user
Authors: Elad Kravi, Ido Guy, Avihai Mejer, David Carmel, Yoelle Maarek, Dan Pelleg, Gilad Tsur
Venue: Conference CIKM 2016
Year: 2016

In this paper, we study multi-click queries - queries for which more than one click is performed by the same user within the same query session. Such queries may reflect a more complex information need, which leads the user to examine a variety of results. We present a comprehensive analysis that reveals unique characteristics of multi-click queries, in terms of their syntax, lexical domains, contextual properties, and returned search results page. We also show that a basic classifier for predicting multi-click queries can reach an accuracy of 75% over a balanced dataset. We discuss the implications of our findings for the design of Web search tools.
Introduction to Social Recommendation
Authors: Irwin King, Michael R. Lyu, Hao Ma 0001
Venue: Conference WWW 2010
Year: 2010

As the exponential growth of information generated on the World Wide Web, Social Recommendation has emerged as one of the hot research topics recently. Social Recommendation forms a specific type of information filtering technique that attempts to suggest information (blogs, news, music, travel plans, web pages, images, tags, etc.) that are likely to interest the users. Social Recommendation involves the investigation of collective intelligence by using computational techniques such as machine learning, data mining, natural language processing, etc. on social behavior data collected from blogs, wikis, recommender systems, question & answer communities, query logs, tags, etc. from areas such as social networks, social search, social media, social bookmarks, social news, social knowledge sharing, and social games. In this tutorial, we will introduce Social Recommendation and elaborate on how the various characteristics and aspects are involved in the social platforms for collective intelligence. Moreover, we will discuss the challenging issues involved in Social Recommendation in the context of theory and models of social networks, methods to improve recommender systems using social contextual information, ways to deal with partial and incomplete information in the social context, scalability and algorithmic issues with social computational techniques.
How intention informed recommendations modulate choices: A field study of spoken word content
Authors: Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, Deborah Estrin
Venue: Conference WWW 2019
Year: 2019

People's content choices are ideally driven by their intentions, aspirations, and plans. However, in reality, choices may be modulated by recommendation systems which are typically trained to promote popular items and to reinforce users' historical behavior. As a result, the utility and user experience of content consumption can be affected implicitly and undesirably. To study this problem, we conducted a 2 × 2 randomized controlled field experiment (105 urban college students) to compare the effects of intention informed recommendations with classical intention agnostic systems. The study was conducted in the context of spoken word web content (podcasts) which is often consumed through subscription sites or apps. We modified a commercial podcast app to include (1) a recommender that takes into account users' stated intentions at onboarding, and (2) a Collaborative Filtering (CF) recommender during daily use. Our study suggests that: (1) intention-aware recommendations can significantly raise users' interactions (subscriptions and listening) with channels and episodes related to intended topics by over 24%, even if such a recommender is only used during onboarding, and (2) the CF-based recommender doubles users' explorations on episodes from not-subscribed channels and improves satisfaction for users onboarded with the intention-aware recommender.
Adaptive Hierarchical Translation-based Sequential Recommendation
Authors: Yin Zhang, Yun He, Jianling Wang, James Caverlee
Venue: Conference WWW 2020
Year: 2020

We propose an adaptive hierarchical translation-based sequential recommendation called HierTrans that first extends traditional item-level relations to the category-level, to help capture dynamic sequence patterns that can generalize across users and time. Then unlike item-level based methods, we build a novel hierarchical temporal graph that contains item multi-relations at the category-level and user dynamic sequences at the item-level. Based on the graph, HierTrans adaptively aggregates the high-order multi-relations among items and dynamic user preferences to capture the dynamic joint influence for next-item recommendation. Specifically, the user translation vector in HierTrans can adaptively change based on both a user’s previous interacted items and the item relations inside the user’s sequences, as well as the user’s personal dynamic preference. Experiments on public datasets demonstrate the proposed model HierTrans consistently outperforms state-of-the-art sequential recommendation methods.
Finally, a Downloadable Test Collection of Tweets
Authors: Royal Sequiera, Jimmy Lin
Venue: Conference SIGIR 2017
Year: 2017

Due to Twitter's terms of service that forbid redistribution of content, creating publicly downloadable collections of tweets for research purposes has been a perpetual problem for the research community. Some collections are distributed by making available the ids of the tweets that comprise the collection and providing tools to fetch the actual content; this approach has scalability limitations. In other cases, evaluation organizers have set up APIs that provide access to collections for specific tasks, without exposing the underlying content. This is a workable solution, but difficult to sustain over the long term since someone has to maintain the APIs. We have noticed that the non-profit Internet Archive has been making available for public download captures of the so-called Twitter "spritzer" stream, which is the same source as the Tweets2013 collection used in the TREC 2013 and 2014 Microblog Tracks. We analyzed both datasets in terms of content overlap and retrieval baselines to show that the Internet Archive data can serve as a drop-in replacement for the Tweets2013 collection, thereby providing the research community with, finally, a downloadable collection of tweets. Beyond this finding, we also study the impact of tweet deletions over time and how they affect the test collections.
Flexible ir pipelines with capreolus
Authors: Andrew Yates, Kevin Martin Jose, Xinyu Zhang, Jimmy Lin
Venue: Conference CIKM 2020
Year: 2020

While a number of recent open-source toolkits for training and using neural information retrieval models have greatly simplified experiments with neural reranking methods, they essentially hard code a "search-then-rerank'' experimental pipeline. These pipelines consist of an efficient first-stage ranking method, like BM25, followed by a neural reranking method. Deviations from this setup often require hacks; some improvements, like adding a second reranking step that uses a more expensive neural method, are infeasible without major code changes. In order to improve the flexibility of such toolkits, we propose implementing experimental pipelines as dependency graphs of functional "IR primitives,'' which we call modules, that can be used and combined as needed. For example, a neural IR pipeline may rerank results from a Searcher module that efficiently retrieves results from an Index module that it depends on. In turn, the Index depends on a Collection to index, which is provided by the pipeline. This Searcher module is self-contained: the pipeline does not need to know about or interact with the Index of the Searcher, which is transparently shared among Searcher modules when possible (e.g., a BM25 and a QL Searcher might share the same Index). Similarly, a Reranker module might depend on a Trainer (e.g., Tensorflow), feature Extractor, Tokenizer, etc. In both cases, the pipeline needs to interact only with the Reranker or Searcher directly; the complexity of their dependencies is hidden and intelligently managed. We rewrite the Capreolus toolkit to take this approach and demonstrate its use. %in a series of code examples and experiments.
PSTIE: Time Information Enhanced Personalized Search
Authors: Zhengyi Ma, Zhicheng Dou, Guanyue Bian, Ji-Rong Wen
Venue: Conference CIKM 2020
Year: 2020

Personalized search aims to improve the search quality by re-ranking the candidate document list based on user's historical behavior. Existing approaches focus on modeling the order information of user's search history by sequential methods such as Recurrent Neural Network (RNN). However, these methods usually ignore the fine-grained time information associated with user actions. In fact, the time intervals between queries can help to capture the evolution of query intent and document interest of users. Besides, the time intervals between past actions and current query can reflect the re-finding tendency more accurately than discrete steps in RNN. In this paper, we propose PSTIE, a fine-grained Time Information Enhanced model to construct more accurate user interest representations for Personalized Search. To capture the short-term interest of users, we design time-aware LSTM architectures for modeling the subtle interest evolution of users in continuous time. We further leverage time in calculating the re-finding possibility of users to capture the long-term user interest. We propose two methods to utilize the time-enhanced user interest into personalized ranking. Experiments on two datasets show that PSTIE can effectively improve the ranking quality over state-of-the-art models.
Multi-objective Optimization for Guaranteed Delivery in Video Service Platform
Authors: Hang Lei, Yin Zhao, Longjun Cai
Venue: Conference KDD 2020
Year: 2020

Guaranteed-Delivery (GD) is one of the important display strategies for the IP videos in video service platform. Different from the traditional recommendation strategy, GD requires the delivery system to guarantee the exposure amount (also called impressions in some works) for the content, where the amount generally comes from the purchase contract or business consideration of the platform. In this paper, we study the problem of how to maximize certain gains, such as video view (VV) or fairness of different contents (CTR variations between contents) under the GD constraints. We formulate such a problem as a constrained nonlinear programming problem, in which the objectives are to maximize the total VVs of contents and the exposure fairness between contents. In order to capture the trends of VV versus the impression number (page views, PV) for each video content, we propose a parameterized ordinary differential equation (ODE) model, and the parameters of the ODE are fitted by the video historical PV and CLICK datas. To solve the constrained nonlinear programming, we use genetic algorithm (GA) with a specific design of coding scheme considering the ODE constraints. The empirical study based on real-world data and online test on Youku.com verifies the effectiveness and superiority of our approach compared with the state of the art in the industry practice.
Alleviating spurious correlations in knowledge-aware recommendations through counterfactual generator
Authors: Shanlei Mu, Yaliang Li, Wayne Xin Zhao, Jingyuan Wang, Bolin Ding, Ji-Rong Wen
Venue: Conference SIGIR 2022
Year: 2022

Limited by the statistical-based machine learning framework, a spurious correlation is likely to appear in existing knowledge-aware recommendation methods. It refers to a knowledge fact that appears causal to the user behaviors (inferred by the recommender) but is not in fact. For tackling this issue, we present a novel approach to discovering and alleviating the potential spurious correlations from a counterfactual perspective. To be specific, our approach consists of two counterfactual generators and a recommender. The counterfactual generators are designed to generate counterfactual interactions via reinforcement learning, while the recommender is implemented with two different graph neural networks to aggregate the information from KG and user-item interactions respectively. The counterfactual generators and recommender are integrated in a mutually collaborative way. With this approach, the recommender helps the counterfactual generators better identify potential spurious correlations and generate high-quality counterfactual interactions, while the counterfactual generators help the recommender weaken the influence of the potential spurious correlations simultaneously. Extensive experiments on three real-world datasets have shown the effectiveness of the proposed approach by comparing it with a number of competitive baselines. Our implementation code is available at: https://github.com/RUCAIBox/CGKR.
Structural Relationship Representation Learning with Graph Embedding for Personalized Product Search
Authors: Shang Liu, Wanli Gu, Gao Cong, Fuzheng Zhang
Venue: Conference CIKM 2020
Year: 2020

To provide more accurate personalized product search (PPS) results, it is compulsory to go beyond modeling user-query-item interaction. Graph embedding techniques open the potential to integrate node information and topological structure information. Existing graph embedding enhanced PPS methods are mostly based on entity-relation-entity graph learning. In this work, we propose to consider structural relationship in users' product search scenario with graph embedding by latent representation learning. We argue that explicitly modeling the structural relationship in graph embedding is essential for more accurate PPS results. We propose a novel method, Graph embedding based Structural Relationship Representation Learning (GraphSRRL), which explicitly models the structural relationship in users-queries-products interaction. It combines three key conjunctive graph patterns to learn graph embedding for better PPS. In addition, GraphSRRL facilitates the learning of affinities between users (resp. queries or products) in the designed geometric operation in low-dimensional latent space. We conduct extensive experiments on four datasets to evaluate GraphSRRL for PPS. Experimental results show that GraphSRRL outperforms the state-of-the-art algorithm on real-world search datasets by at least 50.7% in term of Hit@10 and 48.7% in terms of NDCG@10.
Quantifying Query Ambiguity with Topic Distributions
Authors: Yuki Yano, Yukihiro Tagami, Akira Tajima
Venue: Conference CIKM 2016
Year: 2016

Query ambiguity is a useful metric for search engines to understand users' intents. Existing methods quantify query ambiguity by calculating an entropy of clicks. These methods assign each click to a one-hot vector corresponding to some mutually exclusive groups. However, they cannot incorporate non-obvious structures such as similarity among documents. In this paper, we propose a new approach for quantifying query ambiguity using topic distributions. We show that it is a natural extension of an existing entropy-based method. Further, we use our approach to achieve topic-based extensions of major existing entropy-based methods. Through an evaluation using e-commerce search logs combined with human judgments, our approach successfully extended existing entropy-based methods and improved the quality of query ambiguity measurements.
The Curious Case of IR Explainability: Explaining Document Scores within and across Ranking Models
Authors: Procheta Sen, Debasis Ganguly, Manisha Verma, Gareth J. F. Jones
Venue: Conference SIGIR 2020
Year: 2020

It is often useful for an IR practitioner to analyze the similarity function of an IR model, or for a non-technical search engine user to understand why a document was shown at a certain rank, in terms of the three fundamental aspects of a similarity function, namely the a) frequency of a term in a document, b) frequency of a term in a collection and c) the length of a document. We propose a general methodology of approximating an IR model as the coefficients of a linear function of these three fundamental aspects (and an additional aspect of semantic similarity between terms for neural models), which potentially can help IR practitioners to optimize the relative importance of each aspect on specific document collection and types of queries. Our analysis shows that the coefficients, which represent the relative importance of the three fundamental aspects, are useful to compare a model's different parametric instantiations or compare across different models.
LIRME: Locally Interpretable Ranking Model Explanation
Authors: Manisha Verma, Debasis Ganguly
Venue: Conference SIGIR 2019
Year: 2019

Information retrieval (IR) models often employ complex variations in term weights to compute an aggregated similarity score of a query-document pair. Treating IR models as black-boxes makes it difficult to understand or explain why certain documents are retrieved at top-ranks for a given query. Local explanation models have emerged as a popular means to understand individual predictions of classification models. However, there is no systematic investigation that learns to interpret IR models, which is in fact the core contribution of our work in this paper. We explore three sampling methods to train an explanation model and propose two metrics to evaluate explanations generated for an IR model. Our experiments reveal some interesting observations, namely that a) diversity in samples is important for training local explanation models, and b) the stability of a model is inversely proportional to the number of parameters used to explain the model.
Position-normalized click prediction in search advertising
Authors: Ye Chen, Tak W. Yan
Venue: Conference KDD 2012
Year: 2012

Click-through rate (CTR) prediction plays a central role in search advertising. One needs CTR estimates unbiased by positional effect in order for ad ranking, allocation, and pricing to be based upon ad relevance or quality in terms of click propensity. However, the observed click-through data has been confounded by positional bias, that is, users tend to click more on ads shown in higher positions than lower ones, regardless of the ad relevance. We describe a probabilistic factor model as a general principled approach to studying these exogenous and often overwhelming phenomena. The model is simple and linear in nature, while empirically justified by the advertising domain. Our experimental results with artificial and real-world sponsored search data show the soundness of the underlying model assumption, which in turn yields superior prediction accuracy.
A Difficulty-Aware Framework for Churn Prediction and Intervention in Games
Authors: Jiayu Li, Hongyu Lu, Chenyang Wang, Weizhi Ma, Min Zhang, Xiangyu Zhao, Wei Qi, Yiqun Liu, Shaoping Ma
Venue: Conference KDD 2021
Year: 2021

User's leaving from the system without further return, called user churn, is a severe negative signal in online games. Therefore, churn prediction and intervention are of great value for improving players' experiences and system performance. However, the problem has not been well-studied in the game scenario. Especially, some crucial factors, such as game difficulty, have not been considered for large-scale churn analysis. In this paper, a novel Difficulty-Aware Framework (DAF) for churn prediction and intervention is proposed. Firstly, a Difficulty Flow for each user is proposed, which is utilized to derive users' Personalized Perceived Difficulty during the game process. Then, a survival analysis modelD-Cox-Time is designed to model the Dynamic Influence of Perceived Difficulty on player churn intention. Finally, thePersonalized Perceived Difficulty ~(PPD) andDynamic Difficulty Influence ~(DDI) are incorporated to churn prediction and intervention. The proposed DAF framework has been specified in a real-world puzzle game as an example for churn prediction and intervention. Extensive offline experiments show significant improvements in churn prediction by introducing difficulty-related features. Besides, we conduct an online intervention system to adjust difficulty dynamically in the online game. A/B test results verify that the proposed intervention system enhances user retention and engagement significantly. To the best of our knowledge, it is the first framework in games that illustrates an in-depth understanding and leveraging dynamic and personalized perceived difficulty during game playing, which is easy to be integrated with various churn prediction and intervention models.