OpenRaiser/Intern-Atlas huggingface dataset:
Fair and balanced: learning to present news stories
Authors: Amr Ahmed, Choon Hui Teo, S. V. N. Vishwanathan, Alexander J. Smola
Venue: WSDM
Year: 2012

Relevance, diversity and personalization are key issues when presenting content which is apt to pique a user's interest. This is particularly true when presenting an engaging set of news stories. In this paper we propose an efficient algorithm for selecting a small subset of relevant articles from a streaming news corpus. It offers three key pieces of improvement over past work: 1) It is based on a detailed model of a user's viewing behavior which does not require explicit feedback. 2) We use the notion of submodularity to estimate the propensity of interacting with content. This improves over the classical context independent relevance ranking algorithms. Unlike existing methods, we learn the submodular function from the data. 3) We present an efficient online algorithm which can be adapted for personalization, story adaptation, and factorization models. Experiments show that our system yields a significant improvement over a retrieval system deployed in production.
Harmony and dissonance: organizing the people's voices on political controversies
Authors: Rawia Awadallah, Maya Ramanath, Gerhard Weikum
Venue: WSDM
Year: 2012

The wikileaks documents about the death of Osama Bin Laden and the debates about the economic crisis in Greece and other European countries are some of the controversial topics being played on the news everyday. Each of these topics has many different aspects, and there is no absolute, simple truth in answering questions such as: should the EU guarantee the financial stability of each member country, or should the countries themselves be solely responsible? To understand the landscape of opinions, it would be helpful to know which politician or other stakeholder takes which position - support or opposition - on these aspects of controversial topics.
Collaborative ranking
Authors: Suhrid Balakrishnan, Sumit Chopra
Venue: WSDM
Year: 2012

Typical recommender systems use the root mean squared error (RMSE) between the predicted and actual ratings as the evaluation metric. We argue that RMSE is not an optimal choice for this task, especially when we will only recommend a few (top) items to any user. Instead, we propose using a ranking metric, namely normalized discounted cumulative gain (NDCG), as a better evaluation metric for this task. Borrowing ideas from the learning to rank community for web search, we propose novel models which approximately optimize NDCG for the recommendation task. Our models are essentially variations on matrix factorization models where we also additionally learn the features associated with the users and the items for the ranking task. Experimental results on a number of standard collaborative filtering data sets validate our claims. The results also show the accuracy and efficiency of our models and the benefits of learning features for ranking.
Large-scale analysis of individual and task differences in search result page examination strategies
Authors: Georg Buscher, Ryen W. White, Susan T. Dumais, Jeff Huang
Venue: WSDM
Year: 2012

Understanding the impact of individual and task differences on search result page examination strategies is important in developing improved search engines. Characterizing these effects using query and click data alone is common but insufficient since they provide an incomplete picture of result examination behavior. Cursor- or gaze-tracking studies reveal richer interaction patterns but are often done in small-scale laboratory settings. In this paper we leverage large-scale rich behavioral log data in a naturalistic setting. We examine queries, clicks, cursor movements, scrolling, and text highlighting for millions of queries on the Bing commercial search engine to better understand the impact of user, task, and user-task interactions on user behavior on search result pages (SERPs). By clustering users based on cursor features, we identify individual, task, and user-task differences in how users examine results which are similar to those observed in small-scale studies. Our findings have implications for developing search support for behaviorally-similar searcher cohorts, modeling search behavior, and designing search systems that leverage implicit feedback.
Daily deals: prediction, social diffusion, and reputational ramifications
Authors: John W. Byers, Michael Mitzenmacher, Georgios Zervas
Venue: WSDM
Year: 2012

Daily deal sites have become the latest Internet sensation, providing discounted offers to customers for restaurants, ticketed events, services, and other items. We begin by undertaking a study of the economics of daily deals on the web, based on a dataset we compiled by monitoring Groupon and LivingSocial sales in 20 large cities over several months. We use this dataset to characterize deal purchases; glean insights about operational strategies of these firms; and evaluate customers' sensitivity to factors such as price, deal scheduling, and limited inventory. We then marry our daily deals dataset with additional datasets we compiled from Facebook and Yelp users to study the interplay between social networks and daily deal sites. First, by studying user activity on Facebook while a deal is running, we provide evidence that daily deal sites benefit from significant word-of-mouth effects during sales events, consistent with results predicted by cascade models. Second, we consider the effects of daily deals on the longer-term reputation of merchants, based on their Yelp reviews before and after they run a daily deal. Our analysis shows that while the number of reviews increases significantly due to daily deals, average rating scores from reviewers who mention daily deals are 10% lower than scores of their peers on average.
A noise-aware click model for web search
Authors: Weizhu Chen, Dong Wang, Yuchen Zhang, Zheng Chen, Adish Singla, Qiang Yang
Venue: WSDM
Year: 2012

Recent advances in click model have established it as an attractive approach to infer document relevance. Most of these advances consider the user click/skip behavior as binary events but neglect the context in which a click happens. We show that real click behavior in industrial search engines is often noisy and not always a good indication of relevance. For a considerable percentage of clicks, users select what turn out to be irrelevant documents and these clicks should not be directly used as evidence for relevance inference. Thus in this paper, we put forward an observation that the relevance indication degree of a click is not a constant, but can be differentiated by user preferences and the context in which the user makes her click decision. In particular, to interpret the click behavior discriminatingly, we propose a Noise-aware Click Model (NCM) by characterizing the noise degree of a click, which indicates the quality of the click for inferring relevance. Specifically, the lower the click noise is, the more important the click is in its role for relevance inference. To verify the necessity of explicitly accounting for the uninformative noise in a user click, we conducted experiments on a billion-scale dataset. Extensive experimental results demonstrate that as compared with two state-of-the-art click models in Web Search, NCM can better interpret user click behavior and achieve significant improvements in terms of both perplexity and NDCG.
Efficient misbehaving user detection in online video chat services
Authors: Hanqiang Cheng, Yu-Li Liang, Xinyu Xing, Xue Liu, Richard Han, Qin Lv, Shivakant Mishra
Venue: WSDM
Year: 2012

Online video chat services, such as Chatroulette, Omegle, and vChatter are becoming increasingly popular and have attracted millions of users. One critical problem encountered in such applications is the presence of misbehaving users ("flashers") and obscene content. Automatically filtering out obscene content from these systems in an efficient manner poses a difficult challenge. This paper presents a novel Fine-Grained Cascaded (FGC) classification solution that significantly speeds up the compute-intensive process of classifying misbehaving users by dividing image feature extraction into multiple stages and filtering out easily classified images in earlier stages, thus saving unnecessary computation costs of feature extraction in later stages. Our work is further enhanced by integrating new webcam-related contextual information (illumination and color) into the classification process, and a 2-stage soft margin SVM algorithm for combining multiple features. Evaluation results using real-world data set obtained from Chatroulette show that the proposed FGC based classification solution significantly outperforms state-of-the-art techniques.
Exploration and discovery of user-generated content in large information spaces
Authors: Luca Chiarandini
Venue: WSDM
Year: 2012

The accumulation of large collections of social media data poses new challenges for the design of exploratory experiences, such as when a user browses through a collection to discover content (e.g. exploring photo collections, network of friends, etc). Cardinality and characteristics of the set, together with volatility of the information, resulting from fast and continuous creation, deletion and updating of entries, trigger novel research questions. In this context, we plan to investigate and contribute to the data analysis, and user interface design of exploratory experiences. The proposed approach is an iterative process where analysis and design phases are performed in cycles. The long-term vision is to understand the underlying reasoning in order to be able to automatically replicate it.
WebSets: extracting sets of entities from the web using unsupervised information extraction
Authors: Bhavana Bharat Dalvi, William W. Cohen, Jamie Callan
Venue: WSDM
Year: 2012

We describe a open-domain information extraction method for extracting concept-instance pairs from an HTML corpus. Most earlier approaches to this problem rely on combining clusters of distributionally similar terms and concept-instance pairs obtained with Hearst patterns. In contrast, our method relies on a novel approach for clustering terms found in HTML tables, and then assigning concept names to these clusters using Hearst patterns. The method can be efficiently applied to a large corpus, and experimental results on several datasets show that our method can accurately extract large numbers of concept-instance pairs.
Computational advertising: leveraging user interaction & contextual factors for improved ad relevance & targeting
Authors: Kushal S. Dave
Venue: WSDM
Year: 2012

Computational advertising refers to finding the most relevant ads matching a particular context on the web. The core problem attacked in computational advertising CA is of the match making between the ads and the context. My research work aims at leveraging various user interaction, ad and advertiser related information and contextual information for improving the relevance, ranking and targeting of ads. The research work focuses on the identification of various factors that contribute in retrieving and ranking the most relevant set of ads that match best with the context. Specifically, information associated with the user, publisher and advertiser is leveraged for this purpose.
Exploring social influence via posterior effect of word-of-mouth recommendations
Authors: Junming Huang, Xueqi Cheng, Huawei Shen, Tao Zhou, Xiaolong Jin
Venue: WSDM
Year: 2012

Word-of-mouth has proven an effective strategy for promoting products through social relations. Particularly, existing studies have convincingly demonstrated that word-of-mouth recommendations can boost users' prior expectation and hence encourage them to adopt a certain innovation, such as buying a book or watching a movie. However, less attention has been paid to studying the posterior effect of word-of-mouth recommendations, i.e., whether or not word-of-mouth recommendations can influence users' posterior evaluation on the products or services recommended to them, the answer to which is critical to estimating user satisfaction when proposing a word-of-mouth marketing strategy. In order to fill this gap, in this paper we empirically study the above issue and verify that word-of-mouth recommendations are strongly associated with users' posterior evaluation. Through elaborately designed statistical hypothesis tests we prove the causality that word-of-mouth recommendations directly prompt the posterior evaluation of receivers. Finally, we propose a method for investigating users' social influence, namely, their ability to affect followers' posterior evaluation via word-of-mouth recommendations, by examining the number of their followers and their sensitivity of discovering good items. The experimental results on real datasets show that our method can successfully identify 78% influential friends with strong social influence.
No search result left behind: branching behavior with browser tabs
Authors: Jeff Huang, Thomas Lin, Ryen W. White
Venue: WSDM
Year: 2012

Today's Web browsers allow users to open links in new windows or tabs. This action, which we call 'branching', is sometimes performed on search results when the user plans to eventually visit multiple results. We detect branching behavior on a large commercial search engine with a client-side script on the results page. Two-fifths of all users spawned new tabs on search results in the timeframe of our study; branching usage varied with different query types and vertical. Both branching and backtracking are viable methods for visiting multiple search results. To understand user search strategies, we treat multiple result clicks following a query as ordered events to understand user search strategies. Users branching in a query are more likely to click search results from top to bottom, while users who backtrack are less likely to do so; this is especially true for queries involving more than two clicks. These findings inform an experiment in which we take a popular click model and modify it to account for the differing user behavior when branching. By understanding that users continue examining search results before viewing a branched result, we can improve the click model for branching queries.
Selecting actions for resource-bounded information extraction using reinforcement learning
Authors: Pallika H. Kanani, Andrew McCallum
Venue: WSDM
Year: 2012

Given a database with missing or uncertain content, our goal is to correct and fill the database by extracting specific information from a large corpus such as the Web, and to do so under resource limitations. We formulate the information gathering task as a series of choices among alternative, resource-consuming actions and use reinforcement learning to select the best action at each time step. We use temporal difference q-learning method to train the function that selects these actions, and compare it to an online, error-driven algorithm called SampleRank. We present a system that finds information such as email, job title and department affiliation for the faculty at our university, and show that the learning-based approach accomplishes this task efficiently under a limited action budget. Our evaluations show that we can obtain 92.4% of the final F1, by only using 14.3% of all possible actions.
Learning to rank with multi-aspect relevance for vertical search
Authors: Changsung Kang, Xuanhui Wang, Yi Chang, Belle L. Tseng
Venue: WSDM
Year: 2012

Many vertical search tasks such as local search focus on specific domains. The meaning of relevance in these verticals is domain-specific and usually consists of multiple well-defined aspects (e.g., text matching and distance in local search). Thus the overall relevance between a query and a document is a tradeoff between multiple relevance aspects. Such a tradeoff can vary for different types of queries or in different contexts. In this paper, we explore these vertical-specific aspects in the learning to rank setting. We propose a novel formulation in which the relevance between a query and a document is assessed with respect to each aspect, forming the multi-aspect relevance. In order to compute a ranking function, we study two types of learning-based approaches to estimate the tradeoff between these relevance aspects: a label aggregation method and a model aggregation method. Since there are only a few aspects, a minimal amount of training data is needed to learn the tradeoff. We conduct both offline and online test experiments on a local search engine and the experimental results show that our proposed multi-aspect relevance formulation is very promising. The two types of aggregation methods perform more effectively than a set of baseline methods including a conventional learning to rank method.
Characterizing web content, user interests, and search behavior by reading level and topic
Authors: Jin Young Kim, Kevyn Collins-Thompson, Paul N. Bennett, Susan T. Dumais
Venue: WSDM
Year: 2012

A user's expertise or ability to understand a document on a given topic is an important aspect of that document's relevance. However, this aspect has not been well-explored in information retrieval systems, especially those at Web scale where the great diversity of content, users, and tasks presents an especially challenging search problem. To help improve our modeling and understanding of this diversity, we apply automatic text classifiers, based on reading difficulty and topic prediction, to estimate a novel type of profile for important entities in Web search -- users, websites, and queries. These profiles capture topic and reading level distributions, which we then use in conjunction with search log data to characterize and compare different entities.
Tapping into knowledge base for concept feedback: leveraging conceptnet to improve search results for difficult queries
Authors: Alexander Kotov, ChengXiang Zhai
Venue: WSDM
Year: 2012

Query expansion is an important and commonly used technique for improving Web search results. Existing methods for query expansion have mostly relied on global or local analysis of document collection, click-through data, or simple ontologies such as WordNet. In this paper, we present the results of a systematic study of the methods leveraging the ConceptNet knowledge base, an emerging new Web resource, for query expansion. Specifically, we focus on the methods leveraging ConceptNet to improve the search results for poorly performing (or difficult) queries. Unlike other lexico-semantic resources, such as WordNet and Wikipedia, which have been extensively studied in the past, ConceptNet features a graph-based representation model of commonsense knowledge, in which the terms are conceptually related through rich relational ontology. Such representation structure enables complex, multi-step inferences between the concepts, which can be applied to query expansion. We first demonstrate through simulation experiments that expanding queries with the related concepts from ConceptNet has great potential for improving the search results for difficult queries. We then propose and study several supervised and unsupervised methods for selecting the concepts from ConceptNet for automatic query expansion. The experimental results on multiple data sets indicate that the proposed methods can effectively leverage ConceptNet to improve the retrieval performance of difficult queries both when used in isolation as well as in combination with pseudo-relevance feedback.
Multi-relational matrix factorization using bayesian personalized ranking for social network data
Authors: Artus Krohn-Grimberghe, Lucas Drumond, Christoph Freudenthaler, Lars Schmidt-Thieme
Venue: WSDM
Year: 2012

A key element of the social networks on the internet such as Facebook and Flickr is that they encourage users to create connections between themselves, other users and objects.
Evaluating search in personal social media collections
Authors: Chia-Jung Lee, W. Bruce Croft, Jin Young Kim
Venue: WSDM
Year: 2012

The prevalence of social media applications is generating potentially large personal archives of posts, tweets, and other communications. The existence of these archives creates a need for search tools, which can be seen as an extension of current desktop search services. Little is currently known about the best search techniques for personal archives of social data, because of the difficulty of creating test collections. In this paper, we describe how test collections for personal social data can be created by using games to collect queries. We then compare a range of retrieval models that exploit the semi-structured nature of social data. Our results show that a mixture of language models with field distribution estimation can be effective for this type of data, with certain fields, such as the name of the poster, being particularly important. We also analyze the properties of the queries that were generated by users with two versions of the games.
Machine learning for query-document matching in search
Authors: Hang Li, Jun Xu
Venue: WSDM
Year: 2012

In web search, relevance is one of the most important factors to meet users' satisfaction, and the success of a web search engine heavily depends on its performance on relevance. It has been observed that many hard cases in search relevance are due to term mismatch between query and documnt (e.g., query 'ny times' does not match well with document only containing 'new york times'), and thus it is not exaggerated to say that dealing with mismatch between query and document is one of the most critical research problems in web search. Recently researchers have spent significant effort to address the grand challenge. The major approach is to conduct more query and document understanding, and perform better matching between enriched query and document representations. With the availability of large amount of log data and advanced machine learning techniques, this becomes more feasible and significant progress has been made recently. In this tutorial, we will give a systematic and detailed presentation on newly developed machine learning technologies for query document matching in search. We will focus on the fundamental problems, as well as the novel solutions for query document matching at word form level, word sense level, topic level, and structure level. We will talk about novel technologies about query spelling error correction [3, 13], query rewriting [1, 4, 6, 7], query classification [2], topic modeling of documents [5, 9], query document matching [8, 10, 11, 12], and query document-title translation. The ideas and solutions introduced in this tutorial may motivate industrial practitioners to turn the research fruits into product reality. The summary of the state-of-the-art methods and the discussions on the technical issues in this tutorial may stimulate academic researchers to find new research directions and solutions. Matching between query and document is not limited to search, and similar problems can be observed at online advertisement, recommendation system, and other applications, as matching between objects from two spaces. The technologies we introduce can be generalized into more general machine learning techniques, which we call learning to match.