OpenRaiser/Intern-Atlas huggingface dataset:
Relative confidence sampling for efficient on-line ranker evaluation
Authors: Masrour Zoghi, Shimon Whiteson, Maarten de Rijke, Rémi Munos
Venue: WSDM
Year: 2014

A key challenge in information retrieval is that of on-line ranker evaluation: determining which one of a finite set of rankers performs the best in expectation on the basis of user clicks on presented document lists. When the presented lists are constructed using interleaved comparison methods, which interleave lists proposed by two different candidate rankers, then the problem of minimizing the total regret accumulated while evaluating the rankers can be formalized as a K-armed dueling bandits problem. In this paper, we propose a new method called relative confidence sampling (RCS) that aims to reduce cumulative regret by being less conservative than existing methods in eliminating rankers from contention. In addition, we present an empirical comparison between RCS and two state-of-the-art methods, relative upper confidence bound and SAVAGE. The results demonstrate that RCS can substantially outperform these alternatives on several large learning to rank datasets.
The Information Life of Social Networks
Authors: Lada A. Adamic
Venue: WSDM
Year: 2015

Vast amounts of information are propagated in online social networks such as Facebook. This talk will describe several studies characterizing how information diffuses over social ties, from the growth of individual cascades to the predictability of their eventual size. It will also characterize the diffusion of specific kinds of information, including rumors, memes, and social movements.
Big Data: New Paradigm or "Sound and Fury, Signifying Nothing"?
Authors: Andrei Z. Broder, Lada A. Adamic, Michael J. Franklin, Maarten de Rijke, Eric P. Xing, Kai Yu
Venue: WSDM
Year: 2015

The Gartner's 2014 Hype Cycle released last August moves Big Data technology from the Peak of Inflated Expectations to the beginning of the Trough of Disillusionment when interest starts to wane as reality does not live up to previous promises. As the hype is starting to dissipate it is worth asking what Big Data (however defined) means from a scientific perspective: Did the emergence of gigantic corpora exposed the limits of classical information retrieval and data mining and led to new concepts and challenges, the way say, the study of electromagnetism showed the limits of Newtonian mechanics and led to Relativity Theory, or is it all just "sound and fury, signifying nothing", simply a matter of scaling up well understood technologies? To answer this question, we have assembled a distinguished panel of eminent scientists, from both Industry and Academia: Lada Adamic (Facebook), Michael Franklin (University of California at Berkeley), Maarten de Rijke (University of Amsterdam), Eric Xing (Carnegie Mellon University), and Kai Yu (Baidu) will share their point of view and take questions from the moderator and the audience.
Scalability and Efficiency Challenges in Large-Scale Web Search Engines
Authors: Berkant Barla Cambazoglu, Ricardo Baeza-Yates
Venue: WSDM
Year: 2015

Commercial web search engines need to process thousands of queries every second and provide responses to user queries within a few hundred milliseconds. As a consequence of these tight performance constraints, search engines construct and maintain very large computing infrastructures for crawling the Web, indexing discovered pages, and processing user queries. The scalability and efficiency of these infrastructures require careful performance optimizations in every major component of the search engine. This tutorial aims to provide a fairly comprehensive overview of the scalability and efficiency challenges in large-scale web search engines. In particular, the tutorial provides an in-depth architectural overview of a web search engine, mainly focusing on the web crawling, indexing, and query processing components. The scalability and efficiency issues encountered in the above-mentioned components are presented at four different granularities: at the level of a single computer, a cluster of computers, a single data center, and a multi-center search engine. The tutorial also points at the open research problems and provides recommendations to researchers who are new to the field.
Regressing Towards Simpler Prediction Systems
Authors: Tushar Chandra
Venue: WSDM
Year: 2015

This talk will focus on our experience in managing the complexity of Sibyl, a large scale machine learning system that is widely used within Google. We believe that a large fraction of the challenges faced by Sibyl are inherent to large scale production machine learning and that other production systems are likely to encounter them as well [1]. Thus, these challenges present interesting opportunities for future research. The Sibyl system is complex for a number of reasons. We have learnt that a complete end-to-end machine learning solution has to have subsystems to address a variety of different needs: data ingestion, data analysis, data verification, experimentation, model analysis, model serving, configuration, data transformations, support for different kinds of loss functions and modeling, machine learning algorithm implementations, etc. Machine learning algorithms themselves constitute a relatively small fraction of the overall system.
SimApp: A Framework for Detecting Similar Mobile Applications by Online Kernel Learning
Authors: Ning Chen, Steven C. H. Hoi, Shaohua Li, Xiaokui Xiao
Venue: WSDM
Year: 2015

With the popularity of smart phones and mobile devices, the number of mobile applications (a.k.a. "apps") has been growing rapidly. Detecting semantically similar apps from a large pool of apps is a basic and important problem, as it is beneficial for various applications, such as app recommendation, app search, etc. However, there is no systematic and comprehensive work so far that focuses on addressing this problem. In order to fill this gap, in this paper, we explore multi-modal heterogeneous data in app markets (e.g., description text, images, user reviews, etc.), and present "SimApp" -- a novel framework for detecting similar apps using machine learning. Specifically, it consists of two stages: (i) a variety of kernel functions are constructed to measure app similarity for each modality of data; and (ii) an online kernel learning algorithm is proposed to learn the optimal combination of similarity functions of multiple modalities. We conduct an extensive set of experiments on a real-world dataset crawled from Google Play to evaluate SimApp, from which the encouraging results demonstrate that SimApp is effective and promising.
An Approach to the Problem of Annotation of Research Publications
Authors: Ekaterina Chernyak
Venue: WSDM
Year: 2015

An approach to multiple labelling research papers is explored. We develop techniques for annotating/labeling research papers in informatics and computer sciences with key phrases taken from the ACM Computing Classification System. The techniques utilize a phrase-to-text relevance measure so that only those phrases that are most relevant go to the annotation. Three phrase-to-text relevance measures are experimentally compared in this setting. The measures are: (a) cosine relevance score between conventional vector space representations of the texts coded with tf-idf weighting; (b) popular characteristic of probability of term generation BM25; and (c) an in-house characteristic of conditional probability of symbols averaged over matching fragments in suffix trees representing texts and phrases, CPAMF. In an experiment conducted over a set of texts published in journals of the ACM and manually annotated by their authors, CPAMF outperforms both the cosine measure and BM25 by a wide margin.
WorkerRank: Using Employer Implicit Judgements to Infer Worker Reputation
Authors: Maria Daltayanni, Luca de Alfaro, Panagiotis Papadimitriou
Venue: WSDM
Year: 2015

In online labor marketplaces two parties are involved; employers and workers. An employer posts a job in the marketplace to receive applications from interested workers. After evaluating the match to the job, the employer hires one (or more workers) to accomplish the job via an online contract. At the end of the contract, the employer can provide his worker with some rating that becomes visible in the worker online profile. This form of explicit feedback guides future hiring decisions, since it is indicative of worker true ability. In this paper, first we discuss some of the shortcomings of the existing reputation systems that are based on the end-of-contract ratings. Then we propose a new reputation mechanism that uses Bayesian updates to combine employer implicit feedback signals in a link-analysis approach. The new system addresses the shortcomings of existing approaches, while yielding better signal for the worker quality towards hiring decision.
Diluted Treatment Effect Estimation for Trigger Analysis in Online Controlled Experiments
Authors: Alex Deng, Victor Hu
Venue: WSDM
Year: 2015

Online controlled experiments, also called A/B testing, is playing a central role in many data-driven web-facing companies. It is well known and intuitively obvious to many practitioners that when testing a feature with low coverage, analyzing all data collected without zooming into the part that could be affected by the treatment often leads to under-powered hypothesis testing. A common practice is to use triggered analysis. To estimate the overall treatment effect, certain dilution formula is then applied to translate the estimated effect in triggered analysis back to the original all up population. In this paper, we discuss two different types of trigger analyses. We derive correct dilution formulas and show for a set of widely used metrics, namely ratio metrics, correctly deriving and applying those dilution formulas are not trivial. We observe many practitioners in this industry are often applying approximate formulas or even wrong formulas when doing effect dilution calculation. To deal with that, instead of estimating trigger treatment effect followed by effect translation using dilution formula, we aim at combining these two steps into one streamlined analysis, producing more accurate estimation of overall treatment effect together with even higher statistical power than a triggered analysis. The approach we propose in this paper is intuitive, easy to apply and general enough for all types of triggered analyses and all types of metrics.
Will This Paper Increase Your h-index?: Scientific Impact Prediction
Authors: Yuxiao Dong, Reid A. Johnson, Nitesh V. Chawla
Venue: WSDM
Year: 2015

Scientific impact plays a central role in the evaluation of the output of scholars, departments, and institutions. A widely used measure of scientific impact is citations, with a growing body of literature focused on predicting the number of citations obtained by any given publication. The effectiveness of such predictions, however, is fundamentally limited by the power-law distribution of citations, whereby publications with few citations are extremely common and publications with many citations are relatively rare. Given this limitation, in this work we instead address a related question asked by many academic researchers in the course of writing a paper, namely: "Will this paper increase my h-index?" Using a real academic dataset with over 1.7 million authors, 2 million papers, and 8 million citation relationships from the premier online academic service ArnetMiner, we formalize a novel scientific impact prediction problem to examine several factors that can drive a paper to increase the primary author's h-index. We find that the researcher's authority on the publication topic and the venue in which the paper is published are crucial factors to the increase of the primary author's h-index, while the topic popularity and the co-authors' h-indices are of surprisingly little relevance. By leveraging relevant factors, we find a greater than 87.5% potential predictability for whether a paper will contribute to an author's h-index within five years. As a further experiment, we generate a self-prediction for this paper, estimating that there is a 76% probability that it will contribute to the h-index of the co-author with the highest current h-index in five years. We conclude that our findings on the quantification of scientific impact can help researchers to expand their influence and more effectively leverage their position of "standing on the shoulders of giants."
Engagement Periodicity in Search Engine Usage: Analysis and its Application to Search Quality Evaluation
Authors: Alexey Drutsa, Gleb Gusev, Pavel Serdyukov
Venue: WSDM
Year: 2015

Nowadays, billions of people use the Web in connection with their daily needs. A significant part of the needs are constituted by search tasks that are usually addressed by search engines. Thus, daily search needs result in regular user engagement with a search engine. User engagement with web sites and services was studied in various aspects, but there appear to be no studies of its regularity and periodicity. In this paper, we studied periodicity of the user engagement with a popular search engine through applying spectrum analysis to temporal sequences of different engagement metrics. We found periodicity patterns of user engagement and revealed classes of users whose periodicity patterns do not change over a long period of time. In addition, we used the spectrum series as metrics to evaluate search quality.
Making Sense of Big Data with the Berkeley Data Analytics Stack
Authors: Michael J. Franklin
Venue: WSDM
Year: 2015

The Berkeley AMPLab is creating a new approach to data analytics. Launching in early 2011, the lab aims to seamlessly integrate the three main resources available for making sense of data at scale: Algorithms (machine learning and statistical techniques), Machines (in the form of scalable clusters and elastic cloud computing), and People (both individually as analysts and in crowds). The lab is realizing its ideas through the development of a freely-available Open Source software stack called BDAS: the Berkeley Data Analytics Stack. In the four years the lab has been in operation, we've released major components of BDAS. Several of these components have gained significant traction in industry and elsewhere: the Mesos cluster resource manager, the Spark in-memory computation framework, and the Shark query processing system. BDAS features prominently in many industry discussions of the future of the Big Data analytics ecosystem -- a rare degree of impact for an ongoing academic project. Given this initial success, the lab is continuing on its research path, moving "up the stack" to better integrate and support advanced analytics and to make people a full-fledged resource for making sense of data. In this talk, I'll first outline the motivation and insights behind our research approach and describe how we have organized to address the cross-disciplinary nature of Big Data challenges. I will then describe the current state of BDAS with an emphasis on our newest efforts, including some or all of: the GraphX graph processing system, the Velox and MLBase machine learning platforms, and the SampleClean framework for hybrid human/computer data cleaning. Finally I will present our current views of how all the pieces will fit together to form a system that can adaptively bring the right resources to bear on a given data-driven question to meet time, cost and quality requirements throughout the analytics lifecycle.
DL-WSDM'15: Workshop on Deep Learning for Web Search and Data Mining
Authors: Bin Gao, Jiang Bian
Venue: WSDM
Year: 2015

In recent years, deep learning has been a very hot topic in the machine learning community. It has brought break-through results in image classification and speech recognition. Most recently, researchers have also got many promising results in natural language processing using deep learning techniques. As machine learning techniques are widely used in the Web search and data mining applications, many researchers and practitioners are studying the possibility of applying the recently-developed deep learning techniques into these applications. Some of them have made very promising progress, and thus it is a good time to hold a workshop to discuss and share the problems and progress in using deep learning techniques to improve Web search and data mining tasks.
WSDM'15 Workshop Summary / Scalable Data Analytics: Theory and Applications
Authors: Kaizhu Huang, Haiqin Yang, Irwin King, Michael R. Lyu
Venue: WSDM
Year: 2015

The SDA workshop at WSDM 2015 is the fifth International Workshop on Scalable Data Analytics, following the previous four workshops of SDA respectively held at IEEE Big Data 2013, PAKDD 2014, IEEE Big Data 2014, and IEEE ICDM 2014. This series of workshops aims to provide professionals, researchers, and technologists with a single forum where they can discuss and share the state-of-the-art theories and applications of scalable data analytics technologies. In particular, in the era of information explosion, the scientific, biomedical, and engineering research communities are undergoing a profound transformation where discoveries and innovations increasingly rely on massive amounts of data. The characteristics of volume, velocity, variety and veracity originated in the massive big data then bring challenges to current data analytics techniques. The focus of the fifth SDA is to discuss how we can scale up data analytics techniques for modeling and analyzing big data from various domains.
Global Optimization for Display Ad
Authors: Rong Ji
Venue: WSDM
Year: 2015

Online display advertisement has been examined by numerous studies. Most online display ad systems take the greedy approach, namely they display, for each user, the set of ads that match best with the user's interests. One shortcoming of the greedy approach is that it does not take into account the budget limitation of each advertiser. As a result, we often observed that some ads are popular and match with the interests of millions of users; but due to the budget restriction, these ads can only be presented by a limited times, leading to a suboptimal performance. To make our point clear, let's consider a simple case where we only have two advertisers (i.e. A and B), and two users (i.e. a and b). We assume that both advertisers have only a budget of one display. We further assume that user a is interested in both ads even though he is more interested in ad A, while user b is only interested in ad A. Now, if we take the greedy approach, we will always present ad A to user a; as a result, if user a comes before user b, we will have no appropriate ad to be displayed for user b. On the other hand, if we can take into account the budget limitation of both advertisers, a better approach is to present ad B to user a and ad A to user b. This simple example motivates us to develop the global optimization approach for online display advertisement that explicitly take into account the budget limitation of advertisers when deciding the ad presentation for individual users.
Chronological Scientific Information Recommendation via Supervised Dynamic Topic Modeling
Authors: Zhuoren Jiang
Venue: WSDM
Year: 2015

Scientific information recommendation is crucial to assist scholars for their researches. Citation recommendation is an important field of scientific recommendation. Traditional approaches ignore the chronological nature of the citation recommendation task. In this study, I propose the "Chronological Citation Recommendation," which assumes initial user information need could shift while they are looking for the papers in different time slices. Specifically, I employed a supervised dynamic topic model to characterize the content "time-varying" dynamics and constructed a novel heterogeneous graph that contains dynamic topic-based information, time-decay citation information and word-based information. I applied different meta-paths for different ranking hypotheses, which carried different types of information for citation recommendation in different time slices along with information need shifting. I plan to generate the final "Chronological Citation Recommendation" rankings by feature integration using Learning to Rank. "Chronological Citation Recommendation" will recommend time-series ranking lists based on initial user textual information need. Preliminary experiments on the ACM corpus show that chronological citation recommendation will significantly improve the citation recommendation performance.
Learning from User Interactions
Authors: Thorsten Joachims
Venue: WSDM
Year: 2015

The ability to learn from user interactions can give systems access to unprecedented amounts of world knowledge. This is already evident in search engines, recommender systems, and electronic commerce, and other applications are likely to follow in the near future (e.g., education, smart homes). More generally, the ability to learn from user interactions promises pathways for solving knowledge-intensive tasks ranging from natural language understanding to autonomous robotics. Learning from user interactions, however, means learning from data that does not necessarily fit the assumptions of the standard machine learning models. Since interaction data consists of the choices that humans make, it has to be interpreted with respect to how humans make decisions, which is influenced by the decision context and constraints like human motivation and human abilities.
Mining Groups Stability in Ubiquitous and Social Environments: Communities, Classes and Clusters
Authors: Mark Kibanov
Venue: WSDM
Year: 2015

Ubiquitous Computing is an emerging research area of computer science. Similarly, social network analysis and mining became very important in the last years. We aim to combine these two research areas to explore the nature of processes happening around users. The presented research focuses on exploring and analyzing different groups of persons or entities (communities, clusters and classes), their stability and semantics. An example of ubiquitous social data are social networks captured during scientific conferences using face-to-face RFID proximity tags. Another example of ubiquitous data is crowd-generated environmental sensor data. In this paper we generalize various problems connected to these and further datasets and consider them as a task for measuring group stability. Group stability can be used to improve state-of-the-art methods to analyze data. We also aim to improve the performance of different data mining algorithms, eg. by better handling of data with a skewed density distribution. We describe significant results some experiments that show how the presented approach can be applied and discuss the planned experiments.
Delayed-Dynamic-Selective (DDS) Prediction for Reducing Extreme Tail Latency in Web Search
Authors: Saehoon Kim, Yuxiong He, Seung-won Hwang, Sameh Elnikety, Seungjin Choi
Venue: WSDM
Year: 2015

A commercial web search engine shards its index among many servers, and therefore the response time of a search query is dominated by the slowest server that processes the query. Prior approaches target improving responsiveness by reducing the tail latency of an individual search server. They predict query execution time, and if a query is predicted to be long-running, it runs in parallel, otherwise it runs sequentially. These approaches are, however, not accurate enough for reducing a high tail latency when responses are aggregated from many servers because this requires each server to reduce a substantially higher tail latency (e.g., the 99.99th-percentile), which we call extreme tail latency.