OpenRaiser/Intern-Atlas huggingface dataset:
Neural Networks for Information Retrieval
Authors: Tom Kenter, Alexey Borisov, Christophe Van Gysel, Mostafa Dehghani, Maarten de Rijke, Bhaskar Mitra
Venue: SIGIR
Year: 2017

Machine learning plays a role in many aspects of modern IR systems, and deep learning is applied in all of them. The fast pace of modern-day research has given rise to many different approaches for many different IR problems. The amount of information available can be overwhelming both for junior students and for experienced researchers looking for new research topics and directions. Additionally, it is interesting to see what key insights into IR problems the new technologies are able to give us. The aim of this full-day tutorial is to give a clear overview of current tried-and-trusted neural methods in IR and how they benefit IR research. It covers key architectures, as well as the most promising future directions.
A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms
Authors: Benjamin Kille, Andreas Lommatzsch, Frank Hopfgartner, Martha A. Larson, Arjen P. de Vries
Venue: SIGIR
Year: 2017

Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today's researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests. We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by "replaying" it locally, and ORP makes it possible to take this evaluation "live" in a living lab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in the development of the dataset and ORP.
Personalized PageRank in Uncertain Graphs with Mutually Exclusive Edges
Authors: Jung Hyun Kim, Mao-Lin Li, K. Selçuk Candan, Maria Luisa Sapino
Venue: SIGIR
Year: 2017

Measures of node ranking, such as personalized PageRank, are utilized in many web and social-network based prediction and recommendation applications. Despite their effectiveness when the underlying graph is certain, however, these measures become difficult to apply in the presence of uncertainties, as they are not designed for graphs that include uncertain information, such as edges that mutually exclude each other. While there are several ways to naively extend existing techniques (such as trying to encode uncertainties as edge weights or computing all possible scenarios), as we discuss in this paper, these either lead to large degrees of errors or are very expensive to compute, as the number of possible worlds can grow exponentially with the amount of uncertainty. To tackle with this challenge, in this paper, we propose an efficient Uncertain Personalized PageRank (UPPR) algorithm to approximately compute personalized PageRank values on an uncertain graph with edge uncertainties. UPPR avoids enumeration of all possible worlds, yet it is able to achieve comparable accuracy by carefully encoding edge uncertainties in a data structure that leads to fast approximations. Experimental results show that UPPR is very efficient in terms of execution time and its accuracy is comparable or better than more costly alternatives.
A Task-oriented Search Engine for Evidence-based Medicine
Authors: Bevan Koopman, Guido Zuccon, Jack Russell
Venue: SIGIR
Year: 2017

Evidence-based medicine (EBM) is the practice of making clinical decisions based on rigorous scientific evidence. EBM relies on effective access to peer-reviewed literature - a task hampered by both the exponential growth of medical literature and a lack of efficient and effective means of searching and presenting this literature. This paper describes a search engine specifically designed for searching medical literature for the purpose of EBM and in a clinical decision support setting.
Crowdsourced App Review Manipulation
Authors: Shanshan Li, James Caverlee, Wei Niu, Parisa Kaghazgaran
Venue: SIGIR
Year: 2017

With the rapid adoption of smartphones worldwide and the reliance on app marketplaces to discover new apps, these marketplaces are critical for connecting users with apps. And yet, the user reviews and ratings on these marketplaces may be strategically targeted by app developers. We investigate the use of crowdsourcing platforms to manipulate app reviews. We find that (i) apps targeted by crowdsourcing platforms are rated significantly higher on average than other apps; (ii) the reviews themselves arrive in bursts; (iii) app reviewers tend to repeat themselves by relying on some standard repeated text; and (iv) apps by the same developer tend to share a more similar language model: if one app has been targeted, it is likely that many of the other apps from the same developer have also been targeted.
The Impact of Linkage Methods in Hierarchical Clustering for Active Learning to Rank
Authors: Ziming Li, Maarten de Rijke
Venue: SIGIR
Year: 2017

Document ranking is a central problem in many areas, including information retrieval and recommendation. The goal of learning to rank is to automatically create ranking models from training data. The performance of ranking models is strongly affected by the quality and quantity of training data. Collecting large scale training samples with relevance labels involves human labor which is time-consuming and expensive. Selective sampling and active learning techniques have been developed and proven effective in addressing this problem. However, most active methods do not scale well and need to rebuild the model after selected samples are added to the previous training set. We propose a sampling method which selects a set of instances and labels the full set only once before training the ranking model. Our method is based on hierarchical agglomerative clustering (average linkage) and we also report the performance of other linkage criteria that measure the distance between two clusters of query-document pairs. Another difference from previous hierarchical clustering is that we cluster the instances belonging to the same query, which usually outperforms the baselines.
SPOT: Selecting occuPations frOm Trajectories
Authors: Peipei Li, Junjie Yao, Liping Wang, Xuemin Lin
Venue: SIGIR
Year: 2017

With the pervasive availability of smart devices, billions of users' trajectories are recorded and collected. The aggregated human behaviors reveal users' interests and characteristics, becoming invaluable to reflect their demographic preference, i.e., gender, age, marital status and even personality, occupation. Occupation profiling from trajectory data is an attractive option for advertisement targeting and other applications, without severe privacy concerns. However, it carries great difficulties in sparsity and vagueness.
Personalized Itinerary Recommendation with Queuing Time Awareness
Authors: Kwan Hui Lim, Jeffrey Chan, Shanika Karunasekera, Christopher Leckie
Venue: SIGIR
Year: 2017

Personalized itinerary recommendation is a complex and time-consuming problem, due to the need to recommend popular attractions that are aligned to the interest preferences of a tourist, and to plan these attraction visits as an itinerary that has to be completed within a specific time limit. Furthermore, many existing itinerary recommendation systems do not automatically determine and consider queuing times at attractions in the recommended itinerary, which varies based on the time of visit to the attraction, e.g., longer queuing times at peak hours. To solve these challenges, we propose the PersQ algorithm for recommending personalized itineraries that take into consideration attraction popularity, user interests and queuing times. We also implement a framework that utilizes geo-tagged photos to derive attraction popularity, user interests and queuing times, which PersQ uses to recommend personalized and queue-aware itineraries. We demonstrate the effectiveness of PersQ in the context of five major theme parks, based on a Flickr dataset spanning nine years. Experimental results show that PersQ outperforms various state-of-the-art baselines, in terms of various queuing-time related metrics, itinerary popularity, user interest alignment, recall, precision and F1-score.
Visual Pool: A Tool to Visualize and Interact with the Pooling Method
Authors: Aldo Lipani, Mihai Lupu, Allan Hanbury
Venue: SIGIR
Year: 2017

Every year more than 25 test collections are built among the main Information Retrieval (IR) evaluation campaigns. They are extremely important in IR because they become the evaluation praxis for the forthcoming years. Test collections are built mostly using the pooling method. The main advantage of this method is that it drastically reduces the number of documents to be judged. It does so at the cost of introducing biases, which are sometimes aggravated by non optimal configuration. In this paper we develop a novel visualization technique for the pooling method, and integrate it in a demo application named Visual Pool. This demo application enables the user to interact with the pooling method with ease, and develops visual hints in order to analyze existing test collections, and build better ones.
Deep Learning for Extreme Multi-label Text Classification
Authors: Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, Yiming Yang
Venue: SIGIR
Year: 2017

Extreme multi-label text classification (XMTC) refers to the problem of assigning to each document its most relevant subset of class labels from an extremely large label collection, where the number of labels could reach hundreds of thousands or millions. The huge label space raises research challenges such as data sparsity and scalability. Significant progress has been made in recent years by the development of new machine learning methods, such as tree induction with large-margin partitions of the instance spaces and label-vector embedding in the target space. However, deep learning has not been explored for XMTC, despite its big successes in other related areas. This paper presents the first attempt at applying deep learning to XMTC, with a family of new Convolutional Neural Network (CNN) models which are tailored for multi-label classification in particular. With a comparative evaluation of 7 state-of-the-art methods on 6 benchmark datasets where the number of labels is up to 670,000, we show that the proposed CNN approach successfully scaled to the largest datasets, and consistently produced the best or the second best results on all the datasets. On the Wikipedia dataset with over 2 million documents and 500,000 labels in particular, it outperformed the second best method by 11.7%~15.3% in [email protected] and by 11.5%~11.7% in [email protected] for K = 1,3,5.
RankEval: An Evaluation and Analysis Framework for Learning-to-Rank Solutions
Authors: Claudio Lucchese, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Salvatore Trani
Venue: SIGIR
Year: 2017

In this demo paper we propose RankEval, an open-source tool for the analysis and evaluation of Learning-to-Rank (LtR) models based on ensembles of regression trees. Gradient Boosted Regression Trees (GBRT) is a flexible statistical learning technique for classification and regression at the state of the art for training effective LtR solutions. Indeed, the success of GBRT fostered the development of several open-source LtR libraries targeting efficiency of the learning phase and effectiveness of the resulting models. However, these libraries offer only very limited help for the tuning and evaluation of the trained models. In addition, the implementations provided for even the most traditional IR evaluation metrics differ from library to library, thus making the objective evaluation and comparison between trained models a difficult task. RankEval addresses these issues by providing a common ground for LtR libraries that offers useful and interoperable tools for a comprehensive comparison and in-depth analysis of ranking models.
X-DART: Blending Dropout and Pruning for Efficient Learning to Rank
Authors: Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Salvatore Trani
Venue: SIGIR
Year: 2017

In this paper we propose X-DART, a new Learning to Rank algorithm focusing on the training of robust and compact ranking models. Motivated from the observation that the last trees of MART models impact the prediction of only a few instances of the training set, we borrow from the DART algorithm the dropout strategy consisting in temporarily dropping some of the trees from the ensemble while new weak learners are trained. However, differently from this algorithm we drop permanently these trees on the basis of smart choices driven by accuracy measured on the validation set. Experiments conducted on publicly available datasets shows that X-DART outperforms DART in training models providing the same effectiveness by employing up to 40% less trees.
Mail Search: It's Getting Personal!
Authors: Yoelle Maarek
Venue: SIGIR
Year: 2017

Web Mail has significantly changed in the last decade. It keeps growing with 90% of its traffic being generated by automated scripts or "machines", [1]. At the same time, major mail services offer more and more free storage, ranging from 15GB for Gmail and Outlook.com to 1TB for Yahoo mail. As a result, we keep accumulating messages in our inbox, rarely deleting (and sometimes not even opening) many, [2]. Our inbox has become a big store mixing important information, such as e-tickets or bills, together with newsletters or promotions, from which we forgot to unsubscribe. Search is therefore a critical mechanism in order to retrieve the specific messages we need. Unfortunately, search in mail is far from being as trusted (and used) as in the Web today. Everything is personal and often private, from the content of the mailbox, to the search strategies, users' needs and queries, thus making traditional Web search techniques inapplicable "as is". Failure is evident when we can't find a message that we remember having read, and this increases our frustration. Most mail search services return sorted-by-time results in order for us to scan results chronologically and increase our perception of perfect recall. At the same time, the ranking mechanism drops less relevant results, in order to prevent them from being ranked first if recent. So in order to increase a (false) perception of recall, these systems actually hurt recall! Ranking results by mail-specific relevance would actually increase search success, [3] yet it is not widely adopted, with the exceptions of Inbox by Gmail and Yahoo Mail that show a few relevant results on top of traditional ranked-by-time results [4]. In addition, many of us still struggle with expressing our needs, typically issuing very short queries, like in the early days of the Web [5]. In this talk, I will first highlight the key characteristics of mail search and how they differ from Web search, in terms of searchers' needs and behavior [2,5]. I will then present recent progress in mail ranking [3,4] as well as in query assistance tools [5,6]. Finally, I will discuss directions for future research, and the need to revisit mail search and invent search mechanisms specifically tailored to the personal data store that our inbox has become.
Efficient & Effective Selective Query Rewriting with Efficiency Predictions
Authors: Craig MacDonald, Nicola Tonellotto, Iadh Ounis
Venue: SIGIR
Year: 2017

To enhance effectiveness, a user's query can be rewritten internally by the search engine in many ways, for example by applying proximity, or by expanding the query with related terms. However, approaches that benefit effectiveness often have a negative impact on efficiency, which has impacts upon the user satisfaction, if the query is excessively slow. In this paper, we propose a novel framework for using the predicted execution time of various query rewritings to select between alternatives on a per-query basis, in a manner that ensures both effectiveness and efficiency. In particular, we propose the prediction of the execution time of ephemeral (e.g., proximity) posting lists generated from uni-gram inverted index posting lists, which are used in establishing the permissible query rewriting alternatives that may execute in the allowed time. Experiments examining both the effectiveness and efficiency of the proposed approach demonstrate that a 49% decrease in mean response time (and 62% decrease in 95th-percentile response time) can be attained without significantly hindering the effectiveness of the search engine.
Find Shoes Like These
Authors: Hideyuki Maeda
Venue: SIGIR
Year: 2017

We present an Euclidean embedding image representation, which serves to rank auction item images through wide range of semantic similarity spectrum, in the order of the relevance to the given query image much more effective than the baseline method in terms of a graded relevance measure. Our method uses three stream deep convolutional siamese networks to learn a distance metric and we leverage search query logs of an auction item search of the largest auction service in Japan. Unlike previous approaches, we define the inter-image relevance on the basis of user queries in the logs used to search each auction item, which enables us to acquire the image representation preserving the features concerning user intents in real e-commerce world.
CitySearcher: A City Search Engine For Interests
Authors: Mohamed Abdel Maksoud, Gaurav Pandey, Shuaiqiang Wang
Venue: SIGIR
Year: 2017

We introduce CitySearcher, a vertical search engine that searches for cities when queried for an interest. Generally in search engines, utilization of semantics between words is favorable for performance improvement. Even though ambiguous query words have multiple semantic meanings, search engines can return diversified results to satisfy different users' information needs. But for CitySearcher, mismatched semantic relationships can lead to extremely unsatisfactory results. For example, the city Sale would incorrectly rank high for the interest shopping because of semantic interpretations of the words. Thus in our system, the main challenge is to eliminate the mismatched semantic relationships resulting from the side effect of the semantic models. In the previous case, we aim to ignore the semantics of a city's name which is not indicative of the city's characteristics. In CitySearcher, we use word2vec, a very popular word embedding technique to estimate the semantics of the words and create the initial ranks of the cities. To reduce the effect of the mismatched semantic relationships, we generate a set of features for learning based on a novel clustering-based method. With the generated features, we then utilize learning to rank algorithms to rerank the cities for return. We use the English version of Wikivoyage dataset for evaluation of our system, where we sample a very small dataset for training. Experimental results demonstrate the performance gain of our system over various standard retrieval techniques.
A Comparative Live Evaluation of Multileaving Methods on a Commercial cQA Search
Authors: Tomohiro Manabe, Akiomi Nishida, Makoto P. Kato, Takehiro Yamamoto, Sumio Fujita
Venue: SIGIR
Year: 2017

We present one of the world's first attempts to examine the feasibility of multileaving evaluation of document rankings on a large scale commercial community Question Answering (cQA) service. As a natural enhancement of interleaving evaluation, multileaving merges more than two input rankings into one and measures the search user satisfaction of each input ranking on the basis of user clicks on the multileaved ranking. We evaluated the adequateness of two major multileaving methods, team draft multileaving (TDM) and optimized multileaving (OM), proposing their practical implementation for live services. Our experimental results demonstrated that multileaving methods could precisely evaluate the effectiveness of five rankings with different quality by using clicks from real users. Moreover, we concluded that OM is more efficient than TDM by observing that most of the evaluation results with OM converged after showing multileaved rankings around 40,000 times and an in-depth analysis of their characteristics.
Deep Collaborative Filtering Approaches for Context-Aware Venue Recommendation
Authors: Jarana Manotumruksa
Venue: SIGIR
Year: 2017

In recent years, vast amounts of user-generated data have being created on Location-Based Social Networks (LBSNs) such as Yelp and Foursquare. Making effective personalised venue suggestions to users based on their preferences and surrounding context is a challenging task. Context-Aware Venue Recommendation (CAVR) is an emerging topic that has gained a lot of attention from researchers, where context can be the user's current location for example. Matrix Factorisation (MF) is one of the most popular collaborative filtering-based techniques, which can be used to predict a user's rating on venues by exploiting explicit feedback (e.g. users' ratings on venues). However, such explicit feedback may not be available, particularly for inactive users, while implicit feedback is easier to obtain from LBSNs as it does not require the users to explicitly express their satisfaction with the venues. In addition, the MF-based approaches usually suffer from the sparsity problem where users/venues have very few rating, hindering the prediction accuracy. Although previous works on user-venue rating prediction have proposed to alleviate the sparsity problem by leveraging user-generated data such as social information from LBSNs, research that investigates the usefulness of Deep Neural Network algorithms (DNN) in alleviating the sparsity problem for CAVR remains untouched or partially studied.
Cross-Language Question Re-Ranking
Authors: Giovanni Da San Martino, Salvatore Romeo, Alberto Barrón-Cedeño, Shafiq R. Joty, Lluís Màrquez, Alessandro Moschitti, Preslav Nakov
Venue: SIGIR
Year: 2017

We study how to find relevant questions in community forums when the language of the new questions is different from that of the existing questions in the forum. In particular, we explore the Arabic-English language pair. We compare a kernel-based system with a feed-forward neural network in a scenario where a large parallel corpus is available for training a machine translation system, bilingual dictionaries, and cross-language word embeddings. We observe that both approaches degrade the performance of the system when working on the translated text, especially the kernel-based system, which depends heavily on a syntactic kernel. We address this issue using a cross-language tree kernel, which compares the original Arabic tree to the English trees of the related questions. We show that this kernel almost closes the performance gap with respect to the monolingual system. On the neural network side, we use the parallel corpus to train cross-language embeddings, which we then use to represent the Arabic input and the English related questions in the same space. The results also improve to close to those of the monolingual neural network. Overall, the kernel system shows a better performance compared to the neural network in all cases.