OpenRaiser/Intern-Atlas huggingface dataset:
SOGOU-2012-CRAWL: A Crawl of Search Results in the Sogou 2012 Chinese Query Log
Authors: Stewart Whiting, Joemon M. Jose, Omar Alonso
Venue: SIGIR
Year: 2016

In 2012, Sogou, a major Chinese web search engine released a large-scale query log containing 43.5M user interactions, including submitted queries and clicked web page search results. This query log offers a deep sample of queries over a two day period from 30th December 2011 to 1st January 2012. In August 2013, we identified 1.4M predominantly Chinese language unique search result URLs that were clicked at least three times in this query log. We crawled the HTML content of these URLs to construct the supplementary SOGOU-2012-CRAWL dataset, which we release in this work. A real large-scale query log with accompanying crawl such as this offers several opportunities for reproducible information retrieval (IR) research, including query classification, intent modelling and indexing strategy. In this paper we first detail the query log and crawl dataset construction and characteristics. Following this, to demonstrate potential applications we use the crawl to indicatively analyse various time-based patterns in web content and search behaviour. In particular, we study the distribution of language-independent date expressions in the crawled web content. Based on this, we propose a simple approach for modelling the past/present/future temporal intent of queries based on the date the query was submitted by the user, and the dates appearing in the clicked search results. We observe several prominent temporal patterns which may lead to novel time-aware IR approaches.
AOL's Named Entity Resolver: Solving Disambiguation via Document Strongly Connected Components and Ad-Hoc Edges Construction
Authors: Roni Wiener, Yonatan Ben-Simhon, Anna Chen
Venue: SIGIR
Year: 2016

Named Entity Disambiguation is the task of disambiguating named entity mentions in unstructured text and linking them to their corresponding entries in a large knowledge base such as Freebase. Practically, each text match in a given document should be mapped to the correct entity out of the corresponding entities in the knowledge base or none of them if no correct entity is found (Empty Entry). The case of an empty entry makes the problem at hand more complex, but by solving it, one can successfully cope with missing and erroneous data as well as unknown entities. In this work we present AOL's Named Entity Resolver which was designed to handle real life scenarios including empty entries. As part of the automated news analysis platform, it processes over 500K news articles a day, entities from each article are extracted and disambiguated. According to our experiments, AOL's resolver shows much better results in disambiguating entities mapped to Wikipedia or Freebase compared to industry leading products.
Retrievability: An Independent Evaluation Measure
Authors: Colin Wilkie
Venue: SIGIR
Year: 2016

Information Retrieval systems have traditionally been evaluated in terms of efficiency and performance. These aspects of retrieval systems, whilst very important, do not cover a crucial aspect of the system, the access it provides to the documents of the collection. Retrievability, a document centric evaluation measure, introduced by Azzopardi and Vinay, provides an alternative approach to evaluation [1]. Retrievability is the ease with which a document can be retrieved using a retrieval system. The more queries which retrieve the document, and the higher up the document is returned, the more retrievable it is. It can thus be used to describe how difficult it is to find documents in the collection given a particular configuration of a retrieval system. Unlike typical performance evaluations, performing a retrievability analysis can be done without recourse to relevancy judgements meaning there is no reliance on a test collection. This has major advantages when tuning a retrieval systems parameters as the tuning can be performed on the live collection.
What Makes a Query Temporally Sensitive?
Authors: Craig Willis, Garrick Sherman, Miles Efron
Venue: SIGIR
Year: 2016

This work takes an in-depth look at the factors that affect manual classifications of 'temporally sensitive' information needs. We use qualitative and quantitative techniques to analyze 660 topics from the Text Retrieval Conference (TREC) previously used in the experimental evaluation of temporal retrieval models. Regression analysis is used to identify factors in previous manual classifications. We explore potential problems with the previous classifications, considering principles and guidelines for future work on temporal retrieval models.
Modeling Document Novelty with Neural Tensor Network for Search Result Diversification
Authors: Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, Xueqi Cheng
Venue: SIGIR
Year: 2016

Search result diversification has attracted considerable attention as a means to tackle the ambiguous or multi-faceted information needs of users. One of the key problems in search result diversification is novelty, that is, how to measure the novelty of a candidate document with respect to other documents. In the heuristic approaches, the predefined document similarity functions are directly utilized for defining the novelty. In the learning approaches, the novelty is characterized based on a set of handcrafted features. Both the similarity functions and the features are difficult to manually design in real world due to the complexity of modeling the document novelty. In this paper, we propose to model the novelty of a document with a neural tensor network. Instead of manually defining the similarity functions or features, the new method automatically learns a nonlinear novelty function based on the preliminary representation of the candidate document and other documents. New diverse learning to rank models can be derived under the relational learning to rank framework. To determine the model parameters, loss functions are constructed and optimized with stochastic gradient descent. Extensive experiments on three public TREC datasets show that the new derived algorithms can significantly outperform the baselines, including the state-of-the-art relational learning to rank models.
Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System
Authors: Rui Yan, Yiping Song, Hua Wu
Venue: SIGIR
Year: 2016

To establish an automatic conversation system between humans and computers is regarded as one of the most hardcore problems in computer science, which involves interdisciplinary techniques in information retrieval, natural language processing, artificial intelligence, etc. The challenges lie in how to respond so as to maintain a relevant and continuous conversation with humans. Along with the prosperity of Web 2.0, we are now able to collect extremely massive conversational data, which are publicly available. It casts a great opportunity to launch automatic conversation systems. Owing to the diversity of Web resources, a retrieval-based conversation system will be able to find at least some responses from the massive repository for any user inputs. Given a human issued message, i.e., query, our system would provide a reply after adequate training and learning of how to respond. In this paper, we propose a retrieval-based conversation system with the deep learning-to-respond schema through a deep neural network framework driven by web data. The proposed model is general and unified for different conversation scenarios in open domain. We incorporate the impact of multiple data inputs, and formulate various features and factors with optimization into the deep learning framework. In the experiments, we investigate the effectiveness of the proposed deep neural network structures with better combinations of all different evidence. We demonstrate significant performance improvement against a series of standard and state-of-art baselines in terms of [email protected], MAP, nDCG, and MRR for conversational purposes.
Privacy-Preserving IR 2016: Differential Privacy, Search, and Social Media
Authors: Grace Hui Yang, Ian Soboroff, Li Xiong, Charles L. A. Clarke, Simson L. Garfinkel
Venue: SIGIR
Year: 2016

Due to lack of mature techniques in privacy-preserving information retrieval (IR), concerns about information privacy and security have become serious obstacles that prevent valuable user data to be used in IR research such as studies on query logs, social media, and medical record retrieval. In SIGIR 2014 and SIGIR 2015, we have run the privacy-preserving IR workshops exploring and understanding the privacy and security risks in information retrieval. This year, we continue the efforts of connecting the two disciplines of IR and privacy/security by organizing this workshop. We target on three themes, differential privacy and IR dataset release, privacy in search and browsing, and privacy in social media. The workshop includes panels with researchers from both fields on these three themes, as well as invite industry speakers for real-world challenges. The goals of this workshop include (1) bringing together the two research fields, and (2) yielding fruitful collaborations.
Question Answering with Knowledge Base, Web and Beyond
Authors: Wen-tau Yih, Hao Ma
Venue: SIGIR
Year: 2016

In this tutorial, we give the audience a coherent overview of the research of question answering (QA). We first introduce a variety of QA problems proposed by pioneer researchers and briefly describe the early efforts. By contrasting with the current research trend in this domain, the audience can easily comprehend what technical problems remain challenging and what the main breakthroughs and opportunities are during the past half century. For the rest of the tutorial, we select three categories of the QA problems that have recently attracted a great deal of attention in the research community, and present the tasks with the latest technical survey. We conclude the tutorial by discussing the new opportunities and future directions of QA research.
Rethinking the Cost of Information Search Behavior
Authors: Yinglong Zhang, Jacek Gwizdka
Venue: SIGIR
Year: 2016

In this paper, we present a cognitive-economic approach to examining the cost in information search. Unlike previous studies on economic models, we calculated the cost in information search based on participants' eye-tracking data as well as their behavioral data, such as query formulation, search task duration, SERP and web page visits. Using Principal Component Analysis (PCA), we explored a possible latent factor structure of variables representing the cost in information search. Our results indicated that the cost of information seeking could be associated with two distinct aspects of search, exploratory and validation processes.
Sampling Strategies and Active Learning for Volume Estimation
Authors: Haotian Zhang, Jimmy Lin, Gordon V. Cormack, Mark D. Smucker
Venue: SIGIR
Year: 2016

This paper tackles the challenge of accurately and efficiently estimating the number of relevant documents in a collection for a particular topic. One real-world application is estimating the volume of social media posts (e.g., tweets) pertaining to a topic, which is fundamental to tracking the popularity of politicians and brands, the potential sales of a product, etc. Our insight is to leverage active learning techniques to find all the "easy" documents, and then to use sampling techniques to infer the number of relevant documents in the residual collection. We propose a simple yet effective technique for determining this "switchover" point, which intuitively can be understood as the "knee" in an effort vs. recall gain curve, as well as alternative sampling strategies beyond the knee. We show on several TREC datasets and a collection of tweets that our best technique yields more accurate estimates (with the same effort) than several alternatives.
Discrete Collaborative Filtering
Authors: Hanwang Zhang, Fumin Shen, Wei Liu, Xiangnan He, Huanbo Luan, Tat-Seng Chua
Venue: SIGIR
Year: 2016

We address the efficiency problem of Collaborative Filtering (CF) by hashing users and items as latent vectors in the form of binary codes, so that user-item affinity can be efficiently calculated in a Hamming space. However, existing hashing methods for CF employ binary code learning procedures that most suffer from the challenging discrete constraints. Hence, those methods generally adopt a two-stage learning scheme composed of relaxed optimization via discarding the discrete constraints, followed by binary quantization. We argue that such a scheme will result in a large quantization loss, which especially compromises the performance of large-scale CF that resorts to longer binary codes. In this paper, we propose a principled CF hashing framework called Discrete Collaborative Filtering (DCF), which directly tackles the challenging discrete optimization that should have been treated adequately in hashing. The formulation of DCF has two advantages: 1) the Hamming similarity induced loss that preserves the intrinsic user-item similarity, and 2) the balanced and uncorrelated code constraints that yield compact yet informative binary codes. We devise a computationally efficient algorithm with a rigorous convergence proof of DCF. Through extensive experiments on several real-world benchmarks, we show that DCF consistently outperforms state-of-the-art CF hashing techniques, e.g, though using only 8 bits, DCF is even significantly better than other methods using 128 bits.
Leveraging Context-Free Grammar for Efficient Inverted Index Compression
Authors: Zhaohua Zhang, Jiancong Tong, Haibing Huang, Jin Liang, Tianlong Li, Rebecca J. Stones, Gang Wang, Xiaoguang Liu
Venue: SIGIR
Year: 2016

Large-scale search engines need to answer thousands of queries per second over billions of documents, which is typically done by querying a large inverted index. Many highly optimized integer encoding techniques are applied to compress the inverted index and reduce the query processing time. In this paper, we propose a new grammar-based inverted index compression scheme, which can improve the performance of both index compression and query processing.
Learning for Efficient Supervised Query Expansion via Two-stage Feature Selection
Authors: Zhiwei Zhang, Qifan Wang, Luo Si, Jianfeng Gao
Venue: SIGIR
Year: 2016

Query expansion (QE) is a well known technique to improve retrieval effectiveness, which expands original queries with extra terms that are predicted to be relevant. A recent trend in the literature is Supervised Query Expansion (SQE), where supervised learning is introduced to better select expansion terms. However, an important but neglected issue for SQE is its efficiency, as applying SQE in retrieval can be much more time-consuming than applying Unsupervised Query Expansion (UQE) algorithms. In this paper, we point out that the cost of SQE mainly comes from term feature extraction, and propose a Two-stage Feature Selection framework (TFS) to address this problem. The first stage is adaptive expansion decision, which determines if a query is suitable for SQE or not. For unsuitable queries, SQE is skipped and no term features are extracted at all, which reduces the most time cost. For those suitable queries, the second stage is cost constrained feature selection, which chooses a subset of effective yet inexpensive features for supervised learning. Extensive experiments on four corpora (including three academic and one industry corpus) show that our TFS framework can substantially reduce the time cost for SQE, while maintaining its effectiveness.
Anonymizing Query Logs by Differential Privacy
Authors: Sicong Zhang, Grace Hui Yang, Lisa Singh
Venue: SIGIR
Year: 2016

Query logs are valuable resources for Information Retrieval (IR) research. However, because they are also rich in private and personal information, the huge concern of leaking user privacy prevents query logs from being shared from the search companies to the broad research community. Bothered by the lack of good research data for years, the authors of this paper are motivated to explore ways to generate anonymized query logs that can still be effectively used to support the search task. We introduce a framework to anonymize query logs by differential privacy, the latest development in privacy research. The framework is empirically evaluated against multiple search algorithms on their retrieval utility, measured in standard IR evaluation metrics, using the anonymized logs. The experiments show that our framework is able to achieve a good balance between retrieval utility and privacy.
A Sequential Decision Formulation of the Interface Card Model for Interactive IR
Authors: Yinan Zhang, ChengXiang Zhai
Venue: SIGIR
Year: 2016

The Interface Card model is a promising new theoretical framework for modeling and optimizing interactive retrieval interfaces, but how to systematically instantiate it to solve concrete interface optimization problems remains an open challenge. We propose a novel formulation of the Interface Card model based on sequential decision theory, leading to a general framework for formal modeling of user states and stopping actions. The proposed framework naturally connects optimization of interactive retrieval with Markov Decision Processes and Partially Observable Markov Decision Processes, and enables the use of reinforcement learning algorithms for optimizing interactive retrieval interfaces. Simulation and user study experiments demonstrate the effectiveness of the proposed model in automatically adjusting the interface layout in adaptation to inferred user stopping tendencies in addition to user interaction and screen size.
SECC: A Novel Search Engine Interface with Live Chat Channel
Authors: Cheng Zhang, Peng Zhang, Jingfei Li, Dawei Song
Venue: SIGIR
Year: 2016

Traditional information retrieval systems rank documents according to their relevance to users' input queries. State of the art commercial search engines (SEs) train ranking models and suggest query refinements by exploiting collective intelligence implicitly using global users' query logs. However, they do not provide an explicit channel for users to communicate with each other in the search process. By asking or discussing with other users on the fly, a user could find relevant information more conveniently and gain a better search experience. In this paper, we present a demo of novel Search Engine with a live Chat Channel (SECC). SECC can group users automatically based on their input queries and allow them to communicate with each other in real time through a chat interface.
Temporal Query Intent Disambiguation using Time-Series Data
Authors: Yue Zhao, Claudia Hauff
Venue: SIGIR
Year: 2016

Understanding temporal intents behind users' queries is essential to meet users' time-related information needs. In order to classify queries according to their temporal intent (e.g. Past or Future), we explore the usage of time-series data derived from Wikipedia page views as a feature source. While existing works leverage either proprietary search engine query logs or highly processed and aggregated data (such as Google Trends) for this purpose, we investigate the utility of a freely available data source for this purpose. Our experiments on the NTCIR-12 Temporalia-2 dataset show, that Wikipedia pageview-based time-series data can significantly improve the disambiguation of temporal intents for specific types of queries, in particular those without temporal expressions present in the query string.
How Much Novelty is Relevant?: It Depends on Your Curiosity
Authors: Pengfei Zhao, Dik Lun Lee
Venue: SIGIR
Year: 2016

Traditional recommendation systems (RS's) aim to recommend items that are relevant to the user's interest. Unfortunately, the recommended items will soon become too familiar to the user and hence fail to arouse her interest. Discovery-oriented recommendation systems (DORS's) complement accuracy with "discover utilities" (DU's) such as novelty and diversity and optimize the tradeoff between the DU's and accuracy of the recommendations. Unfortunately, DORS's ignore an important fact that different users have different appetites for DU's. That is, highly curious users can accept highly novel and diversified recommendations whereas conservative users would behave in the opposite manner. In this paper, we propose a curiosity-based recommendation system (CBRS) framework which generates recommendations with a personalized amount of DU's to fit the user's curiosity level. The major contribution of this paper is a computational model of user curiosity, called Probabilistic Curiosity Model (PCM), which is based on the curiosity arousal theory and Wundt curve in psychology research. In PCM, we model a user's curiosity with a curiosity distribution function learnt from the user's access history and compute a curiousness score for each item representing how curious the user is about the item. CBRS then selects items which are both relevant and have high curiousness score, bounded by the constraint that the amount of DU's fits the user's DU appetite. We use joint optimization and co-factorization approaches to incorporate the curiosity signal into the recommendations. Extensive experiments have been performed to evaluate the performance of CBRS against the baselines using a music dataset from last.fm. The results show that compared to the baselines CBRS not only provides more personalized recommendations that adapt to the user's curiosity level but also improves the recommendation accuracy.
HIA 2016: The 2nd International Workshop on Heterogeneous Information Access at SIGIR 2016
Authors: Ke Zhou, Yiqun Liu, Roger Jie Luo, Joemon M. Jose
Venue: SIGIR
Year: 2016

Information access is becoming increasingly heterogeneous. Especially when the user's information need is for exploratory purpose, returning a set of diverse results from different resources could benefit the user. For example, when a user is planning a trip to China, retrieving and showing results from vertical search engines like travel, flight information, map and Q2A sites can satisfy the user's rich and diverse information need. This heterogeneous search paradigm is useful in many contexts and brings many new challenges.
Modelling User Search Behaviour Based on Process
Authors: Mengdie Zhuang
Venue: SIGIR
Year: 2016

Typically, interactive information retrieval (IIR) system evaluations assess search processes and outcomes using a combination of two types of measures: 1. user perception (e.g. users? attitudes of the search experience and outcome); 2. user behaviour (e.g. time and counts of various actions including mouse and keyboard clicks). In general, we assume that they are indicative of the search outcomes (e.g. performance, opinion). However, search is a dynamic process with changing outcomes. Therefore, neither measure solely provides a holistic way of evaluating search. On one hand, user behaviour measures are only descriptive of the outcome, and are not interpretive of the process. That is to say, they lack the rationale behind why those behaviours occurred. Another problem is that some mental activities may not reflect on user behaviour [1]. The challenge with logfiles, which contain behaviour data, is the voluminous number of data points and the need to find a reliable approach to define groups or sets based on behavioural patterns. Not all users are alike and nor do they all take the same approach to search for the same things, as evidenced by the TREC, INEX and CLEF interactive tracks. On the other hand, user perception measures are acquired in such small samples that do not scale to large participant populations, and are rarely measured constantly due to the laborious and time consuming data collection methods (e.g. questionnaire, interview). Moreover, not enough emphasis is put on assessing the reliability of individual perception measures, and the wide usage of likert-type scale limits the interpretation of answers. For a holistic understanding of the search process, we need both perception and behaviour measures. I speculate that user behaviour may predict user perception, and thus we should be able to analyse large-scale files for a greater understanding of the likely human responses.