OpenRaiser/Intern-Atlas huggingface dataset:
The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation
Authors: Tetsuya Sakai
Venue: SIGIR
Year: 2017

Using classical statistical significance tests, researchers can only discuss P(D+|H), the probability of observing the data D at hand or something more extreme, under the assumption that the hypothesis H is true (i.e., the p-value). But what we usually want is P(H|D), the probability that a hypothesis is true, given the data. If we use Bayesian statistics with state-of-the-art Markov Chain Monte Carlo (MCMC) methods for obtaining posterior distributions, this is no longer a problem. That is, instead of the classical p-values and 95% confidence intervals, which are often misinterpreted respectively as "probability that the hypothesis is (in)correct" and "probability that the true parameter value drops within the interval is 95%," we can easily obtain P(H|D) and credible intervals which represent exactly the above. Moreover, with Bayesian tests, we can easily handle virtually any hypothesis, not just "equality of means," and obtain an Expected A Posteriori (EAP) value of any statistic that we are interested in. We provide simple tools to encourage the IR community to take up paired and unpaired Bayesian tests for comparing two systems. Using a variety of TREC and NTCIR data, we compare P(H|D) with p-values, credible intervals with confidence intervals, and Bayesian EAP effect sizes with classical ones. Our results show that (a) p-values and confidence intervals can respectively be regarded as approximations of what we really want, namely, P(H|D) and credible intervals; and (b) sample effect sizes from classical significance tests can differ considerably from the Bayesian EAP effect sizes, which suggests that the former can be poor estimates of population effect sizes. For both paired and unpaired tests, we propose that the IR community report the EAP, the credible interval, and the probability of hypothesis being true, not only for the raw difference in means but also for the effect size in terms of Glass's Δ.
Gaussian Embeddings for Collaborative Filtering
Authors: Ludovic Dos Santos, Benjamin Piwowarski, Patrick Gallinari
Venue: SIGIR
Year: 2017

Most collaborative filtering systems, such as matrix factorization, use vector representations for items and users. Those representations are deterministic, and do not allow modeling the uncertainty of the learned representation, which can be useful when a user has a small number of rated items (cold start), or when there is conflicting information about the behavior of a user or the ratings of an item. In this paper, we leverage recent works in learning Gaussian embeddings for the recommendation task. We show that this model performs well on three representative collections (Yahoo, Yelp and MovieLens) and analyze learned representations.
On Application of Learning to Rank for E-Commerce Search
Authors: Shubhra Kanti Karmaker Santu, Parikshit Sondhi, ChengXiang Zhai
Venue: SIGIR
Year: 2017

E-Commerce (E-Com) search is an emerging important new application of information retrieval. Learning to Rank (LETOR) is a general effective strategy for optimizing search engines, and is thus also a key technology for E-Com search. While the use of LETOR for web search has been well studied, its use for E-Com search has not yet been well explored. In this paper, we discuss the practical challenges in applying learning to rank methods to E-Com search, including the challenges in feature representation, obtaining reliable relevance judgments, and optimally exploiting multiple user feedback signals such as click rates, add-to-cart ratios, order rates, and revenue. We study these new challenges using experiments on industry data sets and report several interesting findings that can provide guidance on how to optimally apply LETOR to E-Com search: First, popularity-based features defined solely on product items are very useful and LETOR methods were able to effectively optimize their combination with relevance-based features. Second, query attribute sparsity raises challenges for LETOR, and selecting features to reduce/avoid sparsity is beneficial. Third, while crowdsourcing is often useful for obtaining relevance judgments for Web search, it does not work as well for E-Com search due to difficulty in eliciting sufficiently fine grained relevance judgments. Finally, among the multiple feedback signals, the order rate is found to be the most robust training objective, followed by click rate, while add-to-cart ratio seems least robust, suggesting that an effective practical strategy may be to initially use click rates for training and gradually shift to using order rates as they become available.
Improving Exploratory Search Experience through Hierarchical Knowledge Graphs
Authors: Bahareh Sarrafzadeh, Edward Lank
Venue: SIGIR
Year: 2017

In information retrieval and information visualization, hierarchies are a common tool to structure information into topics or facets, and network visualizations such as knowledge graphs link related concepts within a domain. In this paper, we explore a multi-layer extension to knowledge graphs, hierarchical knowledge graphs (HKGs), that combines hierarchical and network visualizations into a unified data representation. Through interaction logs, we show that HKGs preserve the benefits of single-layer knowledge graphs at conveying domain knowledge while incorporating the sense-making advantages of hierarchies for knowledge seeking tasks. Specially, this paper describes our algorithm to construct these visualizations, analyzes interaction logs to quantitatively demonstrate performance parity with networks and performance advantages over hierarchies, and synthesizes data from interaction logs, inter- views, and thinkalouds on a testbed data set to demonstrate the utility of the unified hierarchy+network structure in our HKGs.
Leveraging Cross-Network Information for Graph Sparsification in Influence Maximization
Authors: Xiao Shen, Fu-Lai Chung, Sitong Mao
Venue: SIGIR
Year: 2017

When tackling large-scale influence maximization (IM) problem, one effective strategy is to employ graph sparsification as a pre-processing step, by removing a fraction of edges to make original networks become more concise and tractable for the task. In this work, a Cross-Network Graph Sparsification (CNGS) model is proposed to leverage the influence backbone knowledge pre-detected in a source network to predict and remove the edges least likely to contribute to the influence propagation in the target networks. Experimental results demonstrate that conducting graph sparsification by the proposed CNGS model can obtain a good trade-off between efficiency and effectiveness of IM, i.e., existing IM greedy algorithms can run more efficiently, while the loss of influence spread can be made as small as possible in the sparse target networks.
Classification by Retrieval: Binarizing Data and Classifiers
Authors: Fumin Shen, Yadong Mu, Yang Yang, Wei Liu, Li Liu, Jingkuan Song, Heng Tao Shen
Venue: SIGIR
Year: 2017

This paper proposes a generic formulation that significantly expedites the training and deployment of image classification models, particularly under the scenarios of many image categories and high feature dimensions. As the core idea, our method represents both the images and learned classifiers using binary hash codes, which are simultaneously learned from the training data. Classifying an image thereby reduces to retrieving its nearest class codes in the Hamming space. Specifically, we formulate multiclass image classification as an optimization problem over binary variables. The optimization alternatingly proceeds over the binary classifiers and image hash codes. Profiting from the special property of binary codes, we show that the sub-problems can be efficiently solved through either a binary quadratic program (BQP) or a linear program. In particular, for attacking the BQP problem, we propose a novel bit-flipping procedure which enjoys high efficacy and a local optimality guarantee. Our formulation supports a large family of empirical loss functions and is, in specific, instantiated by exponential and linear losses. Comprehensive evaluations are conducted on several representative image benchmarks. The experiments consistently exhibit reduced computational and memory complexities of model training and deployment, without sacrificing classification accuracy.
Document Expansion Using External Collections
Authors: Garrick Sherman, Miles Efron
Venue: SIGIR
Year: 2017

Document expansion has been shown to improve the effectiveness of information retrieval systems by augmenting documents' term probability estimates with those of similar documents, producing higher quality document representations. We propose a method to further improve document models by utilizing external collections as part of the document expansion process. Our approach is based on relevance modeling, a popular form of pseudo-relevance feedback; however, where relevance modeling is concerned with query expansion, we are concerned with document expansion. Our experiments demonstrate that the proposed model improves ad-hoc document retrieval effectiveness on a variety of corpus types, with a particular benefit on more heterogeneous collections of documents.
Building Test Collections: An Interactive Guide for Students and Others Without Their Own Evaluation Conference Series
Authors: Ian Soboroff
Venue: SIGIR
Year: 2017

This is a full-day tutorial on building and validating test collections. The intended audience is advanced students who nd themselves in need of a test collection, or actually in the process of building a test collection, to support their own research. Not everyone can talk TREC, CLEF, INEX, or NTCIR into running a track to build the collection you need. The goal of this tutorial is to lay out issues, procedures, pitfalls, and practical advice.
Smart Media Generation System for Broadcasting Contents
Authors: Jeong Woo Son, Wonjoo Park, Sang-Yun Lee, Jinwoo Kim, Sun-Joong Kim
Venue: SIGIR
Year: 2017

Broadcasting contents are the most plausiable resources for services with video contents. Even though we already have huge amount of produced broadcasting contents, there rarely exists a system to analyze and generate information on broadcasting contents to support content retrieval and recommendation services. This paper proposes a new system for this purpose. In the proposed system, a broadcasting content is segmented into semantic units, scenes, based on its multiple characteristics. The proposed system analyzes scenes and generates their keywords, topics, and stories. Connections among scenes are automatically establishing based on shared keywords, similar topics, and consistency in stories. To support operators, the proposed system offers two tools: Scene Studio and SceneViz. We prepare several Open APIs in the proposed system to provide information and connections for service providers. The feasibility of the proposed system is shown with numerical evaluations on the qualities of generated information. We also introduce two video clip services implemented with our system.
Social Media Advertisement Outreach: Learning the Role of Aesthetics
Authors: Avikalp Srivastava, Madhav Datt, Jaikrishna Chaparala, Shubham Mangla, Priyadarshi Patnaik
Venue: SIGIR
Year: 2017

Corporations spend millions of dollars on developing creative image based promotional content to advertise to their user-base on platforms like Twitter. Our paper is an initial study, where we propose a novel method to evaluate and improve outreach of promotional images from corporations on Twitter, based purely on their describable aesthetic attributes. Existing works in aesthetic based image analysis exclusively focus on the attributes of digital photographs, and are not applicable to advertisements due to the influences of inherent content and context based biases on outreach. Our paper identifies broad categories of biases affecting such images, describes a method for normalizing outreach scores to eliminate effects of those biases, which enables us to subsequently examine the effects of certain handcrafted describable aesthetic features on image outreach. Optimizing on the features resulting from this research is a simple method for corporations to complement their existing marketing strategy to gain significant improvement in user engagement on social media for promotional images.
Learning to Rank Using Localized Geometric Mean Metrics
Authors: Yuxin Su, Irwin King, Michael R. Lyu
Venue: SIGIR
Year: 2017

Many learning-to-rank~(LtR) algorithms focus on query-independent model, in which query and document do not lie in the same feature space, and the rankers rely on the feature ensemble about query-document pair instead of the similarity between query instance and documents. However, existing algorithms do not consider local structures in query-document feature space, and are fragile to irrelevant noise features. In this paper, we propose a novel Riemannian metric learning algorithm to capture the local structures and develop a robust LtR algorithm. First, we design a concept called ideal candidate document to introduce metric learning algorithm to query-independent model. Previous metric learning algorithms aiming to find an optimal metric space are only suitable for query-dependent model, in which query instance and documents belong to the same feature space and the similarity is directly computed from the metric space. Then we extend the new and extremely fast global Geometric Mean Metric Learning (GMML) algorithm to develop a localized GMML, namely L-GMML. Based on the combination of local learned metrics, we employ the popular Normalized Discounted Cumulative Gain~(NDCG) scorer and Weighted Approximate Rank Pairwise~(WARP) loss to optimize the ideal candidate document for each query candidate set. Finally, we can quickly evaluate all candidates via the similarity between the ideal candidate document and other candidates. By leveraging the ability of metric learning algorithms to describe the complex structural information, our approach gives us a principled and efficient way to perform LtR tasks. The experiments on real-world datasets demonstrate that our proposed L-GMML algorithm outperforms the state-of-the-art metric learning to rank methods and the stylish query-independent LtR algorithms regarding accuracy and computational efficiency.
Centered kNN Graph for Semi-Supervised Learning
Authors: Ikumi Suzuki, Kazuo Hara
Venue: SIGIR
Year: 2017

Graph construction is an important process in graph-based semi-supervised learning. Presently, the mutual kNN graph is the most preferred as it reduces hub nodes which can be a cause of failure during the process of label propagation. However, the mutual kNN graph, which is usually very sparse, suffers from over sparsification problem. That is, although the number of edges connecting nodes that have different labels decreases in the mutual kNN graph, the number of edges connecting nodes that have the same labels also reduces. In addition, over sparsification can produce a disconnected graph, which is not desirable for label propagation. So we present a new graph construction method, the centered kNN graph, which not only reduces hub nodes but also avoids the over sparsification problem.
Retrieval Algorithms Optimized for Human Learning
Authors: Rohail Syed, Kevyn Collins-Thompson
Venue: SIGIR
Year: 2017

While search technology is widely used for learning-oriented information needs, the results provided by popular services such as Web search engines are optimized primarily for generic relevance, not effective learning outcomes. As a result, the typical information trail that a user must follow while searching to achieve a learning goal may be an inefficient one involving unnecessarily easy or difficult content, or material that is irrelevant to actual learning progress relative to a user's existing knowledge. We address this problem by introducing a novel theoretical framework, algorithms, and empirical analysis of an information retrieval model that is optimized for learning outcomes instead of generic relevance. We do this by formulating an optimization problem that incorporates a cognitive learning model into a retrieval objective, and then give an algorithm for an efficient approximate solution to find the search results that represent the best 'training set' for a human learner. Our model can personalize results for an individual user's learning goals, as well as account for the effort required to achieve those goals for a given set of retrieval results. We investigate the effectiveness and efficiency of our retrieval framework relative to a commercial search engine baseline ('Google') through a crowdsourced user study involving a vocabulary learning task, and demonstrate the effectiveness of personalized results from our model on word learning outcomes.
Evolution of Information Needs based on Life Event Experiences with Topic Transition
Authors: Naoto Takeda, Yohei Seki, Mimpei Morishita, Yoichi Inagaki
Venue: SIGIR
Year: 2017

We propose a method to clarify the evolution of users' information needs related to a user's interests and actions based upon life events such as "childbirth." First, we extract topic transitions using dynamic topic models from blogs posted by users who have experienced life events. Next, we select the topics by computing the differences in topic probabilities before and after the life event. We evaluated our method based on three life events: "childbirth," "finding employment," and "marriage." Our method selected life event-relevant topics such as "child development," "working life," and "wedding ceremony." We found mothers' information needs such as "how to introduce baby food," employees' information needs such as "preparing an induction programme," and couples' information needs such as "wedding reception planning" in each topic.
On the Reusability of "Living Labs" Test Collections: : A Case Study of Real-Time Summarization
Authors: Luchen Tan, Gaurav Baruah, Jimmy Lin
Venue: SIGIR
Year: 2017

Information retrieval test collections are typically built using data from large-scale evaluations in international forums such as TREC, CLEF, and NTCIR. Previous validation studies on pool-based test collections for ad hoc retrieval have examined their reusability to accurately assess the effectiveness of systems that did not participate in the original evaluation. To our knowledge, the reusability of test collections derived from "living labs" evaluations, based on logs of user activity, has not been explored. In this paper, we performed a "leave-one-out" analysis of human judgment data derived from the TREC 2016 Real-Time Summarization Track and show that those judgments do not appear to be reusable. While this finding is limited to one specific evaluation, it does call into question the reusability of test collections built from living labs in general, and at the very least suggests the need for additional work in validating such experimental instruments.
Neural Network based Reinforcement Learning for Real-time Pushing on Text Stream
Authors: Haihui Tan, Ziyu Lu, Wenjie Li
Venue: SIGIR
Year: 2017

The massive amount of noisy and redundant information in text streams makes it a challenge for users to acquire timely and relevant information in social media. Real-time notification pushing on text stream is of practical importance. In this paper, we formulate the real-time pushing on text stream as a sequential decision making problem and propose a Neural Network based Reinforcement Learning (NNRL) algorithm for real-time decision making, e.g., push or skip the incoming text, with considering both history dependencies and future uncertainty. A novel Q-Network which contains a Long Short Term Memory (LSTM) layer and three fully connected neural network layers is designed to maximize the long-term rewards. Experiment results on the real data from TREC 2016 Real-time Summarization track show that our algorithm significantly outperforms state-of-the-art methods.
ENCORE: External Neural Constraints Regularized Distant Supervision for Relation Extraction
Authors: Siliang Tang, Jinjian Zhang, Ning Zhang, Fei Wu, Jun Xiao, Yueting Zhuang
Venue: SIGIR
Year: 2017

Distant Supervision is a widely used approach for training relation extraction models. It generates noisy training samples by heuristically labeling a corpus using an existing knowledge base. Previous noise reduction methods for distant supervision fail to utilize information such as data credibility and sample confidence. In this paper, we proposed a novel neural framework, named ENCORE (External Neural COnstraints REgularized distant supervision), which allows an integration of other information for standard DS through regularizations under multiple external neural networks. In ENCORE, a teacher-student co-training mechanism is used to iterative distilling information from external neural networks to an existing relation extraction model. The experiment results demonstrated that without increasing any data or reshaping its original structure, ENCORE enhanced a CNN based relation extraction model for over 12%. The enhanced model also outperforms the state-of-the-art relation extraction method on the same dataset.
Learning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture
Authors: Yi Tay, Minh C. Phan, Anh Tuan Luu, Siu Cheung Hui
Venue: SIGIR
Year: 2017

We describe a new deep learning architecture for learning to rank question answer pairs. Our approach extends the long short-term memory (LSTM) network with holographic composition to model the relationship between question and answer representations. As opposed to the neural tensor layer that has been adopted recently, the holographic composition provides the benefits of scalable and rich representational learning approach without incurring huge parameter costs. Overall, we present Holographic Dual LSTM (HD-LSTM), a unified architecture for both deep sentence modeling and semantic matching. Essentially, our model is trained end-to-end whereby the parameters of the LSTM are optimized in a way that best explains the correlation between question and answer representations. In addition, our proposed deep learning architecture requires no extensive feature engineering. Via extensive experiments, we show that HD-LSTM outperforms many other neural architectures on two popular benchmark QA datasets. Empirical studies confirm the effectiveness of holographic composition over the neural tensor layer.
ASTERIX: Ambiguity and Missing Element-Aware XML Keyword Search Engine
Authors: Ba Quan Truong, Sourav S. Bhowmick, Curtis E. Dyreson, Hong Jing Khok
Venue: SIGIR
Year: 2017

Despite a decade of research on XML keyword search (XKS), demonstration of a high quality XKS system has still eluded the information retrieval community. Existing XKS engines primarily suffer from two limitations. First, although the smallest lowest common ancestor (SLCA) algorithm (or a variant, e.g., ELCA) is widely accepted as a meaningful way to identify subtrees containing the query keywords, SLCA typically performs poorly on documents with missing elements, i.e., (sub)elements that are optional, or appear in some instances of an element type but not all. Second, since keyword search can be ambiguous with multiple possible interpretations, it is desirable for an XKS engine to automatically expand the original query by providing a classification of different possible interpretations of the query w.r.t. the original results. However, existing XKS systems do not support such result-based query expansion. We demonstrate ASTERIX, an innovative XKS engine that addresses these limitations.