OpenRaiser/Intern-Atlas huggingface dataset:
Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs
Authors: Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, Ryen W. White
Venue: SIGIR
Year: 2018

The rapidly increasing ubiquity of computing puts a great demand on next-generation human-machine interfaces. Natural language interfaces, exemplified by virtual assistants like Apple Siri and Microsoft Cortana, are widely believed to be a promising direction. However, current natural language interfaces provide users with little help in case of incorrect interpretation of user commands. We hypothesize that the support of fine-grained user interaction can greatly improve the usability of natural language interfaces. In the specific setting of natural language interface to web APIs, we conduct a systematic study to verify our hypothesis. To facilitate this study, we propose a novel modular sequence-to-sequence model to create interactive natural language interfaces. By decomposing the complex prediction process of a typical sequence-to-sequence model into small, highly-specialized prediction units called modules, it becomes straightforward to explain the model prediction to the user, and solicit user feedback to correct possible prediction errors at a fine-grained level. We test our hypothesis by comparing an interactive natural language interface with its non-interactive version through both simulation and human subject experiments with real-world APIs. We show that with the interactive natural language interface, users can achieve a higher success rate and a lower task completion time, which lead to greatly improved user satisfaction.
Efficient Query Processing Infrastructures: A half-day tutorial at SIGIR 2018
Authors: Nicola Tonellotto, Craig Macdonald
Venue: SIGIR
Year: 2018

Typically, techniques that benefit effectiveness of information retrieval (IR) systems have a negative impact on efficiency. Yet, with the large scale of Web search engines, there is a need to deploy efficient query processing techniques to reduce the cost of the infrastructure required. This tutorial aims to provide a detailed overview of the infrastructure of an IR system devoted to the efficient yet effective processing of user queries. This tutorial guides the attendees through the main ideas, approaches and algorithms developed in the last 30 years in query processing. In particular, we illustrate, with detailed examples and simplified pseudo-code, the most important query processing strategies adopted in major search engines, with a particular focus on dynamic pruning techniques. Moreover, we present and discuss the state-of-the-art innovations in query processing, such as impact-sorted and blockmax indexes. We also describe how modern search engines exploit such algorithms with learning-to-rank (LtR) models to produce effective results, exploiting new approaches in LtR query processing. Finally, this tutorial introduces query efficiency predictors for dynamic pruning, and discusses their main applications to scheduling, routing, selective processing and parallelisation of query processing, as deployed by a major search engine.
Towards Intent-Aware Contextual Music Recommendation: Initial Experiments
Authors: Sergey Volokhin, Eugene Agichtein
Venue: SIGIR
Year: 2018

While activity-aware music recommendation has been shown to improve the listener experience, we posit that modeling the \em listening intent can further improve recommendation quality. In this paper, we perform initial exploration of the dominant music listening intents associated with common activities, using music retrieved from popular online music services. We show that these intents can be approximated through audio features of the music itself, and potentially improve recommendation quality. Our initial results, based on 10 common activities and 5 popular listening intents associated with these activities, support our hypothesis, and open a promising direction towards intent-aware contextual music recommendation.
Efficient Exploration of Gradient Space for Online Learning to Rank
Authors: Huazheng Wang, Ramsey Langley, Sonwoo Kim, Eric McCord-Snook, Hongning Wang
Venue: SIGIR
Year: 2018

Online learning to rank (OL2R) optimizes the utility of returned search results based on implicit feedback gathered directly from users. In this paper, we accelerate the online learning process by efficient exploration in the gradient space. Our algorithm, named as Null Space Gradient Descent, reduces the exploration space to only the null space of recent poorly performing gradients. This prevents the algorithm from repeatedly exploring directions that have been discouraged by the most recent interactions with users. To improve sensitivity of the resulting interleaved test, we selectively construct candidate rankers to maximize the chance that they can be differentiated by candidate ranking documents in the current query; and we use historically difficult queries to identify the best ranker when tie occurs in comparing the rankers. Extensive experimental comparisons with the state-of-the-art OL2R algorithms on several public benchmarks confirmed the effectiveness of our proposal algorithm, especially in its fast learning convergence and promising ranking quality at an early stage.
A Large-Scale Study of Mobile Search Examination Behavior
Authors: Xiaochuan Wang, Ning Su, Zexue He, Yiqun Liu, Shaoping Ma
Venue: SIGIR
Year: 2018

With the rapid growth of mobile web search, it is necessary and important to understand user's examination behavior on mobile devices in the absence of clicks. Previous studies used viewport metrics to estimate user's attention. However, there still lacks an in-depth understanding of how search users examine and interact with the mobile SERP. In this work, based on the large-scale real search log collected from a popular commercial mobile search engine, we present a comprehensive analysis of examination behavior. Specifically, we analyze the position bias, the relationship with click behavior, and examination's change as the session continues. The findings shed new light on the understanding of user's examination behavior, and also provide some implication for the improvement and evaluation of mobile search engine.
Do Not Pull My Data for Resale: Protecting Data Providers Using Data Retrieval Pattern Analysis
Authors: Guosai Wang, Shiyang Xiang, Yitao Duan, Ling Huang, Wei Xu
Venue: SIGIR
Year: 2018

Data providers have a profound contribution to many fields such as finance, economy, and academia by serving people with both web-based and API-based query service of specialized data. Among the data users, there are data resellers who abuse the query APIs to retrieve and resell the data to make a profit, which harms the data provider's interests and causes copyright infringement. In this work, we define the "anti-data-reselling" problem and propose a new systematic method that combines feature engineering and machine learning models to provide a solution. We apply our method to a real query log of over 9,000 users with limited labels provided by a large financial data provider and get reasonable results, insightful observations, and real deployments.
Multi-Target Stance Detection via a Dynamic Memory-Augmented Network
Authors: Penghui Wei, Junjie Lin, Wenji Mao
Venue: SIGIR
Year: 2018

Stance detection aims at inferring from text whether the author is in favor of, against, or neutral towards a target entity. Most of the existing studies consider different target entities separately. However, in many scenarios, stance targets are closely related, such as several candidates in a general election and different brands of the same product. Multi-target stance detection, in contrast, aims at jointly detecting stances towards multiple related targets. As stance expression regarding a target can provide additional information to help identify the stances towards other related targets, modeling expressions regarding multiple targets jointly is beneficial for improving the overall performance compared to single-target scheme. In this paper, we propose a dynamic memory-augmented network DMAN for multi-target stance detection. DMAN utilizes a shared external memory, which is dynamically updated through the learning process, to capture and store stance-indicative information for multiple related targets. It then jointly predicts stances towards these targets in a multitask manner. Experimental results show the effectiveness of our DMAN model.
Item Retrieval as Utility Estimation
Authors: Shawn R. Wolfe, Yi Zhang
Venue: SIGIR
Year: 2018

Retrieval systems have greatly improved over the last half century, estimating relevance to a latent user need in a wide variety of areas. One common task in e-commerce and science that has not enjoyed such advancements is searching through a catalog of items. Finding a desirable item in such a catalog requires that the user specify desirable item properties, specifically desirable attribute values. Existing item retrieval systems assume the user can formulate a good Boolean or SQL-style query to retrieve items, as one would do with a database, but this is often challenging, particularly given multiple numeric attributes. Such systems avoid inferring query intent, instead requiring the user to precisely specify what matches the query. A contrasting approach would be to estimate how well items match the user's latent desires and return items ranked by this estimation. Towards this end, we present a retrieval model inspired by multi-criteria decision making theory, concentrating on numeric attributes. In two user studies (choosing airline tickets and meal plans) using Amazon Mechanical Turk, we evaluate our novel approach against the de facto standard of Boolean retrieval and several models proposed in the literature. We use a novel competitive game to motivate test subjects and compare methods based on the results of the subjects' initial query and their success in the game. In our experiments, our new method significantly outperformed the others, whereas the Boolean approaches had the worst performance.
Large Scale Search Engine Marketing (SEM) at Airbnb
Authors: James Wong, Brendan Collins, Ganesh Venkataraman
Venue: SIGIR
Year: 2018

Airbnb is an online marketplace which connects hosts and guests all over the world. Our inventory includes over 4.5 million listings, which enable the travel of over 300 million guests. The growth team at Airbnb is responsible for helping travelers find Airbnb, in part by participating in ad auctions on major search platforms such as Google and Bing. In this talk, we will describe how ad- vertising efficiently on these platforms requires solving several information retrieval and machine learning problems, including query understanding, click value estimation, and realtime pacing of our expenditure.
Identifying and Modeling Information Resumption Behaviors in Cross-Device Search
Authors: Dan Wu, Jing Dong, Yuan Tang
Venue: SIGIR
Year: 2018

Enlightened by task resumption behaviors in cross-session search, we explore information resumption behaviors in cross-device search. In order to find important features of information resumption behaviors, we conducted a user experiment and modeled information resumption behaviors using machine learning. The model of C5.0 Decision Tree outperformed and showed that features of FamiliarityScores, AveEditDistance, AveQueryEffectiveRate and ValidClickRate are of importance.
Identify Shifts of Word Semantics through Bayesian Surprise
Authors: Zhuofeng Wu, Cheng Li, Zhe Zhao, Fei Wu, Qiaozhu Mei
Venue: SIGIR
Year: 2018

Much work has been done recently on learning word embeddings from large corpora, which attempts to find the coordinates of words in a static and high dimensional semantic space. In reality, such corpora often span a sufficiently long time period, during which the meanings of many words may have changed. The co-evolution of word meanings may also result in a distortion of the semantic space, making these static embeddings unable to accurately represent the dynamics of semantics. In this paper, we present a novel computational method to capture such changes and to model the evolution of word semantics. Distinct from existing approaches that learn word embeddings independently from time periods and then align them, our method explicitly establishes the stable topological structure of word semantics and identifies the surprising changes in the semantic space over time through a principled statistical method. Empirical experiments on large-scale real-world corpora demonstrate the effectiveness of the proposed approach, which outperforms the state-of-the-art by a large margin.
Deep Learning for Matching in Search and Recommendation
Authors: Jun Xu, Xiangnan He, Hang Li
Venue: SIGIR
Year: 2018

Matching is the key problem in both search and recommendation, that is to measure the relevance of a document to a query or the interest of a user on an item. Previously, machine learning methods have been exploited to address the problem, which learns a matching function from labeled data, also referred to as "learning to match''. In recent years, deep learning has been successfully applied to matching and significant progresses have been made. Deep semantic matching models for search and neural collaborative filtering models for recommendation are becoming the state-of-the-art technologies. The key to the success of the deep learning approach is its strong ability in learning of representations and generalization of matching patterns from raw data (e.g., queries, documents, users, and items, particularly in their raw forms). In this tutorial, we aim to give a comprehensive survey on recent progress in deep learning for matching in search and recommendation. Our tutorial is unique in that we try to give a unified view on search and recommendation. In this way, we expect researchers from the two fields can get deep understanding and accurate insight on the spaces, stimulate more ideas and discussions, and promote developments of technologies. The tutorial mainly consists of three parts. Firstly, we introduce the general problem of matching, which is fundamental in both search and recommendation. Secondly, we explain how traditional machine learning techniques are utilized to address the matching problem in search and recommendation. Lastly, we elaborate how deep learning can be effectively used to solve the matching problems in both tasks.
A Co-Memory Network for Multimodal Sentiment Analysis
Authors: Nan Xu, Wenji Mao, Guandan Chen
Venue: SIGIR
Year: 2018

With the rapid increase of diversity and modality of data in user-generated contents, sentiment analysis as a core area of social media analytics has gone beyond traditional text-based analysis. Multimodal sentiment analysis has become an important research topic in recent years. Most of the existing work on multimodal sentiment analysis extracts features from image and text separately, and directly combine them to train a classifier. As visual and textual information in multimodal data can mutually reinforce and complement each other in analyzing the sentiment of people, previous research all ignores this mutual influence between image and text. To fill this gap, in this paper, we consider the interrelation of visual and textual information, and propose a novel co-memory network to iteratively model the interactions between visual contents and textual words for multimodal sentiment analysis. Experimental results on two public multimodal sentiment datasets demonstrate the effectiveness of our proposed model compared to the state-of-the-art methods.
Characterizing and Supporting Question Answering in Human-to-Human Communication
Authors: Xiao Yang, Ahmed Hassan Awadallah, Madian Khabsa, Wei Wang, Miaosen Wang
Venue: SIGIR
Year: 2018

Email continues to be one of the most important means of online communication. People spend a significant amount of time sending, reading, searching and responding to email in order to manage tasks, exchange information, etc. In this paper, we focus on information exchange over enterprise email in the form of questions and answers. We study a large scale publicly available email dataset to characterize information exchange via questions and answers in enterprise email. We augment our analysis with a survey to gain insights on the types of questions exchanged, when and how do people get back to them and whether this behavior is adequately supported by existing email management and search functionality. We leverage this understanding to define the task of extracting question/answer pairs from threaded email conversations. We propose a neural network based approach that matches the question to the answer considering comparisons at different levels of granularity. We also show that we can improve the performance by leveraging external data of question and answer pairs. We test our approach using a manually labeled email data collected using a crowd-sourcing annotation study. Our findings have implications for designing email clients and intelligent agents that support question answering and information lookup in email.
A Study of Per-Topic Variance on System Comparison
Authors: Meng Yang, Peng Zhang, Dawei Song
Venue: SIGIR
Year: 2018

Under the notion that the document collection is a sample from a population, the observed per-topic metric (e.g., AP) value varies with different samples, leading to the per-topic variance. The results of the system comparison, such as comparing the ranking of systems according to the summary metric (e.g., MAP) or testing whether there is significant difference between two systems, are affected by the variability of per-topic metric values. In this paper, we study the effect of per-topic variance on the system comparison. To measure such effects, we employ two ranking-based methods, i.e., Error Rate (ER) and Kendall Rank Correlation Coefficient (KRCC), as well as two significance test based methods, namely Achieved Significance Level (ASL) and Estimated Difference (ED). We conduct empirical comparison of TREC participated systems on Robust and Adhoc track, which shows that the effect of per-topic variance on the ranking of systems is not obvious, while the significance test based comparisons are susceptible to the per-topic variance.
Robust Asymmetric Recommendation via Min-Max Optimization
Authors: Peng Yang, Peilin Zhao, Vincent W. Zheng, Lizhong Ding, Xin Gao
Venue: SIGIR
Year: 2018

Recommender systems with implicit feedback (e.g. clicks and purchases) suffer from two critical limitations: 1) imbalanced labels may mislead the learning process of the conventional models that assign balanced weights to the classes; and 2) outliers with large reconstruction errors may dominate the objective function by the conventional $L_2$-norm loss. To address these issues, we propose a robust asymmetric recommendation model. It integrates cost-sensitive learning with capped unilateral loss into a joint objective function, which can be optimized by an iteratively weighted approach. To reduce the computational cost of low-rank approximation, we exploit the dual characterization of the nuclear norm to derive a min-max optimization problem and design a subgradient algorithm without performing full SVD. Finally, promising empirical results demonstrate the effectiveness of our algorithm on benchmark recommendation datasets.
Big Data at Didi Chuxing
Authors: Jieping Ye
Venue: SIGIR
Year: 2018

Didi Chuxing is the largest ride-sharing platform in China, providing transportation services for over 400 million users. Every day, Didi Chuxing's platform generates over 100 TB worth of data, processes more than 40 billion routing requests, and produces over 15 billion location points. In this talk, I will explain how Didi Chuxing applies big data and AI technologies to analyze big transportation data and improve the travel experience for millions of users.
Hide-n-Seek: An Intent-aware Privacy Protection Plugin for Personalized Web Search
Authors: Puxuan Yu, Wasi Uddin Ahmad, Hongning Wang
Venue: SIGIR
Year: 2018

We develop Hide-n-Seek, an intent-aware privacy protection plugin for personalized web search. In addition to users' genuine search queries, Hide-n-Seek submits k cover queries and corresponding clicks to an external search engine to disguise a user's search intent grounded and reinforced in a search session by mimicking the true query sequence. The cover queries are synthesized and randomly sampled from a topic hierarchy, where each node represents a coherent search topic estimated by both n-gram and neural language models constructed over crawled web documents. Hide-n-Seek also personalizes the returned search results by re-ranking them based on the genuine user profile developed and maintained on the client side. With a variety of graphical user interfaces, we present the topic-based query obfuscation mechanism to the end users for them to digest how their search privacy is protected.
Reducing Variance in Gradient Bandit Algorithm using Antithetic Variates Method
Authors: Sihao Yu, Jun Xu, Yanyan Lan, Jiafeng Guo, Xueqi Cheng
Venue: SIGIR
Year: 2018

Policy gradient, which makes use of Monte Carlo method to get an unbiased estimation of the parameter gradients, has been widely used in reinforcement learning. One key issue in policy gradient is reducing the variance of the estimation. From the viewpoint of statistics, policy gradient with baseline, a successful variance reduction method for policy gradient, directly applies the control variates method, a traditional variance reduction technique used in Monte Carlo, to policy gradient. One problem with control variates method is that the quality of estimation heavily depends on the choice of the control variates. To address the issue and inspired by the antithetic variates method for variance reduction, we propose to combine the antithetic variates method with traditional policy gradient for the multi-armed bandit problem. Furthermore, we achieve a new policy gradient algorithm called Antithetic-Arm Bandit (AAB). In AAB, the gradient is estimated through coordinate ascent where at each iteration gradient of the target arm is estimated through: 1) constructing a sequence of arms which is approximately monotonic in terms of estimated gradients, 2) sampling a pair of antithetic arms over the sequence, and 3) re-estimating the target gradient based on the sampled pair. Theoretical analysis proved that AAB achieved an unbiased and variance reduced estimation. Experimental results based on a multi-armed bandit task showed that AAB can achieve state-of-the-art performances.
Ranking Documents by Answer-Passage Quality
Authors: Evi Yulianti, Ruey-Cheng Chen, Falk Scholer, W. Bruce Croft, Mark Sanderson
Venue: SIGIR
Year: 2018

Evidence derived from passages that closely represent likely answers to a posed query can be useful input to the ranking process. Based on a novel use of Community Question Answering data, we present an approach for the creation of such passages. A general framework for extracting answer passages and estimating their quality is proposed, and this evidence is integrated into ranking models. Our experiments on two web collections show that such quality estimates from answer passages provide a strong indication of document relevance and compare favorably to previous passage-based methods. Combining such evidence can significantly improve over a set of state-of-the-art ranking models, including Quality-Biased Ranking, External Expansion, and a combination of both. A final ranking model that incorporates all quality estimates achieves further improvements on both collections.