OpenRaiser/Intern-Atlas huggingface dataset:
A Counterfactual Framework for Seller-Side A/B Testing on Marketplaces
Authors: Viet Ha-Thuc, Avishek Dutta, Ren Mao, Matthew Wood, Yunli Liu
Venue: SIGIR
Year: 2020

Many consumer products are two-sided marketplaces, ranging from commerce products that connect buyers and sellers, such as Amazon, Alibaba, and Facebook Marketplace, to sharing-economy products that connect passengers to drivers or guests to hosts, like Uber and Airbnb. The search and recommender systems behind these products are typically optimized for objectives like click-through, purchase, or booking rates, which are mostly tied to the consumer side of the marketplace (namely buyers, passengers, or guests). For the long-term growth of these products, it is also crucial to consider the value to the providers (sellers, drivers, or hosts). However, optimizing ranking for such objectives is uncommon because it is challenging to measure the causal effect of ranking changes on providers. For instance, if we run a standard seller-side A/B test on Facebook Marketplace that exposes a small percentage of sellers, what we observe in the test would be significantly different from when the treatment is launched to all sellers. To overcome this challenge, we propose a counterfactual framework for seller-side A/B testing. The key idea is that items in the treatment group are ranked the same regardless of experiment exposure rate. Similarly, the items in the control are ranked where they would be if the status quo is applied to all sellers. Theoretically, we show that the framework satisfies the stable unit treatment value assumption since the experience that sellers receive is only affected by their own treatment and independent of the treatment of other sellers. Empirically, both seller-side and buyer-side online A/B tests are conducted on Facebook Marketplace to verify the framework.
When the Human is in the Loop: Cost, Effort and Behavior
Authors: Lei Han 0003
Venue: SIGIR
Year: 2020

Data from human-machine interaction can be used to improve the quality of artificial intelligence (AI) systems. When designing a system with humans in the loop, one of the questions to be asked is how much human work is required to create a reliable data collection. Crowdsourcing has become a popular methodology to collect annotations from crowd workers who successfully complete crowdsourcing tasks. Thus, if workers reach varying task completion stages without finally submitting their work, all their effort would be unrewarded and annotations discarded. Task abandonment remains invisible within the platform. On the other hand, paid crowdsourcing dynamics are often influenced by large batches of similar tasks, allowing workers to learn and develop efficient work strategies. To date, however, there is limited research aiming at understanding how human annotators complete tasks over time. Even for complex tasks in lab studies, understanding how behavioral patterns evolve during task completion remains unexplored.
Dynamic Link Prediction by Integrating Node Vector Evolution and Local Neighborhood Representation
Authors: Xiaorong Hao, Tao Lian, Li Wang 0014
Venue: SIGIR
Year: 2020

Many networks in real applications are constantly evolving as the creation and elimination of nodes and edges. Dynamic link prediction aims to infer whether there will be an edge between a pair of nodes, given the recent evolution history of the network. In this paper, we devise a flexible framework for link prediction on dynamic networks regularly archived as different snapshots. On the basis of node vectors learned on individual snapshots, a gated recurrent unit (GRU) network is utilized to model the node vector evolution series and predict the node representation in the future. Then, the edge representation is not only constructed from the interaction between representations of the target node pair, but also enriched with local neighborhood representations---historical embeddings of their common neighbors. Finally, a binary classifier is trained to perform link prediction. The framework can be instantiated with many off-the-shelf outstanding node embedding and binary classification methods. Extensive experiments on three different datasets demonstrate the effectiveness and flexibility of our proposed framework. Ablation studies show that the node vector evolution and local neighborhood representation both have positive but different effects on dynamic link prediction on diverse networks.
ArTest: The First Test Collection for Arabic Web Search with Relevance Rationales
Authors: Maram Hasanain, Yassmine Barkallah, Reem Suwaileh, Mücahid Kutlu, Tamer Elsayed
Venue: SIGIR
Year: 2020

The scarcity of Arabic test collections has long hindered information retrieval (IR) research over the Arabic Web. In this work, we present ArTest, the first large-scale test collection designed for the evaluation of ad-hoc search over the Arabic Web. ArTest uses ArabicWeb16, a collection of around 150M Arabic Web pages as the document collection, and includes 50 topics, 10,529 relevance judgments, and (more importantly) a rationale behind each judgment. To our knowledge, this is also the first IR test collection that includes rationales of primary assessors (i.e., topic developers) for their relevance judgments, exhibiting a useful resource for understanding the relevance phenomena. Finally, ArTest is made publicly-available for the research community.
Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search
Authors: Helia Hashemi, Hamed Zamani, W. Bruce Croft
Venue: SIGIR
Year: 2020

Asking clarifying questions in response to ambiguous or faceted queries has been recognized as a useful technique for various information retrieval systems, especially conversational search systems with limited bandwidth interfaces. Analyzing and generating clarifying questions have been studied recently but the accurate utilization of user responses to clarifying questions has been relatively less explored. In this paper, we enrich the representations learned by Transformer networks using a novel attention mechanism from external information sources that weights each term in the conversation. We evaluate this Guided Transformer model in a conversational search scenario that includes clarifying questions. In our experiments, we use two separate external sources, including the top retrieved documents and a set of different possible clarifying questions for the query. We implement the proposed representation learning model for two downstream tasks in conversational search; document retrieval and next clarifying question selection. Our experiments use a public dataset for search clarification and demonstrate significant improvements compared to competitive baselines.
Think Beyond the Word: Understanding the Implied Textual Meaning by Digesting Context, Local, and Noise
Authors: Guoxiu He, Zhe Gao, Zhuoren Jiang, Yangyang Kang, Changlong Sun, Xiaozhong Liu 0001, Wei Lu 0019
Venue: SIGIR
Year: 2020

Implied semantics is a complex language act that can appear everywhere on the Cyberspace. The prevalence of implied spam texts, such as implied pornography, sarcasm, and abuse hidden within the novel, tweet, microblog, or review, can be extremely harmful to the physical and mental health of teenagers. The non-literal interpretation of the implied text is hard to be understood by machine models due to its high context-sensitivity and heavy usage of figurative language. In this study, inspired by human reading comprehension, we propose a novel, simple, and effective deep neural framework, called Skim and Intensive Reading Model (SIRM), for figuring out implied textual meaning. The proposed SIRM consists of three main components, namely the skim reading component, intensive reading component, and adversarial training component. N-gram features are quickly extracted from the skim reading component, which is a combination of several convolutional neural networks, as skim (entire) information. An intensive reading component enables a hierarchical investigation for both sentence-level and paragraph-level representation, which encapsulates the current (local) embedding and the contextual information (context) with a dense connection. More specifically, the contextual information includes the near-neighbor information and the skim information mentioned above. Finally, besides the common training loss function, we employ an adversarial loss function as a penalty over the skim reading component to eliminate noisy information (noise) arisen from special figurative words in the training data. To verify the effectiveness, robustness, and efficiency of the proposed architecture, we conduct extensive comparative experiments on an industrial novel dataset involving implied pornography and three sarcasm benchmarks. Experimental results indicate that (1) the proposed model, which benefits from context and local modeling and consideration of figurative language (noise), outperforms existing state-of-the-art solutions, with comparable parameter scale and running speed; (2) the SIRM yields superior robustness in terms of parameter size sensitivity; (3) compared with ablation and addition variants of the SIRM, the final framework is efficient enough.
The Next Generation of Neural Networks
Authors: Geoffrey E. Hinton
Venue: SIGIR
Year: 2020

The most important unsolved problem with artificial neural networks is how to do unsupervised learning as effectively as the brain. There are currently two main approaches to unsupervised learning. In the first approach, exemplified by BERT and Variational Autoencoders, a deep neural network is used to reconstruct its input. This is problematic for images because the deepest layers of the network need to encode the fine details of the image. An alternative approach, introduced by Becker and Hinton in 1992, is to train two copies of a deep neural network to produce output vectors that have high mutual information when given two different crops of the same image as their inputs. This approach was designed to allow the representations to be untethered from irrelevant details of the input.
Query Reformulation in E-Commerce Search
Authors: Sharon Hirsch, Ido Guy, Alexander Nus, Arnon Dagan, Oren Kurland
Venue: SIGIR
Year: 2020

The importance of e-commerce platforms has driven forward a growing body of research work on e-commerce search. We present the first large-scale and in-depth study of query reformulations performed by users of e-commerce search; the study is based on the query logs of eBay's search engine. We analyze various factors including the distribution of different types of reformulations, changes of search result pages retrieved for the reformulations, and clicks and purchases performed upon the retrieved results. We then turn to address a novel challenge in the e-commerce search realm: predicting whether a user will reformulate her query before presenting her the search results. Using a suite of prediction features, most of which are novel to this study, we attain high prediction quality. Some of the features operate prior to retrieval time, whereas others rely on the retrieved results. While the latter are substantially more effective than the former, we show that the integration of these two types of features is of merit. We also show that high prediction quality can be obtained without considering information from the past about the user or the query she posted. Nevertheless, using these types of information can further improve prediction quality.
DataMirror: Reflecting on One's Data Self: A Tool for Social Media Users to Explore Their Digital Footprints
Authors: Amal Htait, Leif Azzopardi, Emma Nicol, Wendy Moncur
Venue: SIGIR
Year: 2020

Small pieces of data that are shared online, over time and across multiple social networks, have the potential to reveal more cumulatively than a person intends. This could result in harm, loss or detriment to them depending what information is revealed, who can access it, and how it is processed. But how aware are social network users of how much information they are actually disclosing? And if they could examine all their data, what cumulative revelations might be found that could potentially increase their risk of various online threats (social engineering, fraud, identify theft, loss of face, etc.)? In this paper, we present DataMirror, an initial prototype tool, that enables social network users to aggregate their online data so that they can search, browse and visualise what they have put online. The aim of the tool is to investigate and explore people's awareness of their data self that is projected online; not only in terms of the volume of information that they might share, but what it may mean when combined together, what pieces of sensitive information may be gleaned from their data, and what machine learning may infer about them given their data.
Modeling Personalized Item Frequency Information for Next-basket Recommendation
Authors: Haoji Hu, Xiangnan He 0001, Jinyang Gao, Zhi-Li Zhang
Venue: SIGIR
Year: 2020

Next-basket recommendation (NBR) is prevalent in e-commerce and retail industry. In this scenario, a user purchases a set of items (a basket) at a time. NBR performs sequential modeling and recommendation based on a sequence of baskets. NBR is in general more complex than the widely studied sequential (session-based) recommendation which recommends the next item based on a sequence of items. Recurrent neural network (RNN) has proved to be very effective for sequential modeling, and thus been adapted for NBR. However, we argue that existing RNNs cannot directly capture item frequency information in the recommendation scenario.
Residual-Duet Network with Tree Dependency Representation for Chinese Question-Answering Sentiment Analysis
Authors: Guangyi Hu, Chongyang Shi 0001, Shufeng Hao, Yu Bai
Venue: SIGIR
Year: 2020

Question-answering sentiment analysis (QASA) is a novel but meaningful sentiment analysis task based on question-answering online reviews. Existing neural network-based models that conduct sentiment analysis of online reviews have already achieved great success. However, the syntax and implicitly semantic connection in the dependency tree have not been made full use of, especially for Chinese which has specific syntax. In this work, we propose a Residual-Duet Network leveraging textual and tree dependency information for Chinese question-answering sentiment analysis. In particular, we explore the synergies of graph embedding with structural dependency links to learn syntactic information. The transverse and longitudinal compression encoders are developed to capture sentiment evidence with disparate types of compression and different residual connections. We evaluate our model on three Chinese QASA datasets in different domains. Experimental results demonstrate the superiority of our proposed model in Chinese question-answering sentiment analysis.
BiANE: Bipartite Attributed Network Embedding
Authors: Wentao Huang 0001, Yuchen Li 0001, Yuan Fang 0001, Ju Fan, Hongxia Yang
Venue: SIGIR
Year: 2020

Network embedding effectively transforms complex network data into a low-dimensional vector space and has shown great performance in many real-world scenarios, such as link prediction, node classification, and similarity search. A plethora of methods have been proposed to learn node representations and achieve encouraging results. Nevertheless, little attention has been paid on the embedding technique for bipartite attributed networks, which is a typical data structure for modeling nodes from two distinct partitions.
Towards Linking Camouflaged Descriptions to Implicit Products in E-commerce
Authors: Longtao Huang, Bo Yuan, Rong Zhang 0006, Quan Lu
Venue: SIGIR
Year: 2020

As the emergence of E-commerce services, billions of products are sold online everyday. How to detect illegal products from the large-scale online products has become an important and practical research problem. In order to evade detection, malicious sellers usually utilize camouflaged text to describe their illegal products implicitly. Thus brings great challenges to the current detection systems since newly camouflaged text can hardly be learned from historical data and the distribution of illegal and normal products is extremely unbalanced. Rather than solving this problem as a classification task in most previous efforts, we reformulate the problem from a perspective of implicit entity linking, which targets at linking a camouflaged description to a known product. In this paper, we introduce three types of context that could help to infer implicit entity from camouflaged descriptions and propose an end-to-end contextual representation model to capture the effect of different context. Furthermore, we involve a symmetric metric to model the matching score of the input title to the product by learning the mutual effect among the context. The experimental results on the datasets collected from a real-world E-commerce site demonstrate the advantage of the proposed model against the state-of-the-art methods.
Evidence Weighted Tree Ensembles for Text Classification
Authors: Md Zahidul Islam 0001, Jixue Liu, Jiuyong Li, Lin Liu 0003, Wei Kang 0004
Venue: SIGIR
Year: 2020

Text documents are often mapped to vectors of binary values where 1 indicates the presence of a word and 0 indicates the absence. The vectors are then used to train predictive models. In tree-based ensemble models, predictions from some decision trees may be made purely from absent words. This type of predictions should be trusted less as absent words can be interpreted in multiple ways. In this work, we propose to improve the comprehensibility and accuracy of ensemble models by distinguishing word presence and absence. The presented method weights predictions based on word presence. Experimental results on 35 real text datasets indicate that our method outperforms state-of-the-art ensemble methods on various text classification tasks.
Accelerated Convergence for Counterfactual Learning to Rank
Authors: Rolf Jagerman, Maarten de Rijke
Venue: SIGIR
Year: 2020

Counterfactual Learning To Rank (LTR) algorithms learn a ranking model from logged user interactions, often collected using a production system. Employing such an offline learning approach has many benefits compared to an online one, but it is challenging as user feedback often contains high levels of bias. Unbiased LTR uses Inverse Propensity Scoring (IPS) to enable unbiased learning from logged user interactions. One of the major difficulties in applying Stochastic Gradient Descent (SGD) approaches to counterfactual learning problems is the large variance introduced by the propensity weights. In this paper we show that the convergence rate of SGD approaches with IPS-weighted gradients suffers from the large variance introduced by the IPS weights: convergence is slow, especially when there are large IPS weights.
Graph Regularization for Multi-lingual Topic Models
Authors: Arnav Kumar Jain, Gundeep Arora, Rahul Agrawal
Venue: SIGIR
Year: 2020

Unsupervised multi-lingual language modeling has gained attraction in the last few years and poly-lingual topic models provide a mechanism to learn aligned document representations. However, training such models require translation-aligned data across languages, which is not always available. Also, in case of short texts like tweets, search queries, etc, the training of topic models continues to be a challenge. In this work, we present a novel strategy of creating a pseudo-parallel dataset followed by training topic models for sponsored search retrieval, that also mitigates the short text challenge. Our data augmentation strategy leverages easily available bipartite click-though graph that allows us to draw similar documents in different languages. The proposed methodology is evaluated on sponsored search system whose performance is measured on correctly matching the user intent, presented via the query, with ads provided by the advertiser. Our experiments substantiate the goodness of the method on EuroParl dataset and live search-engine traffic.
ECOM'20: The SIGIR 2020 Workshop on eCommerce
Authors: Dietmar Jannach, Surya Kallumadi, Tracy Holloway King, Weihua Luo, Shervin Malmasi
Venue: SIGIR
Year: 2020

eCommerce Information Retrieval (IR) is receiving increasing attention in the academic literature and is an essential component of some of the largest web sites (e.g. Amazon, Alibaba, Taobao, eBay, Airbnb, Target, Facebook). eCommerce organisations consistently sponsor SIGIR, reflecting the importance of IR research to them. This workshop (1) brings together researchers and practitioners of eCommerce IR to discuss topics unique to it, (2) determines how to use eCommerce's unique combination of free text, structured data, and customer behavioral data to improve search relevance, and (3) examines how to build data sets and evaluate algorithms in this domain. Since eCommerce customers often do not know exactly what they want to buy, recommendations are valuable for inspiration, serendipitous discovery and basket building. The theme of this year's eCommerce IR workshop is integrating recommendations into search for eCommerce. In addition to the focus on recommender systems in eCommerce search, Rakuten France is sponsoring a data challenge on taxonomy classification using multi-modal (image, text and structured data) input. The data challenge reflects themes from the 2017--2019 SIGIR workshops.
Response Quality in Human-Chatbot Collaborative Systems
Authors: Jiepu Jiang, Naman Ahuja
Venue: SIGIR
Year: 2020

We report the results of a crowdsourcing user study for evaluating the effectiveness of human-chatbot collaborative conversation systems, which aim to extend the ability of a human user to answer another person's requests in a conversation using a chatbot. We examine the quality of responses from two collaborative systems and compare them with human-only and chatbot-only settings. Our two systems both allow users to formulate responses based on a chatbot's top-ranked results as suggestions. But they encourage the synthesis of human and AI outputs to a different extent. Experimental results show that both systems significantly improved the informativeness of messages and reduced user effort compared with a human-only baseline while sacrificing the fluency and humanlikeness of the responses. Compared with a chatbot-only baseline, the collaborative systems provided comparably informative but more fluent and human-like messages.
Large-scale Multi-modal Search and QA at Alibaba
Authors: Rong Jin 0001
Venue: SIGIR
Year: 2020

In this talk, we will present ongoing efforts of Alibaba on large-scale multi-modal search ad question-answering, including image search, video search, cross lingual retrieval, and QA based on machine reading and comprehension (MRC). We will also share in this talk the core technologies that are crucial to the development of the related technologies, including representation learning, data augmentation, and robust learning.
Multi-behavior Recommendation with Graph Convolutional Networks
Authors: Bowen Jin, Chen Gao 0001, Xiangnan He 0001, Depeng Jin, Yong Li 0008
Venue: SIGIR
Year: 2020

Traditional recommendation models that usually utilize only one type of user-item interaction are faced with serious data sparsity or cold start issues. Multi-behavior recommendation taking use of multiple types of user-item interactions, such as clicks and favorites, can serve as an effective solution. Early efforts towards multi-behavior recommendation fail to capture behaviors' different influence strength on target behavior. They also ignore behaviors' semantics which is implied in multi-behavior data. Both of these two limitations make the data not fully exploited for improving the recommendation performance on the target behavior.