OpenRaiser/Intern-Atlas huggingface dataset:
An Efficient Adaptive Transfer Neural Network for Social-aware Recommendation
Authors: Chong Chen, Min Zhang, Chenyang Wang, Weizhi Ma, Minming Li, Yiqun Liu, Shaoping Ma
Venue: SIGIR
Year: 2019

Many previous studies attempt to utilize information from other domains to achieve better performance of recommendation. Recently, social information has been shown effective in improving recommendation results with transfer learning frameworks, and the transfer part helps to learn users' preferences from both item domain and social domain. However, two vital issues have not been well-considered in existing methods: 1) Usually, a static transfer scheme is adopted to share a user's common preference between item and social domains, which is not robust in real life where the degrees of sharing and information richness are varied for different users. Hence a non-personalized transfer scheme may be insufficient and unsuccessful. 2) Most previous neural recommendation methods rely on negative sampling in training to increase computational efficiency, which makes them highly sensitive to sampling strategies and hence difficult to achieve optimal results in practical applications. To address the above problems, we propose an Efficient Adaptive Transfer Neural Network (EATNN). By introducing attention mechanisms, the proposed model automatically assign a personalized transfer scheme for each user. Moreover, we devise an efficient optimization method to learn from the whole training set without negative sampling, and further extend it to support multi-task learning. Extensive experiments on three real-world public datasets indicate that our EATNN method consistently outperforms the state-of-the-art methods on Top-K recommendation task, especially for cold-start users who have few item interactions. Remarkably, EATNN shows significant advantages in training efficiency, which makes it more practical to be applied in real E-commerce scenarios. The code is available at (https://github.com/chenchongthu/EATNN).
One-Class Order Embedding for Dependency Relation Prediction
Authors: Meng-Fen Chiang, Ee-Peng Lim, Wang-Chien Lee, Xavier Jayaraj Siddarth Ashok, Philips Kokoh Prasetyo
Venue: SIGIR
Year: 2019

Learning the dependency relations among entities and the hierarchy formed by these relations by mapping entities into some order embedding space can effectively enable several important applications, including knowledge base completion and prerequisite relations prediction. Nevertheless, it is very challenging to learn a good order embedding due to the existence of partial ordering and missing relations in the observed data. Moreover, most application scenarios do not provide non-trivial negative dependency relation instances. We therefore propose a framework that performs dependency relation prediction by exploring both rich semantic and hierarchical structure information in the data. In particular, we propose several negative sampling strategies based on graph-specific centrality properties, which supplement the positive dependency relations with appropriate negative samples to effectively learn order embeddings. This research not only addresses the needs of automatically recovering missing dependency relations, but also unravels dependencies among entities using several real-world datasets, such as course dependency hierarchy involving course prerequisite relations, job hierarchy in organizations, and paper citation hierarchy. Extensive experiments are conducted on both synthetic and real-world datasets to demonstrate the prediction accuracy as well as to gain insights using the learned order embedding.
Explanatory and Actionable Debugging for Machine Learning: A TableQA Demonstration
Authors: Minseok Cho, Gyeongbok Lee, Seung-won Hwang
Venue: SIGIR
Year: 2019

Question answering from tables (TableQA) extracting answers from tables from the question given in natural language, has been actively studied. Existing models have been trained and evaluated mostly with respect to answer accuracy using public benchmark datasets such as WikiSQL. The goal of this demonstration is to show a debugging tool for such models, explaining answers to humans, known as explanatory debugging. Our key distinction is making it "actionable" to allow users to directly correct models upon explanation. Specifically, our tool surfaces annotation and models errors for users to correct, and provides actionable insights.
The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019)
Authors: Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, Ze Zhong Wu
Venue: SIGIR
Year: 2019

The importance of repeatability, replicability, and reproducibility is broadly recognized in the computational sciences, both in supporting desirable scientific methodology as well as sustaining empirical progress. This workshop tackles the replicability challenge for ad hoc document retrieval, via a common Docker interface specification to support images that capture systems performing ad hoc retrieval experiments on standard test collections.
Solr Integration in the Anserini Information Retrieval Toolkit
Authors: Ryan Clancy, Toke Eskildsen, Nick Ruest, Jimmy Lin
Venue: SIGIR
Year: 2019

Anserini is an open-source information retrieval toolkit built around Lucene to facilitate replicable research. In this demonstration, we examine different architectures for Solr integration in order to address two current limitations of the system: the lack of an interactive search interface and support for distributed retrieval. Two architectures are explored: In the first approach, Anserini is used as a frontend to index directly into a running Solr instance. In the second approach, Lucene indexes built directly with Anserini can be copied into a Solr installation and placed under its management. We discuss the tradeoffs associated with each architecture and report the results of a performance evaluation comparing indexing throughput. To illustrate the additional capabilities enabled by Anserini/Solr integration, we present a search interface built using the open-source Blacklight discovery interface.
Information Retrieval Meets Scalable Text Analytics: Solr Integration with Spark
Authors: Ryan Clancy, Jaejun Lee, Zeynep Akkalyoncu Yilmaz, Jimmy Lin
Venue: SIGIR
Year: 2019

Despite the broad adoption of both Apache Spark and Apache Solr, there is little integration between these two platforms to support scalable, end-to-end text analytics. We believe this is a missed opportunity, as there is substantial synergy in building analytical pipelines where the results of potentially complex faceted queries feed downstream text processing components. This demonstration explores exactly such an integration: we evaluate performance under different analytical scenarios and present three simple case studies that illustrate the range of possible analyses enabled by seamlessly connecting Spark to Solr.
Unbiased Low-Variance Estimators for Precision and Related Information Retrieval Effectiveness Measures
Authors: Gordon V. Cormack, Maura R. Grossman
Venue: SIGIR
Year: 2019

This work describes an estimator from which unbiased measurements of precision, rank-biased precision, and cumulative gain may be derived from a uniform or non-uniform sample of relevance assessments. Adversarial testing supports the theory that our estimator yields unbiased low-variance measurements from sparse samples, even when used to measure results that are qualitatively different from those returned by known information retrieval methods. Our results suggest that test collections using sampling to select documents for relevance assessment yield more accurate measurements than test collections using pooling, especially for the results of retrieval methods not contributing to the pool.
Quantifying Bias and Variance of System Rankings
Authors: Gordon V. Cormack, Maura R. Grossman
Venue: SIGIR
Year: 2019

When used to assess the accuracy of system rankings, Kendall's tau and other rank correlation measures conflate bias and variance as sources of error. We derive from tau a distance between rankings in Euclidean space, from which we can determine the magnitude of bias, variance, and error. Using bootstrap estimation, we show that shallow pooling has substantially higher bias and insubstantially lower variance than probability-proportional-to-size sampling, coupled with the recently released dynAP estimator.
Dynamic Sampling Meets Pooling
Authors: Gordon V. Cormack, Haotian Zhang, Nimesh Ghelani, Mustafa Abualsaud, Mark D. Smucker, Maura R. Grossman, Shahin Rahbariasl, Amira Ghenai
Venue: SIGIR
Year: 2019

A team of six assessors used Dynamic Sampling (Cormack and Grossman 2018) and one hour of assessment effort per topic to form, without pooling, a test collection for the TREC 2018 Common Core Track. Later, official relevance assessments were rendered by NIST for documents selected by depth-10 pooling augmented by move-to-front (MTF) pooling (Cormack et al. 1998), as well as the documents selected by our Dynamic Sampling effort. MAP estimates rendered from dynamically sampled assessments using the xinfAP statistical evaluator are comparable to those rendered from the complete set of official assessments using the standard trec_eval tool. MAP estimates rendered using only documents selected by pooling, on the other hand, differ substantially. The results suggest that the use of Dynamic Sampling without pooling can, for an order of magnitude less assessment effort, yield information-retrieval effectiveness estimates that exhibit lower bias, lower error, and comparable ability to rank system effectiveness.
Towards Better Support for Exploratory Search through an Investigation of Notes-to-self and Notes-to-share
Authors: Anita Crescenzi, Yuan Li, Yinglong Zhang, Robert Capra
Venue: SIGIR
Year: 2019

Recently, there has been interest in integrating information retrieval systems more closely with users' knowledge development processes, especially to support exploratory search. In this work, we investigated how people organize and structure information they discover during exploratory searches. In a lab study, we asked 24 participants to take hand-written notes they could use in the future while they were completing an exploratory search. We then asked participants to organize their findings to share with someone else who wants to explore the topic. Finally, we conducted post-session interviews to gain insights into the types of information saved and how participants organized the information they found. In our qualitative analysis of the notes and interviews, we found that the notes included background information about the topic, key concepts, specific details, useful information sources, and information to help with the broader work task. Notes were primarily structured in lists, and reflected a combination of linear note-taking strategies and grouping by information source or topical themes. Participants changed the content and structure of the shared notes to make them easier to understand and to provide a more thorough overview of the topic. Our findings have implications for the design of search tools to help current searchers organize, structure, and synthesize information and to help future searchers engaged in similar information seeking tasks.
ECOM'19: The SIGIR 2019 Workshop on eCommerce
Authors: Jon Degenhardt, Surya Kallumadi, Utkarsh Porwal, Andrew Trotman
Venue: SIGIR
Year: 2019

eCommerce Information Retrieval is receiving increasing attention in the academic literature, and is an essential component of some of the largest web sites (such as eBay, Amazon, Airbnb, Alibaba, Taobao, Target, Facebook, Home Depot, and others). These kinds of organisations clearly value the importance of research into Information Retrieval. The purpose of this workshop is to bring together researchers and practitioners of eCommerce IR to discuss topics unique to it, to set a research agenda, to examine how to build data sets, and how evaluate algorithms for research into this fascinating topic. eCommerce IR is ripe for research and has a unique set of problems. For example, in eCommerce search there may be no hypertext links between documents (products); there is a click stream, but more importantly, there is often a buy stream. eCommerce problems are wide in scope and range from user interaction modalities through to dynamic updates of a rapidly changing collection on auction sites, and the experienceness of some products (such as Airbnb bookings). This workshop is a follow up to very successful workshops held at SIGIR 2017 and SIGIR 2018. This year we will be running a data challenge (sponsored by eBay) which will allow us to follow up on multiple aspects that were discussed in the previous workshops (in particular, deterministic rank orders and how to evaluate these).
Biomedical Heterogeneous Data Integration and Rank Retrieval using Data Bridges
Authors: Priya Deshpande
Venue: SIGIR
Year: 2019

Digitized world demands data integration systems that combine data repositories from multiple data sources. Vast amounts of clinical and biomedical research data are considered a primary force enabling data-driven research toward advancing health research and for introducing efficiencies in healthcare delivery. Data-driven research may have many goals, including but not limited to improved diagnostics processes, novel biomedical discoveries, epidemiology, and education. However, finding and gaining access to relevant data remains an elusive goal. We identified these challenges and developed an Integrated Radiology Image Search (IRIS) framework that could be a step toward aiding data-driven research. We propose building data bridges to support retrieving ranked relevant documents from integrated repository. My research focuses on biomedical data integration and indexing systems and provide ranked document retrieval from an integrated repository. Though we currently focus on integrating biomedical data sources (for medical professionals), we believe that our proposed framework and methodologies can be used in other domains as well.
ENT Rank: Retrieving Entities for Topical Information Needs through Entity-Neighbor-Text Relations
Authors: Laura Dietz
Venue: SIGIR
Year: 2019

Related work has demonstrated the helpfulness of utilizing information about entities in text retrieval; here we explore the converse: Utilizing information about text in entity retrieval. We model the relevance of Entity-Neighbor-Text (ENT) relations to derive a learning-to-rank-entities model. We focus on the task of retrieving (multiple) relevant entities in response to a topical information need such as "Zika fever". The ENT Rank model is designed to exploit semi-structured knowledge resources such as Wikipedia for entity retrieval. The ENT Rank model combines (1) established features of entity-relevance, with (2) information from neighboring entities (co-mentioned or mentioned-on-page) through (3) relevance scores of textual contexts through traditional retrieval models such as BM25 and RM3.
A Scalable Virtual Document-Based Keyword Search System for RDF Datasets
Authors: Dennis Dosso, Gianmaria Silvello
Venue: SIGIR
Year: 2019

RDF datasets are becoming increasingly useful with the development of knowledge-based web applications. SPARQL is the official structured query language to search and access RDF datasets. Despite its effectiveness, the language is often difficult to use for non-experts because of its syntax and the necessity to know the underlying data structure of the database queries. In this regard, keyword search enables non-expert users to access the data contained in RDF datasets intuitively. This work describes the TSA+VDP keyword search system for effective and efficient keyword search over large RDF datasets. The system is compared with other state-of-the-art methods on different datasets, both real-world and synthetic, using a new evaluation framework that is easily reproducible and sharable.
Effective Online Evaluation for Web Search
Authors: Alexey Drutsa, Gleb Gusev, Eugene Kharitonov, Denis Kulemyakin, Pavel Serdyukov, Igor Yashkov
Venue: SIGIR
Year: 2019

We present you a program of a balanced mix between an overview of academic achievements in the field of online evaluation and a portion of unique industrial practical experience shared by both the leading researchers and engineers from global Internet companies. First, we give basic knowledge from mathematical statistics. This is followed by foundations of main evaluation methods such as A/B testing, interleaving, and observational studies. Then, we share rich industrial experiences on constructing of an experimentation pipeline and evaluation metrics (emphasizing best practices and common pitfalls). A large part of our tutorial is devoted to modern and state-of-the-art techniques (including the ones based on machine learning) that allow to conduct online experimentation efficiently. We invite software engineers, designers, analysts, and managers of web services and software products, as well as beginners, advanced specialists, and researchers to learn how to make web service development effectively data-driven.
Retrieving Multi-Entity Associations: An Evaluation of Combination Modes for Word Embeddings
Authors: Gloria Feher, Andreas Spitz, Michael Gertz
Venue: SIGIR
Year: 2019

Word embeddings have gained significant attention as learnable representations of semantic relations between words, and have been shown to improve upon the results of traditional word representations. However, little effort has been devoted to using embeddings for the retrieval of entity associations beyond pairwise relations. In this paper, we use popular embedding methods to train vector representations of an entity-annotated news corpus, and evaluate their performance for the task of predicting entity participation in news events versus a traditional word cooccurrence network as a baseline. To support queries for events with multiple participating entities, we test a number of combination modes for the embedding vectors. While we find that even the best combination modes for word embeddings do not quite reach the performance of the full cooccurrence network, especially for rare entities, we observe that different embedding methods model different types of relations, thereby indicating the potential for ensemble methods.
Find Relevant Cases in All Cases: Your Journey at Doctrine
Authors: Nicolas Fiorini
Venue: SIGIR
Year: 2019

Domain-specific Information Retrieval (IR) is generally challenging because of the rare datasets or benchmarks, niche vocabularies and more limited literature coverage. Legal IR is no exception and presents other obstacles, reinforcing the need for innovation and, sometimes, paradigm shifts. Doctrine, one of the largest Legaltech companies in Europe, dedicates an entire data science team to advance on these problems and identify new opportunities. In this presentation, we provide some intuition regarding the specificities of legal IR (e.g., what is relevance?), and we introduce some of the solutions currently used on doctrine.fr. Particularly, we show how we use named entity recognition in the various forms of contents we host, and how it enhances the search engine. With knowledge extracted from documents, we may built large enough datasets and train learning-to-rank algorithms. This, combined with several specific-domain vocabulary enrichments to increase recall, dramatically improves the search experience for our users.
USEing Transfer Learning in Retrieval of Statistical Data
Authors: Anton Firsov, Vladimir Bugay, Anton Karpenko
Venue: SIGIR
Year: 2019

DSSM-like models showed good results in retrieval of short documents that semantically match the query. However, these models require large collections of click-through data that are not available in some domains. On the other hand, the recent advances in NLP demonstrated the possibility to fine-tune language models and models trained on one set of tasks to achieve a state of the art results on a multitude of other tasks or to get competitive results using much smaller training sets. Following this trend, we combined DSSM-like architecture with USE (Universal Sentence Encoder) and BERT (Bidirectional Encoder Representations from Transformers) models in order to be able to fine-tune them on a small amount of click-through data and use them for information retrieval. This approach allowed us to significantly improve our search engine for statistical data.
Learning Unsupervised Semantic Document Representation for Fine-grained Aspect-based Sentiment Analysis
Authors: Hao-Ming Fu, Pu-Jen Cheng
Venue: SIGIR
Year: 2019

Document representation is the core of many NLP tasks on machine understanding. A general representation learned in an unsupervised manner reserves generality and can be used for various applications. In practice, sentiment analysis (SA) has been a challenging task that is regarded to be deeply semantic-related and is often used to assess general representations. Existing methods on unsupervised document representation learning can be separated into two families: sequential ones, which explicitly take the ordering of words into consideration, and non-sequential ones, which do not explicitly do so. However, both of them suffer from their own weaknesses. In this paper, we propose a model that overcomes difficulties encountered by both families of methods. Experiments show that our model outperforms state-of-the-art methods on popular SA datasets and a fine-grained aspect-based SA by a large margin.
AgentBuddy: an IR System based on Bandit Algorithms to Reduce Cognitive Load for Customer Care Agents
Authors: Hrishikesh Ganu, Mithun Ghosh, Freddy Jose, Shashi Roshan
Venue: SIGIR
Year: 2019

We describe a human-in-the loop system - AgentBuddy, that is helping Intuit improve the quality of search it offers to its internal Customer Care Agents (CCAs). AgentBuddy aims to reduce the cognitive effort on part of the CCAs while at the same time boosting the quality of our legacy federated search system. Under the hood, it leverages bandit algorithms to improve federated search and other ML models like LDA, Siamese networks to help CCAs zero in on high quality search results. An intuitive UI designed ground up working with the users (CCAs) is another key feature of the system. AgentBuddy has been deployed internally and initial results from User Acceptance Trials indicate a 4x lift in quality of highlights compared to the incumbent system.
From Text to Sound: A Preliminary Study on Retrieving Sound Effects to Radio Stories
Authors: Songwei Ge, Curtis Xuan, Ruihua Song, Chao Zou, Wei Liu, Jin Zhou
Venue: SIGIR
Year: 2019

Sound effects play an essential role in producing high-quality radio stories but require enormous labor cost to add. In this paper, we address the problem of automatically adding sound effects to radio stories with a retrieval-based model. However, directly implementing a tag-based retrieval model leads to high false positives due to the ambiguity of story contents. To solve this problem, we introduce a retrieval-based framework hybridized with a semantic inference model which helps to achieve robust retrieval results. Our model relies on fine-designed features extracted from the context of candidate triggers. We collect two story dubbing datasets through crowdsourcing to analyze the setting of adding sound effects and to train and test our proposed methods. We further discuss the importance of each feature and introduce several heuristic rules for the trade-off between precision and recall. Together with the text-to-speech technology, our results reveal a promising automatic pipeline on producing high-quality radio stories.