OpenRaiser/Intern-Atlas huggingface dataset:
Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me
Authors: Gordon V. Cormack, Maura R. Grossman
Venue: SIGIR
Year: 2017

Technology-assisted review ("TAR") systems seek to achieve "total recall"; that is, to approach, as nearly as possible, the ideal of 100% recall and 100% precision, while minimizing human review effort. The literature reports that TAR methods using relevance feedback can achieve considerably greater than the 65% recall and 65% precision reported by Voorhees as the "practical upper bound on retrieval performance... since that is the level at which humans agree with one another" (Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 2000). This work argues that in order to build - as well as to, evaluate - TAR systems that approach 100% recall and 100% precision, it is necessary to model human assessment, not as absolute ground truth, but as an indirect indicator of the amorphous property known as "relevance." The choice of model impacts both the evaluation of system effectiveness, as well as the simulation of relevance feedback. Models are presented that better fit available data than the infallible ground-truth model. These models suggest ways to improve TAR-system effectiveness so that hybrid human-computer systems can improve on both the accuracy and efficiency of human review alone. This hypothesis is tested by simulating TAR using two datasets: the TREC 4 AdHoc collection, and a dataset consisting of 401,960 email messages that were manually reviewed and classified by a single individual, Roger, in his official capacity as Senior State Records Archivist. The results using the TREC 4 data show that TAR achieves higher recall and higher precision than the assessments by either of two independent NIST assessors, and blind adjudication of the email dataset, conducted by Roger, more than two years after his original review, shows that he could have achieved the same recall and better precision, while reviewing substantially fewer than 401,960 emails, had he employed TAR in place of exhaustive manual review.
SIGIR 2017 Workshop on Neural Information Retrieval (Neu-IR'17)
Authors: Nick Craswell, W. Bruce Croft, Maarten de Rijke, Jiafeng Guo, Bhaskar Mitra
Venue: SIGIR
Year: 2017

In recent years, deep neural networks have yielded significant performance improvements in application areas such as speech recognition, computer vision, and machine translation. This has led to expectations in the information retrieval (IR) community that these novel machine learning approaches are likely to demonstrate a similar scale of breakthroughs on IR tasks within the next couple of years. In the Neu-IR (pronounced "new IR") 2016 workshop, however, there was a growing concern that the lack of availability of large scale training and evaluation datasets may be hindering the research community from making adequate progress in this area. It was also highlighted that the community would benefit from establishing a shared public repository of neural IR models and shared evaluation resources for better reproducibility and speed of experimentation. After the first successful Neu-IR workshop at SIGIR 2016, our goal this year will be to host a highly interactive full-day workshop to bring the neural IR community together to specifically address these key challenges facing this line of research. The workshop will request the community to submit proposals on generating large scale benchmark collections, building a shared model repository, and standardizing frameworks appropriate for evaluating deep neural network models. In addition, the workshop will provide a forum for the growing community of IR researchers to present their recent (published and unpublished) work involving (shallow or deep) neural network based approaches in an interactive poster session.
Learning To Rank Resources
Authors: Zhuyun Dai, Yubin Kim, Jamie Callan
Venue: SIGIR
Year: 2017

We present a learning-to-rank approach for resource selection. We develop features for resource ranking and present a training approach that does not require human judgments. Our method is well-suited to environments with a large number of resources such as selective search, is an improvement over the state-of-the-art in resource selection for selective search, and is statistically equivalent to exhaustive search even for recall-oriented metrics such as [email protected], an area in which selective search was lacking.
Gauging the Quality of Relevance Assessments using Inter-Rater Agreement
Authors: Tadele Tedla Damessie, Thao P. Nghiem, Falk Scholer, J. Shane Culpepper
Venue: SIGIR
Year: 2017

In recent years, gathering relevance judgments through non-topic originators has become an increasingly important problem in Information Retrieval. Relevance judgments can be used to measure the effectiveness of a system, and are often needed to build supervised learning models in learning-to-rank retrieval systems. The two most popular approaches to gathering bronze level judgments - where the judge is not the originator of the information need for which relevance is being assessed, and is not a topic expert - is through a controlled user study, or through crowdsourcing. However, judging comes at a cost (in time, and usually money) and the quality of the judgments can vary widely. In this work, we directly compare the reliability of judgments using three different types of bronze assessor groups. Our first group is a controlled Lab group; the second and third are two different crowdsourcing groups, CF-Document where assessors were free to judge any number of documents for a topic, and CF-Topic where judges were required to judge all of the documents from a single topic, in a manner similar to the Lab group. Our study shows that Lab assessors exhibit a higher level of agreement with a set of ground truth judgments than CF-Topic and CF-Document assessors. Inter-rater agreement rates show analogous trends. These finding suggests that in the absence of ground truth data, agreement between assessors can be used to reliably gauge the quality of relevance judgments gathered from secondary assessors, and that controlled user studies are more likely to produce reliable judgments despite being more costly.
SIGIR 2017 Workshop on eCommerce (ECOM17)
Authors: Jon Degenhardt, Surya Kallumadi, Maarten de Rijke, Luo Si, Andrew Trotman, Yinghui Xu
Venue: SIGIR
Year: 2017

eCommerce Information Retrieval has received little attention in the academic literature, yet it is an essential component of some of the largest web sites (such as eBay, Amazon, Airbnb, Alibaba, Taobao, Target, Facebook, and others). SIGIR has for several years seen sponsorship from these kinds of organizations, who clearly value the importance of research into Information Retrieval. This workshop brings together researchers and practitioners of eCommerce IR to discuss topics unique to it, to set a research agenda, and to examine how to build a dataset for research into this fascinating topic. eCommerce IR is ripe for research and has a unique set of problems. For example, in eCommerce search there may be no hypertext links between documents (products); there is a click stream, but more importantly, there is often a buy stream. eCommerce problems are wide in scope and range from user interaction modalities (the kinds of search seen in when buying are different from those of web-page search (i.e. it is not clear how shopping and buying relate to the standard web-search interaction models)) through to dynamic updates of a rapidly changing collection on auction sites, and the experienceness of some products (such as Airbnb bookings).
Neural Ranking Models with Weak Supervision
Authors: Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, W. Bruce Croft
Venue: SIGIR
Year: 2017

Despite the impressive improvements achieved by unsupervised deep neural networks in computer vision and NLP tasks, such improvements have not yet been observed in ranking for information retrieval. The reason may be the complexity of the ranking problem, as it is not obvious how to learn from queries and documents when no supervised signal is available. Hence, in this paper, we propose to train a neural ranking model using weak supervision, where labels are obtained automatically without human annotators or any external resources (e.g., click data). To this aim, we use the output of an unsupervised ranking model, such as BM25, as a weak supervision signal. We further train a set of simple yet effective ranking models based on feed-forward neural networks. We study their effectiveness under various learning scenarios (point-wise and pair-wise models) and using different input representations (i.e., from encoding query-document pairs into dense/sparse vectors to using word embedding representation). We train our networks using tens of millions of training instances and evaluate it on two standard collections: a homogeneous news collection (Robust) and a heterogeneous large-scale web collection (ClueWeb). Our experiments indicate that employing proper objective functions and letting the networks to learn the input representation based on weakly supervised data leads to impressive performance, with over 13% and 35% MAP improvements over the BM25 model on the Robust and the ClueWeb collections. Our findings also suggest that supervised neural ranking models can greatly benefit from pre-training on large amounts of weakly labeled data that can be easily obtained from unsupervised IR models.
Spotify: Music Access At Scale
Authors: Fernando Diaz
Venue: SIGIR
Year: 2017

Spotify provides users with access to a massive repository of streaming music. While some aspects of music access are familiar to the information retrieval community (e.g. semistructured data, item recommendation), nuances of the music domain require the development of new models of user understanding, intent modeling, relevance, and content understanding. These models can be studied using the large amount of content and usage data at Spotify, allowing us to extend previous results in the music information retrieval community. In this presentation, we will highlight the research involved in developing Spotify and outline a research program for large scale music access.
The First Workshop on Knowledge Graphs and Semantics for Text Retrieval and Analysis (KG4IR)
Authors: Laura Dietz, Chenyan Xiong, Edgar Meij
Venue: SIGIR
Year: 2017

Knowledge graphs have been used throughout the history of information retrieval for a variety of tasks. Technological advances in knowledge acquisition and alignment technology from the last few years gave rise to a body of new approaches for utilizing knowledge graphs in text retrieval tasks. It is therefore time to consolidate the community efforts in studying how knowledge graph technology can be employed in information retrieval systems in the most effective way. It is also time to start a dialogue with researchers working on knowledge acquisition and alignment to ensure that resulting technologies and algorithms meet the demands posed by information retrieval tasks. The goal of this workshop is to bring together a community of researchers and practitioners who are interested in using, aligning, and constructing knowledge graphs and similar semantic resources for information retrieval applications.
POI Popularity Prediction via Hierarchical Fusion of Multiple Social Clues
Authors: Yaqian Duan, Xinze Wang, Yang Yang, Zi Huang, Ning Xie, Heng Tao Shen
Venue: SIGIR
Year: 2017

Predicting the popularity of Point of Interest (POI) has become increasingly crucial for location-based services, such as POI recommendation. Most of the existing methods can seldom achieve satisfactory performance due to the scarcity of POI's information, which tendentiously confines the recommendation to popular scenic spots, and ignores the unpopular attractions with potentially precious values. In this paper, we propose a novel approach, termed Hierarchical Multi-Clue Fusion (HMCF), for predicting the popularity of POIs. Specifically, we devise an effective hierarchy to comprehensively describe POI by integrating various types of media information (e.g., image and text) from multiple social sources. For each individual POI, we simultaneously inject semantic knowledge as well as multi-clue representative power. We collect a multi-source POI dataset from four widely-used tourism platforms. Extensive experimental results show that the proposed method can significantly improve the performance of predicting the attractions' popularity as compared to several baselines.
Neural Citation Network for Context-Aware Citation Recommendation
Authors: Travis Ebesu, Yi Fang
Venue: SIGIR
Year: 2017

The accelerating rate of scientific publications makes it difficult to find relevant citations or related work. Context-aware citation recommendation aims to solve this problem by providing a curated list of high-quality candidates given a short passage of text. Existing literature adopts bag-of-word representations leading to the loss of valuable semantics and lacks the ability to integrate metadata or generalize to unseen manuscripts in the training set. We propose a flexible encoder-decoder architecture called Neural Citation Network (NCN), embodying a robust representation of the citation context with a max time delay neural network, further augmented with an attention mechanism and author networks. The recurrent neural network decoder consults this representation when determining the optimal paper to recommend based solely on its title. Quantitative results on the large-scale CiteSeer dataset reveal NCN cultivates a significant improvement over competitive baselines. Qualitative evidence highlights the effectiveness of the proposed end-to-end neural network revealing a promising research direction for citation recommendation.
A Metric for Sentence Ordering Assessment Based on Topic-Comment Structure
Authors: Liana Ermakova, Josiane Mothe, Anton Firsov
Venue: SIGIR
Year: 2017

Sentence ordering (SO) is a key component of verbal ability. It is also crucial for automatic text generation. While numerous researchers developed various methods to automatically evaluate the informativeness of the produced contents, the evaluation of readability is usually performed manually. In contrast to that, we present a self-sufficient metric for SO assessment based on text topic-comment structure. We show that this metric has high accuracy.
BioNex: A System For Biomedical News Event Exploration
Authors: Patrick Ernst, Arunav Mishra, Avishek Anand, Vinay Setty
Venue: SIGIR
Year: 2017

We demonstrate BioNex, a system to mine, rank and visualize biomedical news events. BioNex takes biomedical queries such as "Ebola virus disease" and retrieves the k most relevant news events for them. To achieve this we first mine the generic news events by clustering them on a daily basis using general named entities and textual features. These clusters are also tagged with disambiguated biomedical entities which aid in biomedical news event exploration. The clusters are then used to compute the importance scores for the event clusters based on a combination of textual, semantic, popularity and historical importance features. BioNex also visualizes the retrieved event clusters to highlight the top news events and corresponding news articles for the given query. The visualization also provides the context for news events using (1) a chain of historically relevant news event clusters, and (2) other non-biomedical events from the same day.
LiveMaps: Converting Map Images into Interactive Maps
Authors: Michael R. Evans, Dragomir Yankov, Pavel Berkhin, Pavel Yudin, Florin Teodorescu, Wei Wu
Venue: SIGIR
Year: 2017

Image search is a popular application on web search engines. Issuing a location-related query in image search engines often returns multiple images of maps among the top ranked results. Traditionally, clicking on such images either opens the image in a new browser tab or takes users to a web page containing the image. However, finding the area of intent on an interactive web map is a manual process. In this paper, we describe a novel system, LiveMaps, for analyzing and retrieving an appropriate map viewport for a given image of a map. This allows annotation of images of maps returned by image search engines, allowing users to directly open a link to an interactive map centered on the location of interest.
Cross-Domain Recommendation via Clustering on Multi-Layer Graphs
Authors: Aleksandr Farseev, Ivan Samborskii, Andrey Filchenkov, Tat-Seng Chua
Venue: SIGIR
Year: 2017

Venue category recommendation is an essential application for the tourism and advertisement industries, wherein it may suggest attractive localities within close proximity to users' current location. Considering that many adults use more than three social networks simultaneously, it is reasonable to leverage on this rapidly growing multi-source social media data to boost venue recommendation performance. Another approach to achieve higher recommendation results is to utilize group knowledge, which is able to diversify recommendation output. Taking into account these two aspects, we introduce a novel cross-network collaborative recommendation framework C3R, which utilizes both individual and group knowledge, while being trained on data from multiple social media sources. Group knowledge is derived based on new cross-source user community detection approach, which utilizes both inter-source relationship and the ability of sources to complement each other. To fully utilize multi-source multi-view data, we process user-generated content by employing state-of-the-art text, image, and location processing techniques. Our experimental results demonstrate the superiority of our multi-source framework over state-of-the-art baselines and different data source combinations. In addition, we suggest a new approach for automatic construction of inter-network relationship graph based on the data, which eliminates the necessity of having pre-defined domain knowledge.
Unsupervised Query-Focused Multi-Document Summarization using the Cross Entropy Method
Authors: Guy Feigenblat, Haggai Roitman, Odellia Boni, David Konopnicki
Venue: SIGIR
Year: 2017

We present a novel unsupervised query-focused multi-document summarization approach. To this end, we generate a summary by extracting a subset of sentences using the Cross-Entropy (CE) Method. The proposed approach is generic and requires no domain knowledge. Using an evaluation over DUC 2005-2007 datasets with several other state-of-the-art baseline methods, we demonstrate that, our approach is both effective and efficient.
On Including the User Dynamic in Learning to Rank
Authors: Nicola Ferro, Claudio Lucchese, Maria Maistro, Raffaele Perego
Venue: SIGIR
Year: 2017

Ranking query results effectively by considering user past behaviour and preferences is a primary concern for IR researchers both in academia and industry. In this context, LtR is widely believed to be the most effective solution to design ranking models that account for user-interaction features that have proved to remarkably impact on IR effectiveness. In this paper, we explore the possibility of integrating the user dynamic directly into the LtR algorithms. Specifically, we model with Markov chains the behaviour of users in scanning a ranked result list and we modify Lambdamart, a state-of-the-art LtR algorithm, to exploit a new discount loss function calibrated on the proposed Markovian model of user dynamic. We evaluate the performance of the proposed approach on publicly available LtR datasets, finding that the improvements measured over the standard algorithm are statistically significant.
Traditional IR Meets Ontology Engineering in Search for Data
Authors: Anton Firsov
Venue: SIGIR
Year: 2017

The amount of available data grows every day. The data can help to make better decisions. However, with growing volume and variety it becomes increasingly more difficult to find the necessary data. Traditional search engines such as ElasticSearch or Apache Solr are primarily designed to search for text documents. Whereas a search for data has its own specifics: there is less text and more structure. Knoema's search engine is designed specifically to search for data by leveraging data's structure in order to get better results compared to document-oriented search engines.
Support for Interactive Identification of Mentioned Entities in Conversational Speech
Authors: Ning Gao, Douglas W. Oard, Mark Dredze
Venue: SIGIR
Year: 2017

Searching conversational speech poses several new challenges, among which is how the searcher will make sense of what they find. This paper describes our initial experiments with a freely available collection of Enron telephone conversations. Our goal is to help the user make sense of search results by finding information about mentioned people, places and organizations. Because automated entity recognition is not yet sufficiently accurate on conversational telephone speech, we ask the user to transcribe just the name, and to indicate where in the recording it was heard. We then seek to link that mention to other mentions of the same entity in a variety of sources (in our experiments, in email and in Wikipedia). We cast this as an entity linking problem, and achieve promising results by utilizing social network features to help compensate for the limited accuracy of automatic transcription for this challenging content.
Counter Deanonymization Query: H-index Based k-Anonymization Privacy Protection for Social Networks
Authors: Jianliang Gao, Bo Song, Zheng Chen, Weimao Ke, Wanying Ding, Xiaohua Hu
Venue: SIGIR
Year: 2017

In this paper, we propose a novel k-anonymization scheme to counter deanonymization queries on social networks. With this scheme, all entities are protected by k-anonymization, which means the attackers cannot re-identify a target with confidence higher than 1/k. The proposed scheme minimizes the modification on original networks, and accordingly maximizes the utility preservation of published data while achieving k-anonymization privacy protection. Extensive experiments on real data sets demonstrate the effectiveness of the proposed scheme, where the efficacy of the k-anonymized networks is verified with the distributions of pagerank, betweenness, and their Kolmogorov-Smirnov (K-S) test.
Target Type Identification for Entity-Bearing Queries
Authors: Darío Garigliotti, Faegheh Hasibi, Krisztian Balog
Venue: SIGIR
Year: 2017

Identifying the target types of entity-bearing queries can help improve retrieval performance as well as the overall search experience. In this work, we address the problem of automatically detecting the target types of a query with respect to a type taxonomy. We propose a supervised learning approach with a rich variety of features. Using a purpose-built test collection, we show that our approach outperforms existing methods by a remarkable margin.
Event Detection on Curated Tweet Streams
Authors: Nimesh Ghelani, Salman Mohammed, Shine Wang, Jimmy Lin
Venue: SIGIR
Year: 2017

We present a system for identifying interesting social media posts on Twitter and delivering them to users' mobile devices in real time as push notifications. In our problem formulation, users are interested in broad topics such as politics, sports, and entertainment: our system processes tweets in real time to identify relevant, novel, and salient content. There are three interesting aspects to our work: First, instead of attempting to tame the cacophony of unfiltered tweets, we exploit a smaller, but still sizeable, collection of curated tweet streams corresponding to the Twitter accounts of different media outlets. Second, we apply distant supervision to extract topic labels from curated streams that have a specific focus, which can then be leveraged to build high-quality topic classifiers essentially "for free". Finally, our system delivers content via Twitter direct messages, supporting in situ interactions modeled after conversations with intelligent agents. These ideas are demonstrated in an end-to-end working prototype.