OpenRaiser/Intern-Atlas huggingface dataset:
Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search
Authors: Pepa Atanasova, Georgi Karadzhov, Yasen Kiprov, Preslav Nakov, Fabrizio Sebastiani
Venue: SIGIR
Year: 2019

In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links" metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, along came chatbots, conversational systems, and messaging platforms, where the user needs could be better served with the system asking follow-up questions in order to better understand the user's intent. While typically a user would expect a single response at any utterance, a system could also return multiple options for the user to select from, based on different system understandings of the user's intent. However, this possibility should not be overused, as this practice could confuse and/or annoy the user. How to produce good variable-length lists, given the conflicting objectives of staying short while maximizing the likelihood of having a correct answer included in the list, is an underexplored problem. It is also unclear how to evaluate a system that tries to do that. Here we aim to bridge this gap. In particular, we define some necessary and some optional properties that an evaluation measure fit for this purpose should have. We further show that existing evaluation measures from the IR tradition are not entirely suitable for this setup, and we propose novel evaluation measures that address it satisfactorily.
Building Economic Models and Measures of Search
Authors: Leif Azzopardi, Alistair Moffat, Paul Thomas, Guido Zuccon
Venue: SIGIR
Year: 2019

Economics provides an intuitive and natural way to formally represent the costs and benefits of interacting with applications, interfaces and devices. By using economic models it is possible to reason about interaction, make predictions about how changes to the system will affect behavior, and measure the performance of people's interactions with the system. In this tutorial, we first provide an overview of relevant economic theories, before showing how they can be applied to formulate different ranking principles to provide the optimal ranking to users. This is followed by a session showing how economics can be used to model how people interact with search systems, and how to use these models to generate hypotheses about user behavior. The third session focuses on how economics has been used to underpin the measurement of information retrieval systems and applications using the CWL framework (which reports the expected utility, expected total utility, expected total cost, and so on) -- and how different models of user interaction lead to different metrics. We then show how information foraging theory can be used to measure the performance of an information retrieval system -- connecting the theory of how people search with how we measure it. The final session of the day will be spent building economic models and measures of search. Here sample problems will be provided to challenge participants, or participants can bring their own.
Decoding The Style And Bias of Song Lyrics
Authors: Manash Pratim Barman, Amit Awekar, Sambhav Kothari
Venue: SIGIR
Year: 2019

The central idea of this paper is to gain a deeper understanding of song lyrics computationally. We focus on two aspects: style and biases of song lyrics. All prior works to understand these two aspects are limited to manual analysis of a small corpus of song lyrics. In contrast, we analyzed more than half a million songs spread over five decades. We characterize the lyrics style in terms of vocabulary, length, repetitiveness, speed, and readability. We have observed that the style of popular songs significantly differs from other songs. We have used distributed representation methods and WEAT test to measure various gender and racial biases in the song lyrics. We have observed that biases in song lyrics correlate with prior results on human subjects. This correlation indicates that song lyrics reflect the biases that exist in society. Increasing consumption of music and the effect of lyrics on human emotions makes this analysis important.
Evaluating Risk-Sensitive Text Retrieval
Authors: Rodger Benham
Venue: SIGIR
Year: 2019

Search engines with a loyal user-base face the difficult task of improving overall effectiveness while maintaining the quality of existing work-flows. Risk-sensitive evaluation tools are designed to address that task, but, they currently do not support inference over multiple baselines. Our research objectives are to: 1) Survey and revisit risk evaluation, taking into account frequentist and Bayesian inference approaches for comparing against multiple baselines; 2) Apply that new approach, evaluating a novel web search technique that leverages previously run queries to improve the effectiveness of a new user query; and 3) Explore how risk-sensitive component interactions affect end-to-end effectiveness in a search pipeline.
Network Embedding and Change Modeling in Dynamic Heterogeneous Networks
Authors: Ranran Bian, Yun Sing Koh, Gillian Dobbie, Anna Divoli
Venue: SIGIR
Year: 2019

Network embedding learns the vector representations of nodes. Most real world networks are heterogeneous and evolve over time. There are, however, no network embedding approaches designed for dynamic heterogeneous networks so far. Addressing this research gap is beneficial for analyzing and mining real world networks. We develop a novel representation learning method, change2vec, which considers a dynamic heterogeneous network as snapshots of networks with different time stamps. Instead of processing the whole network at each time stamp, change2vec models changes between two consecutive static networks by capturing newly-added and deleted nodes with their neighbour nodes as well as newly-formed or deleted edges that caused core structural changes known as triad closure or open processes. Change2vec leverages metapath based node embedding and change modeling to preserve both heterogeneous and dynamic features of a network. Experimental results show that change2vec outperforms two state-of-the-art methods in terms of clustering performance and efficiency.
ROME 2019: Workshop on Reducing Online Misinformation Exposure
Authors: Guillaume Bouchard, Guido Caldarelli, Vassilis Plachouras
Venue: SIGIR
Year: 2019

The spread of misinformation online is a challenge that may have an impact on society by misleading and undermining the trust of people in domains such as politics or public health. While fact-checking is one way to identify misinformation, it is a slow process and requires significant effort. Improving the efficiency of fact-checking by automating parts of the process or defining new processes to validate claims is a challenging task with a need for expertise from multiple disciplines. The aim of ROME 2019 is to bring together researchers from various fields such as Information Retrieval, Natural Language Processing, Semantic Web and Complex Networks to discuss these problems and define new directions in the area of automated fact-checking.
Sparse Tensor Co-clustering as a Tool for Document Categorization
Authors: Rafika Boutalbi, Lazhar Labiod, Mohamed Nadif
Venue: SIGIR
Year: 2019

To deal with document clustering, we usually rely on document-term matrices. However, from additional available information like keywords, co-authors, citations we might rather exploit a reorganization of the data in the form of a tensor. In this paper, we extend the use of the Sparse Poisson Latent Block Model to deal with sparse tensor data using jointly all information arising from documents. The proposed model is parsimonious and tailored for this kind of data. To estimate the parameters, we derive a suitable tensor co-clustering algorithm. Empirical results on several real-world text datasets highlight the advantages of our proposal which improves the clustering results of documents.
An Analysis of the Change in Discussions on Social Media with Bitcoin Price
Authors: Andrew Burnie, Emine Yilmaz
Venue: SIGIR
Year: 2019

We develop a new approach to temporalizing word2vec-based topic modelling that determines which topics on social media vary with shifts in the phases of a time series to understand potential interactions. This is particularly relevant for the highly volatile bitcoin price with its distinct four phases across 2017-18. We statistically test which words change in frequency between the different stages and compare four word2vec models to assess their consistency in relating connected words in weighted, undirected graphs. For words that fall in frequency when prices shift from rising to falling, all eight topics are identified with the four approaches; for words rising in frequency, three out of the five topics remain constant. These topics are intuitive and match with actual events in the news.
Similarity-Based Synthetic Document Representations for Meta-Feature Generation in Text Classification
Authors: Sérgio D. Canuto, Thiago Salles, Thierson Couto Rosa, Marcos André Gonçalves
Venue: SIGIR
Year: 2019

We propose new solutions that enhance and extend the already very successful application of meta-features to text classification. Our newly proposed meta-features are capable of: (1) improving the correlation of small pieces of evidence shared by neighbors with labeled categories by means of synthetic document representations and (local and global) hyperplane distances; and (2) estimating the level of error introduced by these newly proposed and the existing meta-features in the literature, specially for hard-to-classify regions of the feature space. Our experiments with large and representative number of datasets show that our new solutions produce the best results in all tested scenarios, achieving gains of up to 12% over the strongest meta-feature proposal of the literature.
Using Trails to Support Users with Tasks of Varying Scope
Authors: Robert Capra, Jaime Arguello
Venue: SIGIR
Year: 2019

A search trail is an interactive visualization of how a previous searcher approached a related task. Using search trails to assist users requires understanding aspects of the task, user, and trails. In this paper, we examine two questions. First, what are task characteristics that influence a user's ability to gain benefits from others' trails? Second, what is the impact of a "mismatch" between a current user's task and previous user's task which originated the trail? We report on a study that investigated the influence of two factors on participants' perceptions and behaviors while using search trails to complete tasks. Our first factor, task scope, focused on the scope of the task assigned to the participant (broad to narrow). Our manipulation of this factor involved varying the number of constraints associated with tasks. Our second factor, trail scope, focused on the scope of the task that originated the search trails given to participants. We investigated how task scope and trail scope affected participants' (RQ1) pre-task perceptions, (RQ2) post-task perceptions, and (RQ3) search behaviors. We discuss implications of our results for systems that use search trails to provide assistance.
Enhanced News Retrieval: Passages Lead the Way!
Authors: Matteo Catena, Ophir Frieder, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto
Venue: SIGIR
Year: 2019

We observe that most relevant terms in unstructured news articles are primarily concentrated towards the beginning and the end of the document. Exploiting this observation, we propose a novel version of the classical BM25 weighting model, called BM25 Passage (BM25P), which scores query results by computing a linear combination of term statistics in the different portions of news articles. Our experimentation, conducted using three publicly available news datasets, demonstrates that BM25P markedly outperforms BM25 in term of effectiveness by up to 17.44% in NDCG@5 and 85% in NDCG@1.
Multiple Query Processing via Logic Function Factoring
Authors: Matteo Catena, Nicola Tonellotto
Venue: SIGIR
Year: 2019

Some extensions to search systems require support for multiple query processing. This is the case with query variations, i.e., different query formulations of the same information need. The results of their processing can be fused together to improve effectiveness, but this requires to traverse more than once the query terms' posting lists, thus prolonging the multiple query processing time. In this work, we propose an approach to optimize the processing of query variations to reduce their overall response time. Similarly to the standard Boolean model, we firstly represent a group of query variations as a logic function where Boolean variables represent query terms. We then apply factoring to such function, in order to produce a more compact but logically equivalent representation. The factored form is used to process the query variations in a single pass over the inverted index. We experimentally show that our approach can improve by up to 1.95× the mean processing time of a multiple query with no statistically significant degradation in terms of NDCG@10.
Harvesting Drug Effectiveness from Social Media
Authors: Zi Chai, Xiaojun Wan, Zhao Zhang, Minjie Li
Venue: SIGIR
Year: 2019

Drug effectiveness describes the capacity of a drug to cure a disease, which is of great importance for drug safety. To get this information, a number of real-world patient-oriented outcomes are required. However, current surveillance systems can only capture a small portion of them, and there is a time lag in processing the reported data. Since social media provides quantities of patient-oriented user posts in real-time, it is of great value to automatically extract drug effectiveness from these data. To this end, we build a dataset containing 25K tweets describing drug use, and further harvest drug effectiveness by performing Relation Extraction (RE) between chemicals and diseases. Most prior works about RE deal with mention pairs independently, which is not suitable for our task since interactions across mention pairs are widespread. In this paper, we propose a model regarding mention pairs as nodes connected by multiple types of edges. With the help of graph-based information transfers over time, it deals with all mention pairs simultaneously to capture their interactions. Besides, a novel idea is used to perform multiple instance learning, a big challenge in general RE tasks. Extensive experimental results show that our model outperforms previous work by a substantial margin.
Neural Compatibility Ranking for Text-based Fashion Matching
Authors: Suthee Chaidaroon, Yi Fang, Min Xie, Alessandro Magnani
Venue: SIGIR
Year: 2019

When shopping for fashion, customers often look for products which can complement their current outfit. For example, customers want to buy a jacket which can go well with their jeans and sneakers. To address the task of fashion matching, we propose a neural compatibility model for ranking fashion products based on the compatibility matching with the input outfit. The contribution of our work is twofold. First, we demonstrate that product descriptions contain rich information about product comparability which has not been fully utilized in the prior work. Secondly, we exploit such useful information from text data by taking advantages of semantic matching and lexical matching both of which are important for fashion matching. The proposed model is evaluated on a real-world fashion outfit dataset and achieves the state-of-the-art results by comparing to the competitive baselines. In the future work, we plan to extend the model by incorporating product images which are the major data source in the prior work on fashion matching.
Developing Evaluation Metrics for Instant Search Using Mixed Methods Methods
Authors: Praveen Chandar, Jean Garcia-Gathright, Christine Hosey, Brian St. Thomas, Jennifer Thom
Venue: SIGIR
Year: 2019

Instant search has become a popular search paradigm in which users are shown a new result page in response to every keystroke triggered. Over recent years, the paradigm has been widely adopted in several domains including personal email search, e-commerce, and music search. However, the topic of evaluation and metrics of such systems has been less explored in the literature thus far. In this work, we describe a mixed methods approach to understanding user expectations and evaluating an instant search system in the context of music search. Our methodology involves conducting a set of user interviews to gain a qualitative understanding of users' behaviors and their expectations. The hypotheses from user research are then extended and verified by a large-scale quantitative analysis of interaction logs. Using music search as a lens, we show that researchers and practitioners can interpret the behavior logs more effectively when accompanied by insights from qualitative research. Further, we also show that user research eliminates the guesswork involved in identifying users signals that estimate user satisfaction. Finally, we demonstrate that metrics identified using our approach are more sensitive than the commonly used click-through rate metric for instant search.
Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019)
Authors: Muthu Kumar Chandrasekaran, Philipp Mayr, Michihiro Yasunaga, Dayne Freitag, Dragomir R. Radev, Min-Yen Kan
Venue: SIGIR
Year: 2019

The deluge of scholarly publication poses a challenge for scholars find relevant research and policy makers to seek in-depth information and understand research impact. Information retrieval (IR), natural language processing (NLP) and bibliometrics could enhance scholarly search, retrieval and user experience, but their use in digital libraries is not widespread. To address this gap, we propose the 4th Joint Workshop on BIRNDL and the 5th CL-SciSumm Shared Task. We seek to foster collaboration among researchers in NLP, IR and Digital Libraries (DL), and to stimulate the development of new methods in NLP, IR, recommendation systems and scientometrics toward improved scholarly document understanding, analysis, and retrieval at scale.
Personalized Fashion Recommendation with Visual Explanations based on Multimodal Attention Network: Towards Visually Explainable Recommendation
Authors: Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, Hongyuan Zha
Venue: SIGIR
Year: 2019

Fashion recommendation has attracted increasing attention from both industry and academic communities. This paper proposes a novel neural architecture for fashion recommendation based on both image region-level features and user review information. Our basic intuition is that: for a fashion image, not all the regions are equally important for the users, i.e., people usually care about a few parts of the fashion image. To model such human sense, we learn an attention model over many pre-segmented image regions, based on which we can understand where a user is really interested in on the image, and correspondingly, represent the image in a more accurate manner. In addition, by discovering such fine-grained visual preference, we can visually explain a recommendation by highlighting some regions of its image. For better learning the attention model, we also introduce user review information as a weak supervision signal to collect more comprehensive user preference. In our final framework, the visual and textual features are seamlessly coupled by a multimodal attention network. Based on this architecture, we can not only provide accurate recommendation, but also can accompany each recommended item with novel visual explanations. We conduct extensive experiments to demonstrate the superiority of our proposed model in terms of Top-N recommendation, and also we build a collectively labeled dataset for evaluating our provided visual explanations in a quantitative manner.
Answer-enhanced Path-aware Relation Detection over Knowledge Base
Authors: Daoyuan Chen, Min Yang, Hai-Tao Zheng, Yaliang Li, Ying Shen
Venue: SIGIR
Year: 2019

Knowledge Based Question Answering (KBQA) is one of the most promising approaches to provide suitable answers for the queries posted by users. Relation detection that aims to take full advantage of the substantial knowledge contained in knowledge base (KB) becomes increasingly important. Significant progress has been made in performing relation detection over KB. However, recent deep neural networks that achieve the state of the art on KB-based relation detection task only consider the context information of question sentences rather than the relatedness between question and answer candidates, and exclusively extract the relation from KB triple rather than learn informative relational path. In this paper, we propose a Knowledge-driven Relation Detection network (KRD) to interactively learn answer-enhanced question representations and path-aware relation representations for relation detection. A Siamese LSTM is employed into a similarity matching process between the question representation and relation representation. Experimental results on the SimpleQuestions and WebQSP datasets demonstrate that KRD outperforms the state-of-the-art methods. In addition, a series of ablation test show the robust superiority of the proposed method.
Information Cascades Modeling via Deep Multi-Task Learning
Authors: Xueqin Chen, Kunpeng Zhang, Fan Zhou, Goce Trajcevski, Ting Zhong, Fengli Zhang
Venue: SIGIR
Year: 2019

Effectively modeling and predicting the information cascades is at the core of understanding the information diffusion, which is essential for many related downstream applications, such as fake news detection and viral marketing identification. Conventional methods for cascade prediction heavily depend on the hypothesis of diffusion models and hand-crafted features. Owing to the significant recent successes of deep learning in multiple domains, attempts have been made to predict cascades by developing neural networks based approaches. However, the existing models are not capable of capturing both the underlying structure of a cascade graph and the node sequence in the diffusion process which, in turn, results in unsatisfactory prediction performance. In this paper, we propose a deep multi-task learning framework with a novel design of shared-representation layer to aid in explicitly understanding and predicting the cascades. As it turns out, the learned latent representation from the shared-representation layer can encode the structure and the node sequence of the cascade very well. Our experiments conducted on real-world datasets demonstrate that our method can significantly improve the prediction accuracy and reduce the computational cost compared to state-of-the-art baselines.