OpenRaiser/Intern-Atlas huggingface dataset:
VAE++ Variational AutoEncoder for Heterogeneous One-Class Collaborative Filtering
Authors: Wanqi Ma, Xiancong Chen, Weike Pan, Zhong Ming
Venue: Conference WSDM 2022
Year: 2022

Neural network-based models for collaborative filtering have received widespread attention, among which variational autoencoder (VAE) has shown unique advantages in the task of item recommendation. However, most existing VAE-based models only focus on one type of user feedback, leading to their performance bottlenecks. To overcome this limitation, we propose a novel VAE-based recommendation model called VAE++, which can effectively utilize heterogeneous feedback to boost recommendation performance. Specifically, it combines three different types of signals, i.e., purchase feedback, examination feedback and their mixed feedback, via two well-designed modules, i.e., a target representation enhancement module and a target representation refinement module. The former exploits the mixed feedback to improve the learning of purchase representations, while the latter leverages the examination feedback to further refine them. In particular, purchase and examination preferences are jointly decoded in one decoder to ensure the high quality of the reconstructed samples. Extensive experiments on three public datasets show that our VAE++ achieves the best results compared with several state-of-the-art methods.
StruBERT: Structure-aware BERT for Table Search and Matching
Authors: Mohamed Trabelsi, Zhiyu Chen, Shuo Zhang, Brian D. Davison, Jeff Heflin
Venue: Conference WWW 2022
Year: 2022

A large amount of information is stored in data tables. Users can search for data tables using a keyword-based query. A table is composed primarily of data values that are organized in rows and columns providing implicit structural information. A table is usually accompanied by secondary information such as the caption, page title, etc., that form the textual information. Understanding the connection between the textual and structural information is an important yet neglected aspect in table retrieval as previous methods treat each source of information independently. In addition, users can search for data tables that are similar to an existing table, and this setting can be seen as a content-based table retrieval. In this paper, we propose StruBERT, a structure-aware BERT model that fuses the textual and structural information of a data table to produce context-aware representations for both textual and tabular content of a data table. StruBERT features are integrated in a new end-to-end neural ranking model to solve three table-related downstream tasks: keyword- and content-based table retrieval, and table similarity. We evaluate our approach using three datasets, and we demonstrate substantial improvements in terms of retrieval and classification metrics over state-of-the-art methods.
On the Challenges of Podcast Search at Spotify
Authors: Mi Tian, Claudia Hauff, Praveen Chandar
Venue: Conference CIKM 2022
Year: 2022

Online music streaming is enjoying ever-growing popularity in the last decades, enabled by the abundance of music content in digital format and online streaming services. In the recent years, podcasts, as a talk-focused media format, have witnessed a rapid growth among listeners. Podcasts come in many forms and sizes. They range from 20-minute daily meditation sessions, weekly recaps of global news, interviews with celebrities, and hosts bantering with each other for hours. More and more streaming services are now expanding their catalogs to support both music and podcasts on the same platform, such as Amazon Music, Pandora and Spotify. This setup requires an effective aggregated search system to assemble information from heterogeneous information sources and content types (e.g., artist profiles, playlists, podcast shows, etc.) into one result interface in order to support diverse information needs.
News Recommendation with Candidate-Aware User Modeling
Authors: Tao Qi, Fangzhao Wu, Chuhan Wu, Yongfeng Huang
Venue: Conference SIGIR 2022
Year: 2022

News recommendation aims to match news with personalized user interest. Existing methods for news recommendation usually model user interest from historical clicked news without the consideration of candidate news. However, each user usually has multiple interests, and it is difficult for these methods to accurately match a candidate news with a specific user interest. In this paper, we present a candidate-aware user modeling method for personalized news recommendation, which can incorporate candidate news into user modeling for better matching between candidate news and user interest. We propose a candidate-aware self-attention network that uses candidate news as clue to model candidate-aware global user interest. In addition, we propose a candidate-aware CNN network to incorporate candidate news into local behavior context modeling and learn candidate-aware short-term user interest. Besides, we use a candidate-aware attention network to aggregate previously clicked news weighted by their relevance with candidate news to build candidate-aware user representation. Experiments on real-world datasets show the effectiveness of our method in improving news recommendation performance.
Ranking-Incentivized Quality Preserving Content Modification
Authors: Gregory Goren, Oren Kurland, Moshe Tennenholtz, Fiana Raiber
Venue: Conference SIGIR 2020
Year: 2020

The Web is a canonical example of a competitive retrieval setting where many documents' authors consistently modify their documents to promote them in rankings. We present an automatic method for quality-preserving modification of document content --- i.e., maintaining content quality --- so that the document is ranked higher for a query by a non-disclosed ranking function whose rankings can be observed. The method replaces a passage in the document with some other passage. To select the two passages, we use a learning-to-rank approach with a bi-objective optimization criterion: rank promotion and content-quality maintenance. We used the approach as a bot in content-based ranking competitions. Analysis of the competitions demonstrates the merits of our approach with respect to human content modifications in terms of rank promotion, content-quality maintenance and relevance.
Searchx: Empowering collaborative search research
Authors: Sindunuraga Rikarno Putra, Felipe Moraes, Claudia Hauff
Venue: Conference SIGIR 2018
Year: 2018

Collaborative search has been an active area of research within the IR community for many years. While for "single-user'' research a variety of up-to-date open-source search systems exist, few "multi-user'' search tools are open-source and even fewer are being maintained. In this paper, we present SearchX, an open-source collaborative search system we are currently developing-and using for our research. We designed and built SearchX using the modern Web stack (and are thus not siloed by an operating system or a particular browser type), enabling efficient research across platforms (Desktop, mobile) and with online users (e.g. crowdworkers). A video, describing the demo can be found at https: //www.youtube.com/watch?v=uf24m6p3vts.
The Challenge of High Recall in Biomedical Systematic Search
Authors: Sarvnaz Karimi, Justin Zobel, Stefan Pohl, Falk Scholer
Venue: Conference CIKM 2009
Year: 2009

Clinical systematic reviews are based on expert, laborious search of well-annotated literature. Boolean search on bibliographic databases, such as MEDLINE, continues to be the preferred discovery method, but the size of these databases, now approaching 20 million records, makes it impossible to fully trust these searching methods. We are investigating the trade-offs between Boolean and ranked retrieval. Our findings show that although Boolean search has limitations, it is not obvious that ranking is superior, and illustrate that a single query cannot be used to resolve an information need. Our experiments show that a combination of less complicated Boolean queries and ranked retrieval outperforms either of them individually, leading to possible time savings over the current process.
Systematic review automation tools for end-to-end query formulation
Authors: Hang Li 0009, Harrisen Scells, Guido Zuccon
Venue: Conference SIGIR 2020
Year: 2020

Systematic reviews are used widely in the biomedical and healthcare domains. Systematic reviews aim to provide a complete and exhaustive overview of the medical literature for a specific research question. Core to the construction of a systematic review is the search strategy. The main component of a search strategy is a complex Boolean query, typically developed by information specialists (e.g., librarians). The aim of the search strategy is to retrieve relevant studies that will contribute to the outcomes of the systematic review. One barrier information specialists face when developing a search strategy is the enormous amount of medical literature that exists in databases. This vast amount of literature means that search strategies often suffer from biases (e.g., lack of expertise, overconfidence, limited knowledge of the domain) and are incomplete, or retrieve far too many studies (possibly as a result of the biases, but also due to the tools used to develop search strategies). Retrieving too many studies impacts the time and financial costs of the review, and retrieving too few studies may impact the outcomes of the review. Therefore, it is vital to support expert searchers develop effective search strategies. In this paper, we present a novel end-to-end set of advanced tools for information specialists. These tools are tightly integrated into an existing Open Source search strategy refining package (searchrefiner). These tools aim to address the problems associated with search strategy development by providing a complete framework from query development, to refinement, to documentation. The implementation of these tools also offers a glimpse at the ease at which related tools may be implemented within the searchrefiner ecosystem. More information about the tools including installation, documentation, and screenshots is made available on the searchrefiner website: https://ielab.io/searchrefiner.
searchrefiner: A Query Visualisation and Understanding Tool for Systematic Reviews
Authors: Harrisen Scells, Guido Zuccon
Venue: Conference CIKM 2018
Year: 2018

We present an open source tool, searchrefiner, for researchers that conduct medical systematic reviews to assist in formulating, visualising, and understanding Boolean queries. The searchrefiner web interface allows researchers to explore how Boolean queries retrieve citations in existing, popular query syntaxes used in systematic review literature search. The web interface allows researchers to perform tasks such as using validation citations to ensure queries are retrieving a minimum set of known relevant citations, and editing Boolean queries by dragging and dropping clauses in a structured editor. In addition, the tools provided by the searchrefiner interface allow researchers to visualise why the queries they formulate retrieve citations, and ways to understand how to refine queries into more effective ones. searchrefiner is targeted at both experts and novices, as a tool for query formulation and refinement, and as a tool for training users to search for literature to compile systematic reviews. The searchrefiner website located at https://ielab.io/searchrefiner contains information about how to download, install, and use the tool, as well as a link to an online hosted version for demonstration purposes.
Automatic Boolean Query Refinement for Systematic Review Literature Search
Authors: Harrisen Scells, Guido Zuccon, Bevan Koopman
Venue: Conference WWW 2019
Year: 2019

In the medical domain, systematic reviews are a highly trustworthy evidence source used to inform clinical diagnosis and treatment, and governmental policy making. Systematic reviews must be complete in that all relevant literature for the research question of the review must be synthesised in order to produce a recommendation. To identify the literature to screen for inclusion in systematic reviews, information specialists construct complex Boolean queries that capture the information needs defined by the research questions of the systemic review. However, in the quest for total recall, these Boolean queries return many non relevant results. In this paper, we propose automatic methods for Boolean query refinement in the context of systematic review literature retrieval with the aim of alleviating this high-recall, low-precision problem. To do this, we build upon current literature and define additional semantic transformations for Boolean queries in the form of query expansion and reduction. Empirical evaluation is done on a set of real systematic review queries to show how our method performs in a realistic setting. We found that query refinement strategies produced queries that were more effective than the original in terms of six information retrieval evaluation measures. In particular, queries were refined to increase precision, while maintaining, or even increasing, recall - this, in turn, translates into both time and cost savings when creating laborious and expensive systematic reviews.
Automatic Boolean Query Formulation for Systematic Review Literature Search
Authors: Harrisen Scells, Guido Zuccon, Bevan Koopman, Justin Clark
Venue: Conference WWW 2020
Year: 2020

Formulating Boolean queries for systematic review literature search is a challenging task. Commonly, queries are formulated by information specialists using the protocol specified in the review and interactions with the research team. Information specialists have in-depth experience on how to formulate queries in this domain, but may not have in-depth knowledge about the reviews’ topics. Query formulation requires a significant amount of time and effort, and is performed interactively; specialists repeatedly formulate queries, attempt to validate their results, and reformulate specific Boolean clauses. In this paper, we investigate the possibility of automatically formulating a Boolean query from the systematic review protocol. We propose a novel five-step approach to automatic query formulation, specific to Boolean queries in this domain, which approximates the process by which information specialists formulate queries. In this process, we use syntax parsing to derive the logical structure of high-level concepts in a query, automatically extract and map concepts to entities in order to perform entity expansion, and finally apply post-processing operations (such as stemming and search filters). Automatic query formulation for systematic review literature search has several benefits: (i) it can provide reviewers with an indication of the types of studies that will be retrieved, without the involvement of an information specialist, (ii) it can provide information specialists with an initial query to begin the formulation process, (iii) it can provide researchers that perform rapid reviews with a method to quickly perform searches.
MeSH Suggester: A Library and System for MeSH Term Suggestion for Systematic Review Boolean Query Construction
Authors: Shuai Wang, Hang Li, Guido Zuccon
Venue: Conference WSDM 2023
Year: 2023

Boolean query construction is often critical for medical systematic review literature search. To create an effective Boolean query, systematic review researchers typically spend weeks coming up with effective query terms and combinations. One challenge to creating an effective systematic review Boolean query is the selection of effective MeSH Terms to include in the query. In our previous work, we created neural MeSH term suggestion methods and compared them to state-of-the-art MeSH term suggestion methods. We found neural MeSH term suggestion methods to be highly effective. In this demonstration, we build upon our previous work by creating (1) a Web-based MeSH term suggestion prototype system that allows users to obtain suggestions from a number of underlying methods and (2) a Python library that implements ours and others' MeSH term suggestion methods and that is aimed at researchers who want to further investigate, create or deploy such type of methods. We describe the architecture of the web-based system and how to use it for the MeSH term suggestion task. For the Python library, we describe how the library can be used for advancing further research and experimentation, and we validate the results of the methods contained in the library on standard datasets. Our web-based prototype system is available at http://ielab-mesh-suggest.uqcloud.net, while our Python library is at https://github.com/ielab/meshsuggestlib.
Hard negatives or false negatives: Correcting pooling bias in training neural ranking models
Authors: Yinqiong Cai, Jiafeng Guo, Yixing Fan, Qingyao Ai, Ruqing Zhang, Xueqi Cheng
Venue: Conference CIKM 2022
Year: 2022

Neural ranking models (NRMs) have become one of the most important techniques in information retrieval (IR). Due to the limitation of relevance labels, the training of NRMs heavily relies on negative sampling over unlabeled data. In general machine learning scenarios, it has shown that training with hard negatives (i.e., samples that are close to positives) could lead to better performance. Surprisingly, we find opposite results from our empirical studies in IR. When sampling top-ranked results (excluding the labeled positives) as negatives from a stronger retriever, the performance of the learned NRM becomes even worse. Based on our investigation, the superficial reason is that there are more false negatives (i.e., unlabeled positives) in the top-ranked results with a stronger retriever, which may hurt the training process; The root is the existence of pooling bias in the dataset constructing process, where annotators only judge and label very few samples selected by some basic retrievers. Therefore, in principle, we can formulate the false negative issue in training NRMs as learning from labeled datasets with pooling bias. To solve this problem, we propose a novel Coupled Estimation Technique (CET) that learns both a relevance model and a selection model simultaneously to correct the pooling bias for training NRMs. Empirical results on three retrieval benchmarks show that NRMs trained with our technique can achieve significant gains on ranking effectiveness against other baseline strategies.
Towards better measurement of attention and satisfaction in mobile search
Authors: Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam
Venue: Conference SIGIR 2014
Year: 2014

Web Search has seen two big changes recently: rapid growth in mobile search traffic, and an increasing trend towards providing answer-like results for relatively simple information needs (e.g., [weather today]). Such results display the answer or relevant information on the search page itself without requiring a user to click. While clicks on organic search results have been used extensively to infer result relevance and search satisfaction, clicks on answer-like results are often rare (or meaningless), making it challenging to evaluate answer quality. Together, these call for better measurement and understanding of search satisfaction on mobile devices. In this paper, we studied whether tracking the browser viewport (visible portion of a web page) on mobile phones could enable accurate measurement of user attention at scale, and provide good measurement of search satisfaction in the absence of clicks. Focusing on answer-like results in web search, we designed a lab study to systematically vary answer presence and relevance (to the user's information need), obtained satisfaction ratings from users, and simultaneously recorded eye gaze and viewport data as users performed search tasks. Using this ground truth, we identified increased scrolling past answer and increased time below answer as clear, measurable signals of user dissatisfaction with answers. While the viewport may contain three to four results at any given time, we found strong correlations between gaze duration and viewport duration on a per result basis, and that the average user attention is focused on the top half of the phone screen, suggesting that we may be able to scalably and reliably identify which specific result the user is looking at, from viewport data alone.
Contextualizing trending entities in news stories
Authors: Marco Ponza, Diego Ceccarelli, Paolo Ferragina, Edgar Meij, Sambhav Kothari
Venue: Conference WSDM 2021
Year: 2021

Trends are those keywords, phrases, or names that are mentioned most often on social media or in news in a particular timeframe.They are an effective way for human news readers to both discover and stay focused on the most relevant information of the day. In this work, we consider trends that correspond to an entity in a knowledge graph and introduce the new and as-yet unexplored task of identifying other entities that may help explain the "why" an entity is trending. We refer to these retrieved entities as contextual entities. Some of them are more important than others in the context of the trending entity and we thus determine a ranking of entities according to how useful they are in contextualizing the trend.We propose two solutions for ranking contextual entities. The first one is fully unsupervised and based on Personalized PageRank, calculated over a trending entity-specific graph of other entities where the edges encode a notion of directional similarity based on embedded background knowledge. Our second method is based on learning to rank and combines the intuitions behind the unsupervised model with signals derived from hand-crafted features in a supervised setting. We compare our models on this novel task by using a new, purpose-built test collection created using crowdsourcing. Our methods improve over the strongest baseline in terms ofPrecision at 1 by 7% (unsupervised) and 13% (supervised). We find that the salience of a contextual entity and how coherent it is with respect to the news story are strong indicators of relevance in both unsupervised and supervised settings.
Code recommendation for open source software developers
Authors: Yiqiao Jin, Yunsheng Bai, Yanqiao Zhu, Yizhou Sun, Wei Wang
Venue: Conference WWW 2023
Year: 2023

Open Source Software (OSS) is forming the spines of technology infrastructures, attracting millions of talents to contribute. Notably, it is challenging and critical to consider both the developers' interests and the semantic features of the project code to recommend appropriate development tasks to OSS developers. In this paper, we formulate the novel problem of code recommendation, whose purpose is to predict the future contribution behaviors of developers given their interaction history, the semantic features of source code, and the hierarchical file structures of projects. Considering the complex interactions among multiple parties within the system, we propose CODER, a novel graph-based code recommendation framework for open source software developers. CODER jointly models microscopic user-code interactions and macroscopic user-project interactions via a heterogeneous graph and further bridges the two levels of information through aggregation on file-structure graphs that reflect the project hierarchy. Moreover, due to the lack of reliable benchmarks, we construct three large-scale datasets to facilitate future research in this direction. Extensive experiments show that our CODER framework achieves superior performance under various experimental settings, including intra-project, cross-project, and cold-start recommendation. We will release all the datasets, code, and utilities for data retrieval upon the acceptance of this work.