OpenRaiser/Intern-Atlas huggingface dataset:
Fast Attention-based Learning-To-Rank Model for Structured Map Search
Authors: Chiqun Zhang, Michael R. Evans, Max Lepikhin, Dragomir Yankov
Venue: SIGIR
Year: 2021

Recent works show that Transformer-based learning-to-rank (LTR) approaches can outperform previous well-established ranking methods, such as gradient-boosted decision trees (GBDT), on document and passage re-ranking problems. A common assumption in these works is that the query and the result documents are comprised of purely textual information without explicit structure. In map search, the relevance of results is determined based on rich heterogeneous features - textual features derived from the query and the results, geospatial features such as proximity of a result to the user, structured features reflecting the address format of the result, and the perceived structure of the query. In this work, we propose a novel deep neural network LTR architecture, capable of seamlessly handling heterogeneous inputs, similar to GBDT-based methods. At the same time, unlike GBDT, the architecture does not require human input via (numerous) carefully-crafted features. Instead, features are inferred through a self-attention mechanism. Our model implements two lightweight attention layers optimized for ranking: the first layer computes query-result similarities, the second implements listwise ranking inference. We perform evaluation on several single language and one multilingual dataset. Our model outperforms by a wide margin other Transformer-based ranking architectures and has equal or better performance than GBDT models. Equally important, runtime inference is orders of magnitude faster than other Transformer architectures, significantly reducing hardware serving costs. The model is a low-cost alternative suitable to power ranking in industrial map search engines across a variety of languages and markets.
A Web-based Knowledge Hub for Exploration of Multiple Research Article Collections
Authors: Wei Emma Zhang, Miao Liu, Alan Pallath, Gokul Tamilventhan
Venue: SIGIR
Year: 2021

Medical decision-making is guided by the results of rich medical research and clinical trials. Doctors, practitioners and researchers urgently need to get updated by the most recent research and clinical outputs to make correct decisions, especially when a new virus such as Covid-19 is causing a global epidemic. However, medical literature for a certain topic could be from different aspects thus archived in different literature databases, resulting in the searching for all related articles become a laborious and time-consuming task. It becomes worse when there is a rapid growth of the number of published literature for the given topic. In this work, we build an online knowledge hub (http://covid19knowledgehub.herokuapp.com/) particularly for Covid-19 related research articles per the requirement from researchers in a hospital. The system is built on top of nine medical research article databases, which covers a wide range of medical aspects. It allows users to easily retrieve and explore the articles from multiple literature databases at one-stop. The system also provides the statistics of article distributions to offer an overview of the status of research under this topic. This real-demand driven system is deployed in a research team of Renmin Hospital, Wuhan University, and largely reduces their time for searching the latest articles. Although this project focuses on Covid-19 related research articles, the approach at the back could be applied to any topic in any domain.
Chatty Goose: A Python Framework for Conversational Search
Authors: Edwin Zhang, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, Rodrigo Nogueira 0001, Jimmy Lin
Venue: SIGIR
Year: 2021

Chatty Goose is an open-source Python conversational search framework that provides strong, reproducible reranking pipelines built on recent advances in neural models. The framework comprises extensible modular components that integrate with popular libraries such as Transformers by HuggingFace and ParlAI by Facebook. Our aim is to lower the barrier of entry for research in conversational search by providing reproducible baselines that researchers can build on top of. We provide an overview of the framework and demonstrate how to instantiate a new system from scratch. Chatty Goose incorporates improvements to components that we introduced in the TREC 2019 Conversational Assistance Track (CAsT), where our submission represented the top-performing system. Using our framework, a comparable run can be reproduced with just a few lines of code.
Binary Neural Network Hashing for Image Retrieval
Authors: Wanqian Zhang, Dayan Wu, Yu Zhou 0015, Bo Li 0063, Weiping Wang 0005, Dan Meng 0002
Venue: SIGIR
Year: 2021

Hashing has become increasingly important for large-scale image retrieval, of which the low storage cost and fast searching are two key properties. However, existing methods adopt large neural networks, which are hard to be deployed in resource-limited devices due to the unacceptable memory and runtime overhead. We address that this huge overhead of neural networks somewhatviolates the appealing properties of hashing. In this paper, we propose a novel deep hashing method, called Binary Neural Network Hashing (BNNH) for fast image retrieval. Specifically, we construct an efficient binarized network architecture to provide lighter model and faster inference, which directly generates binary outputs as the desired hash codes without introducing the quantization loss. Besides, in order to circumvent the huge performance degradation caused by the extremely quantized activations, we introduce a simple yet effective activation-aware loss to explicitly guide the updating of activations in intermediate layers. Extensive experiments conducted on three benchmarks show that the proposed method outperforms the state-of-the-art binarization methods by large margins and validate the efficiency of BNNH.
RLNF: Reinforcement Learning based Noise Filtering for Click-Through Rate Prediction
Authors: Pu Zhao 0004, Chuan Luo 0002, Cheng Zhou, Bo Qiao 0001, Jiale He, Liangjie Zhang, Qingwei Lin
Venue: SIGIR
Year: 2021

Click-through rate (CTR) prediction aims to recall the advertisements that users are interested in and to lead users to click, which is of critical importance for a variety of online advertising systems. In practice, CTR prediction is generally formulated as a conventional binary classification problem, where the clicked advertisements are positive samples and the others are negative samples. However, directly treating unclicked advertisements as negative samples would suffer from the severe label noise issue, since there exist many reasons why users are interested in a few advertisements but do not click. To address such serious issue, we propose a reinforcement learning based noise filtering approach, dubbed RLNF, which employs a noise filter to select effective negative samples. In RLNF, such selected, effective negative samples can be used to enhance the CTR prediction model, and meanwhile the effectiveness of the noise filter can be enhanced through reinforcement learning using the performance of CTR prediction model as reward. Actually, by alternating the enhancements of the noise filter and the CTR prediction model, the performance of both the noise filter and the CTR prediction model is improved. In our experiments, we equip 7 state-of-the-art CTR prediction models with RLNF. Extensive experiments on a public dataset and an industrial dataset present that RLNF significantly improves the performance of all these 7 CTR prediction models, which indicates both the effectiveness and the generality of RLNF.
Robust Neural Text Classification and Entailment via Mixup Regularized Adversarial Training
Authors: Jiahao Zhao 0001, Penghui Wei, Wenji Mao
Venue: SIGIR
Year: 2021

Recent studies show that neural models for natural language processing are usually fragile under adversarial attacks (e.g., character-level insertion and word-level synonym substitution), which exposes the lack of robustness. Most defense techniques are tailored to specific semantic level attacks and do not possess the ability to mitigate multi-level attack simultaneously. Adversarial training has been shown the effectiveness of increasing model robustness. However, it often suffers from degradation on normal data, especially when the proportion of adversarial examples increase. To address this, we propose mixup regularized adversarial training (MRAT) against multi-level attack. Our method can utilize multiple adversarial examples to increase model intrinsic robustness without sacrificing the performance on normal data. We evaluate our method on text classification and entailment tasks. Experimental results on different text encoders (BERT, LSTM and CNN) with multi-level attack show that our method outperforms adversarial training consistently.
PYA0: A Python Toolkit for Accessible Math-Aware Search
Authors: Wei Zhong, Jimmy Lin
Venue: SIGIR
Year: 2021

Mathematical Information Retrieval (MIR) has been actively studied in recent years and many fruitful results have emerged. Among those, the Approach Zero system is one of the few math-aware search engines that is able to perform substructure matching efficiently. Furthermore, it has been deployed in ARQMath2020, the most recent community-wide MIR evaluation, as a strong baseline due to its empirical effectiveness and ability to handle structured math content. However, in order to implement a retrieval model that handles structured queries efficiently, Approach Zero is written in C from the ground up, requiring special pipelines for processing math content and queries. Thus, the system is not conveniently accessible and reusable to the community as a research tool. In this paper, we present PyA0, an easy-to-use Python toolkit built on Approach Zero that improves its accessibility to researchers. We introduce the toolkit interface and report evaluation results on popular MIR datasets to demonstrate the effectiveness and efficiency of our toolkit. We have made PyA0 source code publicly accessible at https://github.com/approach0/pya0, which includes a link to a notebook demo.
Temporal Augmented Graph Neural Networks for Session-Based Recommendations
Authors: Huachi Zhou, Qiaoyu Tan, Xiao Huang 0001, Kaixiong Zhou, Xiaoling Wang
Venue: SIGIR
Year: 2021

Session-based recommendation aims to predict the next item that is most likely to be clicked by an anonymous user, based on his/her clicking sequence within one visit. It becomes an essential function of many recommender systems since it protects privacy. However, as the accumulated session records keep increasing, it becomes challenging to model the user interests since they would drift when the time span is large. Efforts have been devoted to handling dynamic user interests by modeling all historical sessions at one time or conducting offline retraining regularly. These solutions are far from practical requirements in terms of efficiency and capturing timely user interests. To this end, we propose a memory-efficient framework - TASRec. It constructs a graph for each day to model the relations among items. Thus, the same item on different days could have different neighbors, corresponding to the drifting user interests. We design a tailored graph neural network to embed this dynamic graph of items and learn temporal augmented item representations. Based on this, we leverage a sequential neural architecture to predict the next item of a given sequence. Experiments on real-world datasets demonstrate that TASRec outperforms state-of-the-art session-based recommendation methods.
ECG Data Modeling and Analyzing via Deep Representation Learning and Nonparametric Hidden Markov Models
Authors: Jiaojiao Zhu, Wentao Fan 0001
Venue: SIGIR
Year: 2021

In modern clinical medicine, electrocardiogram (ECG) is a common diagnosis technique of cardiovascular diseases. The purpose of this paper is to propose a novel model-based clustering approach for analyzing ECG data. Our approach is composed of two modules: representation learning and ECG data clustering. In the module of representation learning, a deep generative model referred to as the hyperspherical variational recurrent autoencoder (HVRAE) is developed to extract the representation of observed ECG data, based on the variational autoencoder (VAE) with long short-term memory (LSTM) networks. In the module of ECG data clustering, we develop a nonparametric hidden Markov model (NHMM) based on Dirichlet process in which the number of hidden states is inferred automatically during the learning process. Moreover, the emission density of each hidden state of our NHMM follows a mixture of von Mises-Fisher (VMF) distributions which have better capability for modeling ECG representations than other commonly used distributions (such as the Gaussian distribution). To learn the proposed VMF-based NHMM, we theoretically develop an effective learning algorithm based on variational Bayes. The merits of our model-based clustering approach for analyzing ECG data are verified through experiments on publicly available ECG data sets.
Transfer-Meta Framework for Cross-domain Recommendation to Cold-Start Users
Authors: Yongchun Zhu, Kaikai Ge, Fuzhen Zhuang, Ruobing Xie, Dongbo Xi, Xu Zhang 0028, Leyu Lin, Qing He 0003
Venue: SIGIR
Year: 2021

Cold-start problems are enormous challenges in practical recommender systems. One promising solution for this problem is cross-domain recommendation (CDR) which leverages rich information from an auxiliary (source) domain to improve the performance of recommender system in the target domain. In these CDR approaches, the family of Embedding and Mapping methods for CDR (EMCDR) is very effective, which explicitly learn a mapping function from source embeddings to target embeddings with overlapping users. However, these approaches suffer from one serious problem: the mapping function is only learned on limited overlapping users, and the function would be biased to the limited overlapping users, which leads to unsatisfying generalization ability and degrades the performance on cold-start users in the target domain. With the advantage of meta learning which has good generalization ability to novel tasks, we propose a transfer-meta framework for CDR (TMCDR) which has a transfer stage and a meta stage. In the transfer (pre-training) stage, a source model and a target model are trained on source and target domains, respectively. In the meta stage, a task-oriented meta network is learned to implicitly transform the user embedding in the source domain to the target feature space. In addition, the TMCDR is a general framework that can be applied upon various base models, e.g., MF, BPR, CML. By utilizing data from Amazon and Douban, we conduct extensive experiments on 6 cross-domain tasks to demonstrate the superior performance and compatibility of TMCDR.
Communication Efficient Distributed Hypergraph Clustering
Authors: Chun Jiang Zhu, Qinqing Liu, Jinbo Bi
Venue: SIGIR
Year: 2021

Hypergraphs can capture higher-order relations between subsets of objects instead of only pairwise relations as in graphs. Hypergraph clustering is an important task in information retrieval and machine learning. We study the problem of distributed hypergraph clustering in the message passing communication model using small communication cost. We propose an algorithm framework for distributed hypergraph clustering based on spectral hypergraph sparsification. For an n-vertex hypergraph G with hyperedges of maximum size r distributed at s sites arbitrarily and a parameter ε∈ (0,1), our algorithm can produce a vertex set with conductance O(√1+ε/1-ε √φG), where φG is the conductance of G, using communication cost ~O(nr2s/εO(1)) (~O hides a polylogarithmic factor). The theoretical results are complemented with extensive experiments to demonstrate the efficiency and effectiveness of the proposed algorithm under different real-world datasets. Our source code is publicly available at github.com/chunjiangzhu/dhgc.
Learning to Warm Up Cold Item Embeddings for Cold-start Recommendation with Meta Scaling and Shifting Networks
Authors: Yongchun Zhu, Ruobing Xie, Fuzhen Zhuang, Kaikai Ge, Ying Sun 0006, Xu Zhang 0028, Leyu Lin, Juan Cao 0001
Venue: SIGIR
Year: 2021

Recently, embedding techniques have achieved impressive success in recommender systems. However, the embedding techniques are data demanding and suffer from the cold-start problem. Especially, for the cold-start item which only has limited interactions, it is hard to train a reasonable item ID embedding, called cold ID embedding, which is a major challenge for the embedding techniques. The cold item ID embedding has two main problems: (1) A gap is existing between the cold ID embedding and the deep model. (2) Cold ID embedding would be seriously affected by noisy interaction. However, most existing methods do not consider both two issues in the cold-start problem, simultaneously. To address these problems, we adopt two key ideas: (1) Speed up the model fitting for the cold item ID embedding (fast adaptation). (2) Alleviate the influence of noise. Along this line, we propose Meta Scaling and Shifting Networks to generate scaling and shifting functions for each item, respectively. The scaling function can directly transform cold item ID embeddings into warm feature space which can fit the model better, and the shifting function is able to produce stable embeddings from the noisy embeddings. With the two meta networks, we propose Meta Warm Up Framework (MWUF) which learns to warm up cold ID embeddings. Moreover, MWUF is a general framework that can be applied upon various existing deep recommendation models. The proposed model is evaluated on three popular benchmarks, including both recommendation and advertising datasets. The evaluation results demonstrate its superior performance and compatibility.
How do Online Learning to Rank Methods Adapt to Changes of Intent?
Authors: Shengyao Zhuang, Guido Zuccon
Venue: SIGIR
Year: 2021

Online learning to rank (OLTR) uses interaction data, such as clicks, to dynamically update rankers. OLTR has been thought to capture user intent change overtime - a task that is impossible for rankers trained on statistic datasets such as in offline and counterfactual learning to rank. However, this feature has never been demonstrated and empirically studied, as previous work only considered simulated online data with single user intent or real online data with no explicit notion of intents and how they change over interactions. In this paper, we address this gap by study the capability of OLTR algorithms to adapt to user intent change. Our empirical experiments show that the adaptation to intent change does vary across OLTR methods, and is also dependent on the amount of noise in the implicit feedback signal. This is an important result, as it highlights that intent change adaptation should be studied alongside online and offline performance. Investigating how OLTR algorithms adapt to intent change is challenging as current LTR datasets do not explicitly contain the required intent data. Along with the main findings reported in this paper related to intent change, we also contribute a methodology to investigate this aspect of OLTR methods. Specifically, we create a collection for OLTR with explicit intent change by adapting an existing TREC collection to this task. We further introduce methods to model and simulate click behaviour related to intent change. We further propose novel evaluation metrics tailored to study different aspects of how OLTR methods adapt to intent change.
FedCMR: Federated Cross-Modal Retrieval
Authors: Linlin Zong, Qiujie Xie, Jiahui Zhou, Peiran Wu, Xianchao Zhang 0001, Bo Xu 0009
Venue: SIGIR
Year: 2021

Deep cross-modal retrieval methods have shown their competitiveness among different cross-modal retrieval algorithms. Generally, these methods require a large amount of training data. However, aggregating large amounts of data will incur huge privacy risks and high maintenance costs. Inspired by the recent success of federated learning, we propose the federated cross-modal retrieval (FedCMR), which learns the model with decentralized multi-modal data. Specifically, we first train the cross-modal retrieval model and learn the common space across multiple modalities in each client using its local data. Then, we jointly learn the common subspace of multiple clients on the trusted central server. Finally, each client updates the common subspace of the local model based on the aggregated common subspace on the server, so that all clients participated in the training can benefit from federated learning. Experiment results on four benchmark datasets demonstrate the effectiveness proposed method.
Unsupervised Ensemble Learning with Noisy Label Correction
Authors: Xupeng Zou, Zhongnan Zhang, Zhen He, Liang Shi
Venue: SIGIR
Year: 2021

Unsupervised ensemble learning aims to estimate ground-truth labels via integrating noisy and unreliable labeling results from multiple annotators. Although many techniques have been proposed to deal with this challenging task, there still exists some "tough" instances with noisy labels that are misclassified after the integration, which significantly affect the classification performance. This paper introduces a novel approach to improve the label accuracy based on unsupervised ensemble learning. First, we apply the expectation maximization (EM) algorithm to aggregate labels for all the instances. Then we identify instances that are most likely to be "tough" through a two-stage filtering method. Finally, an ensemble of AdaBoost-based classification models is trained on the high-quality dataset, and predicts new labels for these "tough" instances. The results of empirical investigation on binary classification task show that: (1) our approach can identify "tough" instances from the input dataset effectively; (2) our approach achieves a better performance on improving the accuracy of labels produced by unsupervised ensemble algorithms.
Offline Retrieval Evaluation Without Evaluation Metrics
Authors: Fernando Diaz, Andres Ferraro
Venue: SIGIR
Year: 2022

Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scalar metric such as average precision or normalized discounted cumulative gain. We can use this metric to compare the performance of multiple systems for the same request. Although evaluation metrics provide a convenient summary of system performance, they also collapse subtle differences across users into a single number and can carry assumptions about user behavior and utility not supported across retrieval scenarios. We propose recall-paired preference (RPP), a metric-free evaluation method based on directly computing a preference between ranked lists. RPP simulates multiple user subpopulations per query and compares systems across these pseudo-populations. Our results across multiple search and recommendation tasks demonstrate that RPP substantially improves discriminative power while correlating well with existing metrics and being equally robust to incomplete data.