OpenRaiser/Intern-Atlas huggingface dataset:
Learning term discrimination
Authors: Jibril Frej, Philippe Mulhem, Didier Schwab, Jean-Pierre Chevallet
Venue: Conference SIGIR 2020
Year: 2020

Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stop-word removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled query-document pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3~times faster), without degrading retrieval quality.
A comparison of retrieval models using term dependencies
Authors: Samuel J. Huston, W. Bruce Croft
Venue: Conference CIKM 2014
Year: 2014

A number of retrieval models incorporating term dependencies have recently been introduced. Most of these modify existing "bag-of-words" retrieval models by including features based on the proximity of pairs of terms (or bi-terms). Although these term dependency models have been shown to be significantly more effective than the bag-of-words models, there have been no previous systematic comparisons between the different approaches that have been proposed. In this paper, we compare the effectiveness of recent bi-term dependency models over a range of TREC collections, for both short (title) and long (description) queries. To ensure the reproducibility of our study, all experiments are performed on widely available TREC collections, and all tuned retrieval model parameters are made public. These comparisons show that the weighted sequential dependence model is at least as effective as, and often significantly better than, any other model across this range of collections and queries. We observe that dependency features are much more valuable in improving the performance of longer queries than for shorter queries. We then examine the effectiveness of dependence models that incorporate proximity features involving more than two terms. The results show that these features can improve effectiveness, but not consistently, over the available data sets.
Prop: Pre-training with Representative Words Prediction for Ad-hoc Retrieval
Authors: Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, Xueqi Cheng
Venue: Conference WSDM 2021
Year: 2021

Recently pre-trained language representation models such as BERT have shown great success when fine-tuned on downstream tasks including information retrieval (IR). However, pre-training objectives tailored for ad-hoc retrieval have not been well explored. In this paper, we propose Pre-training with Representative wOrds Prediction (PROP) for ad-hoc retrieval. PROP is inspired by the classical statistical language model for IR, specifically the query likelihood model, which assumes that the query is generated as the piece of text representative of the "ideal" document. Based on this idea, we construct the representative words prediction (ROP) task for pre-training. Given an input document, we sample a pair of word sets according to the document language model, where the set with higher likelihood is deemed as more representative of the document. We then pre-train the Transformer model to predict the pairwise preference between the two word sets, jointly with the Masked Language Model (MLM) objective. By further fine-tuning on a variety of representative downstream ad-hoc retrieval tasks, PROP achieves significant improvements over baselines without pre-training or with other pre-training methods. We also show that PROP can achieve exciting performance under both the zero- and low-resource IR settings.
Dc-bert: Decoupling question and document for efficient contextual encoding
Authors: Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, Daxin Jiang
Venue: Conference SIGIR 2020
Year: 2020

Recent studies on open-domain question answering have achieved prominent performance improvement using pre-trained language models such as BERT. State-of-the-art approaches typically follow the "retrieve and read" pipeline and employ BERT-based reranker to filter retrieved documents before feeding them into the reader module. The BERT retriever takes as input the concatenation of question and each retrieved document. Despite the success of these approaches in terms of QA accuracy, due to the concatenation, they can barely handle high-throughput of incoming questions each with a large collection of retrieved documents. To address the efficiency problem, we propose DC-BERT, a decoupled contextual encoding framework that has dual BERT models: an online BERT which encodes the question only once, and an offline BERT which pre-encodes all the documents and caches their encodings. On SQuAD Open and Natural Questions Open datasets, DC-BERT achieves 10x speedup on document retrieval, while retaining most (about 98%) of the QA performance compared to state-of-the-art approaches for open-domain question answering.
Extending average precision to graded relevance judgments
Authors: Stephen E. Robertson, Evangelos Kanoulas, Emine Yilmaz
Venue: Conference SIGIR 2010
Year: 2010

Evaluation metrics play a critical role both in the context of comparative evaluation of the performance of retrieval systems and in the context of learning-to-rank (LTR) as objective functions to be optimized. Many different evaluation metrics have been proposed in the IR literature, with average precision (AP) being the dominant one due a number of desirable properties it possesses. However, most of these measures, including average precision, do not incorporate graded relevance. In this work, we propose a new measure of retrieval effectiveness, the Graded Average Precision (GAP). GAP generalizes average precision to the case of multi-graded relevance and inherits all the desirable characteristics of AP: it has a nice probabilistic interpretation, it approximates the area
Novo: learnable and interpretable document identifiers for model-based ir
Authors: Zihan Wang, Yujia Zhou, Yiteng Tu, Zhicheng Dou
Venue: Conference CIKM 2023
Year: 2023

Model-based Information Retrieval (Model-based IR) has gained attention due to advancements in generative language models. Unlike traditional dense retrieval methods relying on dense vector representations of documents, model-based IR leverages language models to retrieve documents by generating their unique discrete identifiers (docids). This approach effectively reduces the requirements to store separate document representations in an index. Most existing model-based IR approaches utilize pre-defined static docids, i.e., these docids are fixed and are not learnable by training on the retrieval tasks. However, these docids are not specifically optimized for retrieval tasks, which makes it difficult to learn semantics and relationships between documents and achieve satisfactory retrieval performance. To address the above limitations, we propose Neural Optimized VOcabularial (NOVO) docids. NOVO docids are unique n-gram sets identifying each document. They can be generated in any order to retrieve the corresponding document and can be optimized through training to better learn semantics and relationships between documents. We propose to optimize NOVO docids through query denoising modeling and retrieval tasks, allowing for optimizing both semantic and token representations for such docids. Experiments on two datasets under the normal and zero-shot settings show that NOVO exhibits strong performance in more effective and interpretable model-based IR.
Planning ahead in generative retrieval: Guiding autoregressive generation through simultaneous decoding
Authors: Hansi Zeng, Chen Luo, Hamed Zamani
Venue: Conference SIGIR 2024
Year: 2024

This paper introduces PAG-a novel optimization and decoding approach that guides autoregressive generation of document identifiers in generative retrieval models through simultaneous decoding. To this aim, PAG constructs a set-based and sequential identifier for each document. Motivated by the bag-of-words assumption in information retrieval, the set-based identifier is built on lexical tokens. The sequential identifier, on the other hand, is obtained via quantizing relevance-based representations of documents. Extensive experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin (e.g., 15.6% MRR improvements on MS MARCO), while achieving 22× speed up in terms of query latency.
Term necessity prediction
Authors: Le Zhao, Jamie Callan
Venue: Conference CIKM 2010
Year: 2010

The probability that a term appears in relevant documents (P(t | R)) is a fundamental quantity in several probabilistic retrieval models, however it is difficult to estimate without relevance judgments or a relevance model. We call this value term necessity because it measures the percentage of relevant documents retrieved by the term - how necessary a term's occurrence is to document relevance. Prior research typically either set this probability to a constant, or estimated it based on the term's inverse document frequency, neither of which was very effective.
GAKG: A multimodal geoscience academic knowledge graph
Authors: Cheng Deng, Yuting Jia, Hui Xu, Chong Zhang, Jingyao Tang, Luoyi Fu, Weinan Zhang, Haisong Zhang, Xinbing Wang, Chenghu Zhou
Venue: Conference CIKM 2021
Year: 2021

The research of geoscience plays a strong role in helping people gain a better understanding of the Earth. To effectively represent the knowledge (KG) from enormous geoscience research papers, knowledge graphs can be a powerful means. In the face of enormous geoscience research papers, knowledge graphs can be a powerful means to manage the relationships of data and integrate knowledge extracted from them. However, the existing geoscience KGs mainly focus on the external connection between concepts, whereas the potential abundant information contained in the internal multimodal data of the paper is largely overlooked for more fine-grained knowledge mining. To this end, we propose GAKG, a large-scale multimodal academic KG based on 1.12 million papers published in various geoscience-related journals. In addition to the bibliometrics elements, we also extracted the internal illustrations, tables, and text information of the articles, and dig out the knowledge entities of the papers and the era and spatial attributes of the articles, coupling multimodal academic data and features. Specifically, GAKG realizes knowledge entity extraction under our proposed Human-In-the-Loop framework, the novelty of which is to combine the techniques of machine reading and information retrieval with manual annotation of geoscientists in the loop. Considering the fact that literature of geoscience often contains more abundant illustrations and time scale information compared with that of other disciplines, we extract all the geographical information and era from the geoscience papers' text and illustrations, mapping papers to the atlas and chronology. Based on GAKG, we build several knowledge discovery benchmarks for finding geoscience communities and predicting potential links. GAKG and its services have been made publicly available and user-friendly.
Generative causal interpretation model for spatio-temporal representation learning
Authors: Yu Zhao 0021, Pan Deng, Junting Liu, Xiaofeng Jia, Jianwei Zhang
Venue: Conference KDD 2023
Year: 2023

Learning, interpreting, and predicting from complex and high-dimensional spatio-temporal data is a natural ability of humans and other intelligent agents, and one of the most important and difficult challenges of AI. Although objects may present different observed phenomena under different situations, their causal mechanism and generation rules are stable and invariant. Different from most existing studies that focus on dynamic correlation, we explore the latent causal structure and mechanism of causal descriptors in the spatio-temporal dimension at the microscopic level, thus revealing the generation principle of observation. In this paper, we regard the causal mechanism as a spatio-temporal causal process modulated by non-stationary exogenous variables. To this end, we propose a theoretically-grounded Generative Causal Interpretation Model (GCIM), which infers explanatory-capable microscopic causal descriptors from observational data via spatio-temporal causal representations. The core of GCIM is to estimate the prior distribution of causal descriptors by using the spatio-temporal causal structure and transition process under the constraints of identifiable conditions, thus extending the Variational Auto Encoder (VAE). Furthermore, our method is able to automatically capture domain information from observations to model non-stationarity. We further analyze the model identifiability, showing that the proposed model learned from observations recovers the true one up to a certain degree. Experiments on synthetic and real-world datasets show that GCIM can successfully identify latent causal descriptors and structures, and accurately predict future data.
Neural contextual bandits for personalized recommendation
Authors: Yikun Ban, Yunzhe Qi, Jingrui He
Venue: Conference WWW 2024
Year: 2024

In the dynamic landscape of online businesses, recommender systems are pivotal in enhancing user experiences. While traditional approaches have relied on static supervised learning, the quest for adaptive, user-centric recommendations has led to the emergence of the formulation of contextual bandits. This tutorial investigates the contextual bandits as a powerful framework for personalized recommendations. We delve into the challenges, advanced algorithms and theories, collaborative strategies, and open challenges and future prospects within this field. Different from existing related tutorials, (1) we focus on the exploration perspective of contextual bandits to alleviate the "Matthew Effect'' in the recommender systems, i.e., the rich get richer and the poor get poorer, concerning the popularity of items; (2) in addition to the conventional linear contextual bandits, we will also dedicated to neural contextual bandits which have emerged as an important branch in recent years, to investigate how neural networks benefit contextual bandits for personalized recommendation both empirically and theoretically; (3) we will cover the latest topic, collaborative neural contextual bandits, to incorporate both user heterogeneity and user correlations customized for recommender system; (4) we will provide and discuss the new emerging challenges and open questions for neural contextual bandits with applications in the personalized recommendation, especially for large neural models. Compared with other greedy personalized recommendation approaches, Contextual Bandits techniques provide distinct ways of modeling user preferences. We believe this tutorial can benefit researchers and practitioners by appreciating the power of exploration and the performance guarantee brought by neural contextual bandits, as well as rethinking the challenges caused by the increasing complexity of neural models and the magnitude of data.
Natural and artificial dynamics in gnns: A tutorial
Authors: Dongqi Fu, Zhe Xu, Hanghang Tong, Jingrui He
Venue: Conference WSDM 2023
Year: 2023

In the big data era, the relationship between entities becomes more complex. Therefore, graph (or network) data attracts increasing research attention for carrying complex relational information. For a myriad of graph mining/learning tasks, graph neural networks (GNNs) have been proven as effective tools for extracting informative node and graph representations, which empowers a broad range of applications such as recommendation, fraud detection, molecule design, and many more. However, real-world scenarios bring pragmatic challenges to GNNs. First, the input graphs are evolving, i.e., the graph structure and node features are time-dependent. Integrating temporal information into the GNNs to enhance their representation power requires additional ingenious designs. Second, the input graphs may be unreliable, noisy, and suboptimal for a variety of downstream graph mining/learning tasks. How could end-users deliberately modify the given graphs (e.g., graph topology and node features) to boost GNNs' utility (e.g., accuracy and robustness)? Inspired by the above two kinds of dynamics, in this tutorial, we focus on topics of natural dynamics and artificial dynamics in GNNs and introduce the related works systematically. After that, we point out some promising but under-explored research problems in the combination of these two dynamics. We hope this tutorial could be beneficial to researchers and practitioners in areas including data mining, machine learning, and general artificial intelligence.
Graph sanitation with application to node classification
Authors: Zhe Xu, Boxin Du, Hanghang Tong
Venue: Conference WWW 2022
Year: 2022

The past decades have witnessed the prosperity of graph mining, with a multitude of sophisticated models and algorithms designed for various mining tasks, such as ranking, classification, clustering and anomaly detection. Generally speaking, the vast majority of the existing works aim to answer the following question, that is, given a graph, what is the best way to mine it? In this paper, we introduce the graph sanitation problem, to answer an orthogonal question. That is, given a mining task and an initial graph, what is the best way to improve the initially provided graph? By learning a better graph as part of the input of the mining model, it is expected to benefit graph mining in a variety of settings, ranging from denoising, imputation to defense. We formulate the graph sanitation problem as a bilevel optimization problem, and further instantiate it by semi-supervised node classification, together with an effective solver named GaSoliNe. Extensive experimental results demonstrate that the proposed method is (1) broadly applicable with respect to various graph neural network models and flexible graph modification strategies, (2) effective in improving the node classification accuracy on both the original and contaminated graphs in various perturbation scenarios. In particular, it brings up to 25% performance improvement over the existing robust graph neural network methods.
Prose: Graph structure learning via progressive strategy
Authors: Huizhao Wang, Yao Fu 0006, Tao Yu 0006, Linghui Hu, Weihao Jiang, Shiliang Pu
Venue: Conference KDD 2023
Year: 2023

Graph Neural Networks (GNNs) have been a powerful tool to acquire high-quality node representations dealing with graphs, which strongly depends on a promising graph structure. In the real world scenarios, it is inevitable to introduce noises in graph topology. To prevent GNNs from the disturbance of irrelevant edges or missing edges, graph structure learning is proposed and has attracted considerable attentions in recent years. In this paper, we argue that current graph structure learning methods still pay no regard to the status of nodes and just judge all of their connections simultaneously using a monotonous standard, which will lead to indeterminacy and instability in the optimization process. We designate these methods as status-unaware models. To demonstrate the rationality of our point of view, we conduct exploratory experiments on publicly available datasets, and discover some exciting observations. Afterwards, we propose a new model named Graph Structure Learning via Progressive Strategy (PROSE) according to the observations, which uses a progressive strategy to acquire ideal graph structure in a status-aware way. Concretely, PROSE consists of progressive structure splitting module (PSS) and progressive structure refining module (PSR) to modify node connections according to their global potency, and we also introduce horizontal position encoding and vertical position encoding in order to capture fruitful graph topology information ignored by previous methods. On several widely-used graph datasets, we conduct extensive experiments to demonstrate the effectiveness of our model, and the source code 1 https://github.com/tigerbunny2023/PROSE is provided.
Netguard: Protecting commercial web apis from model inversion attacks using gan-generated fake samples
Authors: Xueluan Gong, Ziyao Wang, Yanjiao Chen, Qian Wang, Cong Wang, Chao Shen
Venue: Conference WWW 2023
Year: 2023

Recently more and more cloud service providers (e.g., Microsoft, Google, and Amazon) have commercialized their well-trained deep learning models by providing limited access via web API interfaces. However, it is shown that these APIs are susceptible to model inversion attacks, where attackers can recover the training data with high fidelity, which may cause serious privacy leakage.Existing defenses against model inversion attacks, however, hinder the model performance and are ineffective for more advanced attacks, e.g., Mirror [4]. In this paper, we proposed NetGuard, a novel utility-aware defense methodology against model inversion attacks (MIAs). Unlike previous works that perturb prediction outputs of the victim model, we propose to mislead the MIA effort by inserting engineered fake samples during the training process. A generative adversarial network (GAN) is carefully built to construct fake training samples to mislead the attack model without degrading the performance of the victim model. Besides, we adopt continual learning to further improve the utility of the victim model. Extensive experiments on CelebA, VGG-Face, and VGG-Face2 datasets show that NetGuard is superior to existing defenses, including DP [37] and Ad-mi [32] on state-of-the-art model inversion attacks, i.e., DMI [8], Mirror [4], Privacy [12], and Alignment [34].