OpenRaiser/Intern-Atlas huggingface dataset:
What do users really ask large language models? an initial log analysis of google bard interactions in the wild
Authors: Johanne R. Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, Luke Gallagher
Venue: Conference SIGIR 2024
Year: 2024

Advancements in large language models (LLMs) have changed information retrieval, offering users a more personalised and natural search experience with technologies like OpenAI ChatGPT, Google Bard (Gemini), or Microsoft Copilot. Despite these advancements, research into user tasks and information needs remains scarce. This preliminary work analyses a Google Bard prompt log with 15,023 interactions called the Bard Intelligence and Dialogue Dataset (BIDD), providing an understanding akin to query log analyses. We show that Google Bard prompts are often verbose and structured, encapsulating a broader range of information needs and imperative (e.g., directive) tasks distinct from traditional search queries. We show that LLMs can support users in tasks beyond the three main types based on user intent: informational, navigational, and transactional. Our findings emphasise the versatile application of LLMs across content creation, LLM writing style preferences, and information extraction. We document diverse user interaction styles, showcasing the adaptability of users to LLM capabilities.
Tevatron 2.0: Unified document retrieval toolkit across scale, language, and modality
Authors: Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, Jimmy Lin
Venue: Conference SIGIR 2025
Year: 2025

Recent advancements in large language models (LLMs) have driven interest in billion-scale retrieval models with strong generalization across retrieval tasks and languages. Additionally, progress in large vision-language models has created new opportunities for multimodal retrieval. In response, we have updated the Tevatron toolkit, introducing a unified pipeline that enables researchers to explore retriever models at different scales, across multiple languages, and with various modalities. This demo paper highlights the toolkit's key features, bridging academia and industry by supporting efficient training, inference, and evaluation of neural retrievers. We showcase a unified dense retriever achieving strong multilingual and multimodal effectiveness, and conduct a cross-modality zero-shot study to demonstrate its research potential. Alongside, we release OmniEmbed, to the best of our knowledge, the first embedding model that unifies text, image document, video, and audio retrieval, serving as a baseline for future research.
Mmadapt: A knowledge-guided multi-source multi-class domain adaptive framework for early health misinformation detection
Authors: Lanyu Shang, Yang Zhang, Bozhang Chen, Ruohan Zong, Zhenrui Yue, Huimin Zeng, Na Wei, Dong Wang
Venue: Conference WWW 2024
Year: 2024

This paper studies a critical problem of emergent health misinformation detection, aiming to mitigate the spread of misinformation in emergent health domains to support well-informed healthcare decisions towards a Web for good health. Our work is motivated by the lack of timely resources (e.g., medical knowledge, annotated data) during the initial phases of an emergent health event or topic. In this paper, we develop a multi-source domain adaptive framework that jointly exploits medical knowledge and annotated data from different high-resource source domains (e.g., cancer, COVID-19) to detect misleading posts in an emergent target domain (e.g., mpox, polio). Two important challenges exist in developing our solution: 1) how to accurately detect the partially misleading and unverifiable content in an emergent target domain? 2) How to identify the conflicting knowledge facts from different source domains to accurately detect emergent misinformation in the target domain? To address these challenges, we develop MMAdapt, a multi-source multi-class domain adaptive misinformation detection framework that effectively explores diverse knowledge facts from different source domains to accurately detect not only the outright misleading but also the partially misleading or unverifiable posts on the Web. Extensive experimental results on four real-world misinformation datasets demonstrate that MMAdapt substantially outperforms state-of-the-art baselines in accurately detecting misinformation in an emergent health domain.
Traceback of poisoning attacks to retrieval-augmented generation
Authors: Baolei Zhang, Haoran Xin, Minghong Fang, Zhuqing Liu, Biao Yi, Tong Li, Zheli Liu
Venue: Conference WWW 2025
Year: 2025

Large language models (LLMs) integrated with retrieval-augmented generation (RAG) systems improve accuracy by leveraging external knowledge sources. However, recent research has revealed RAG's susceptibility to poisoning attacks, where the attacker injects poisoned texts into the knowledge database, leading to attacker-desired responses. Existing defenses, which predominantly focus on inference-time mitigation, have proven insufficient against sophisticated attacks. In this paper, we introduce RAGForensics, the first traceback system for RAG, designed to identify poisoned texts within the knowledge database that are responsible for the attacks. RAGForensics operates iteratively, first retrieving a subset of texts from the database and then utilizing a specially crafted prompt to guide an LLM in detecting potential poisoning texts. Empirical evaluations across multiple datasets demonstrate the effectiveness of RAGForensics against state-of-the-art poisoning attacks. This work pioneers the traceback of poisoned texts in RAG systems, providing a practical and promising defense mechanism to enhance their security. Our code is available at: https://github.com/zhangbl6618/RAG-Responsibility-Attribution
Lumos: Empowering multimodal llms with scene text recognition
Authors: Ashish Shenoy, Yichao Lu, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Abhay Harpale, Vikas Bhardwaj, Di Xu 0009, Shicong Zhao, Longfang Zhao, Ankit Ramchandani, Xin Luna Dong, Anuj Kumar
Venue: Conference KDD 2024
Year: 2024

We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.
Axiomatic causal interventions for reverse engineering relevance computation in neural retrieval models
Authors: Catherine Chen, Jack Merullo, Carsten Eickhoff
Venue: Conference SIGIR 2024
Year: 2024

Neural models have demonstrated remarkable performance across diverse ranking tasks. However, the processes and internal mechanisms along which they determine relevance are still largely unknown. Existing approaches for analyzing neural ranker behavior with respect to IR properties rely either on assessing overall model behavior or employing probing methods that may offer an incomplete understanding of causal mechanisms. To provide a more granular understanding of internal model decision-making processes, we propose the use of causal interventions to reverse engineer neural rankers, and demonstrate how mechanistic interpretability methods can be used to isolate components satisfying term-frequency axioms within a ranking model. We identify a group of attention heads that detect duplicate tokens in earlier layers of the model, then communicate with downstream heads to compute overall document relevance. More generally, we propose that this style of mechanistic analysis opens up avenues for reverse engineering the processes neural retrieval models use to compute relevance. This work aims to initiate granular interpretability efforts that will not only benefit retrieval model development and training, but ultimately ensure safer deployment of these models.
Glider: A reinforcement learning approach to extract ui scripts from websites
Authors: Yuanchun Li, Oriana Riva
Venue: Conference SIGIR 2021
Year: 2021

Web automation scripts (tasklets) are used by personal AI assistants to carry out human tasks such as reserving a car or buying movie tickets. Generating tasklets today is a tedious job which requires much manual effort. We propose Glider, an automated and scalable approach to generate tasklets from a natural language task query and a website URL. A major advantage of Glider is that it does not require any pre-training. Glider models tasklet extraction as a state space search, where agents can explore a website's UI and get rewarded when making progress towards task completion. The reward is computed based on the agent's navigating pattern and the similarity between its trajectory and the task query. A hierarchical reinforcement learning policy is used to efficiently find the action sequences that maximize the reward. To evaluate Glider, we used it to extract tasklets for tasks in various categories (shopping, real-estate, flights, etc.); in 79% of cases a correct tasklet was generated.
Statistical precision of information retrieval evaluation
Authors: Gordon V. Cormack, Thomas R. Lynam
Venue: Conference SIGIR 2006
Year: 2006

We introduce and validate bootstrap techniques to compute confidence intervals that quantify the effect of test-collection variability on average precision (AP) and mean average precision (MAP) IR effectiveness measures. We consider the test collection in IR evaluation to be a representative of a population of materially similar collections, whose documents are drawn from an infinite pool with similar characteristics. Our model accurately predicts the degree of concordance between system results on randomly selected halves of the TREC-6 ad hoc corpus. We advance a framework for statistical evaluation that uses the same general framework to model other sources of chance variation as a source of input for meta-analysis techniques.
Query understanding via intent description generation
Authors: Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Xueqi Cheng
Venue: Conference CIKM 2020
Year: 2020

Query understanding is a fundamental problem in information retrieval (IR), which has attracted continuous attention through the past decades. Many different tasks have been proposed for understanding users' search queries, e.g., query classification or query clustering. However, it is not that precise to understand a search query at the intent class/cluster level due to the loss of many detailed information. As we may find in many benchmark datasets, e.g., TREC and SemEval, queries are often associated with a detailed description provided by human annotators which clearly describes its intent to help evaluate the relevance of the documents. If a system could automatically generate a detailed and precise intent description for a search query, like human annotators, that would indicate much better query understanding has been achieved. In this paper, therefore, we propose a novel Query-to-Intent-Description (Q2ID) task for query understanding. Unlike those existing ranking tasks which leverage the query and its description to compute the relevance of documents, Q2ID is a reverse task which aims to generate a natural language intent description based on both relevant and irrelevant documents of a given query. To address this new task, we propose a novel Contrastive Generation model, namely CtrsGen for short, to generate the intent description by contrasting the relevant documents with the irrelevant documents given a query. We demonstrate the effectiveness of our model by comparing with several state-of-the-art generation models on the Q2ID task. We discuss the potential usage of such Q2ID technique through an example application.
Explainable link prediction in knowledge hypergraphs
Authors: Zirui Chen, Xin Wang, Chenxu Wang, Jianxin Li
Venue: Conference CIKM 2022
Year: 2022

Link prediction in knowledge hypergraphs has been recognized as a critical issue in various downstream tasks for knowledge-enabled applications, from question answering to recommender systems. However, most existing approaches are primarily performed in a black-box fashion, which learn low-dimensional embeddings for inference, thus cannot provide human-understandable interpretation. In this paper, we present HyperMLN, an n-ary, mixed, and explainable framework that interprets the path-reasoning process with first-order logic, which provides a knowledge-enhanced interpretable prediction framework, in which domain knowledge in the logic rules improves the performance of embedding models, while semantic information in the embedding space can optimize the weight of the logic rules in turn. To provide benchmark rule sets for explainable link prediction methods, three types of meta-logic rules in each popular dataset are mined for interpreting results. While achieving explainability, our framework also realizes an average improvement of 3.2% on Hits@1 compared to the state-of-the-art knowledge hypergraph embedding method. Our code is available at https://github.com/zirui-chen/HyperMLN.
Enhance: Enhanced entity and relation embedding for knowledge hypergraph link prediction
Authors: Chenxu Wang, Zhao Li, Xin Wang, Zirui Chen
Venue: Conference WWW 2023
Year: 2023

Knowledge Hypergraphs, as the generalization of knowledge graphs, have attracted increasingly widespread attention due to their friendly compatibility with real-world facts. However, link prediction in knowledge hypergraph is still an underexplored field despite the ubiquity of n-ary facts in the real world. Several recent representative embedding-based knowledge hypergraph link prediction methods have proven to be effective in a series of benchmarks, however, they only consider the position (or role) information, ignoring the neighborhood structure among entities and rich semantic information within each fact. To this end, we propose a model named EnhancE for effective link prediction in knowledge hypergraphs. On the one hand, a more expressive entity representation is obtained with both position and neighborhood information added to the initial embedding. On the other hand, rich semantic information of the involved entities within each tuple is incorporated into relation embedding for enhanced representation. Extensive experimental results over real datasets of both knowledge hypergraph and knowledge graph demonstrate the excellent performance of EnhancE compared with a variety of state-of-the-art baselines.
Inductive link prediction on n-ary relational facts via semantic hypergraph reasoning
Authors: Gongzhu Yin, Hongli Zhang, Yuchen Yang, Yi Luo
Venue: Conference KDD 2025
Year: 2025

N-ary relational facts represent semantic correlations among more than two entities. While recent studies have developed link prediction (LP) methods to infer missing relations for knowledge graphs (KGs) containing n-ary relational facts, they are generally limited to transductive settings. Fully inductive settings, where predictions are made on previously unseen entities, remain a significant challenge. As existing methods are mainly entity embedding-based, they struggle to capture entity-independent logical rules. To fill in this gap, we propose an n-ary subgraph reasoning framework for fully inductive link prediction (ILP) on n-ary relational facts. This framework reasons over local subgraphs and has a strong inductive inference ability to capture n-ary patterns. Specifically, we introduce a novel graph structure, the n-ary semantic hypergraph, to facilitate subgraph extraction. Moreover, we develop a subgraph aggregating network, NS-HART, to effectively mine complex semantic correlations within subgraphs. Theoretically, we provide a thorough analysis from the score function optimization perspective to shed light on NS-HART's effectiveness for n-ary ILP tasks. Empirically, we conduct extensive experiments on a series of inductive benchmarks, including transfer reasoning (with and without entity features) and pairwise subgraph reasoning. The results highlight the superiority of the n-ary subgraph reasoning framework and the exceptional inductive ability of NS-HART. The source code of this paper has been made publicly available at https://github.com/yin-gz/Nary-Inductive-SubGraph.
Genre identification and goal-focused summarization
Authors: Jade Goldstein, Gary M. Ciany, Jaime G. Carbonell
Venue: Conference CIKM 2007
Year: 2007

In this paper, we present a novel technique of first performing document genre identification, then utilizing the genre for producing tailored summaries based on a user's information seeking needs - genre oriented goal-focused summarization - such as a plot or opinion summary of a movie review. We create a test corpus to determine genre classification accuracy for 16 genres, and examine performance on various amounts of training data for machine learning algorithms - Random Forests, SVM light and Naïve Bayes. Results show that Random Forests outperforms SVM light and Naïve Bayes. The genre tag is used to inform a downstream summarization engine. We define types of summaries for 7 genres, create a ground truth corpus and analyze the results of genre oriented goal-focused summarization, showing that this type of user based summarization requires different algorithms than the leading sentence baseline which is known to perform well in the case of news articles.
A generic opinion-fact classifier with application in understanding opinionatedness in various news section
Authors: Ankan Mullick, Shivam Maheshwari, Soumya C., Pawan Goyal, Niloy Ganguly
Venue: Conference WWW 2017
Year: 2017

In this paper, we build a generic opinion-fact classifier to detect opinions and facts from online news articles and social media datasets such as Youtube comments and idiom hashtags. We further use this classification model to compare opinionatedness of various news article sections. The proposed classifier produces better results than the existing methods over four different datasets, and the opinion fraction of various sections of news articles provides very interesting patterns.
Improving health mention classification through emphasising literal meanings: A study towards diversity and generalisation for public health surveillance
Authors: Olanrewaju Tahir Aduragba, Jialin Yu, Alexandra I. Cristea, Yang Long
Venue: Conference WWW 2023
Year: 2023

People often use disease or symptom terms on social media and online forums in ways other than to describe their health. Thus the NLP health mention classification (HMC) task aims to identify posts where users are discussing health conditions literally, not figuratively. Existing computational research typically only studies health mentions within well-represented groups in developed nations. Developing countries with limited health surveillance abilities fail to benefit from such data to manage public health crises. To advance the HMC research and benefit more diverse populations, we present the Nairaland health mention dataset (NHMD), a new dataset collected from a dedicated web forum for Nigerians. NHMD consists of 7,763 manually labelled posts extracted based on four prevalent diseases (HIV/AIDS, Malaria, Stroke and Tuberculosis) in Nigeria. With NHMD, we conduct extensive experiments using current state-of-the-art models for HMC and identify that, compared to existing public datasets, NHMD contains out-of-distribution examples. Hence, it is well suited for domain adaptation studies. The introduction of the NHMD dataset imposes better diversity coverage of vulnerable populations and generalisation for HMC tasks in a global public health surveillance setting. Additionally, we present a novel multi-task learning approach for HMC tasks by combining literal word meaning prediction as an auxiliary task. Experimental results demonstrate that the proposed approach outperforms state-of-the-art methods statistically significantly (p < 0.01, Wilcoxon test) in terms of F1 score over the state-of-the-art and shows that our new dataset poses a strong challenge to the existing HMC methods.
A data­ driven model for monocular face tracking
Authors: Salih Burak Göktürk, Jean-Yves Bouguet, Radek Grzeszczuk
Venue: Conference ICCV 2001
Year: 2001

This paper describes a two-stage system for 3D tracking of pose and deformation of the human face in monocular image sequences without the use of special markers. The first stage of the system learns the space of all possible facial deformations by applying principal component analysis on real stereo tracking data. The resulting model approximates any generic shape as a linear combination of shape basis vectors. The second stage of the system uses this low-complexity deformable model for simultaneous tracking of pose and deformation of the face from a single image sequence. This stage is known as model-based monocular tracking. There are three main contributions of this paper. First we demonstrate that a data-driven approach for model construction is suitable for tracking non rigid objects and offers an elegant and practical alternative to the task of manual construction of models using 3D scanners or CAD modelers. Second, we show that such a method exhibits good tracking accuracy (errors less than 5 mm) and robustness characteristics. Third, we demonstrate that our system exhibits very promising generalization properties in enabling tracking of multiple persons with the same 3D model.
Resynthesizing facial animation through 3D model-based tracking
Authors: Frédéric H. Pighin, Richard Szeliski, David Salesin
Venue: Conference ICCV 1999
Year: 1999

Given video footage of a person's face, we present new techniques to automatically recover the face position and the facial expression from each frame in the video sequence. A 3D face model is fitted to each frame using a continuous optimization technique. Our model is based on a set of 3D face models that are linearly combined using 3D morphing. Our method has the advantages over previous techniques of fitting directly a realistic 3-dimensional face model and of recovering parameters that can be used directly in an animation system. We also explore many applications, including performance-driven animation (applying the recovered position and expression of the face to a synthetic character to produce an animation that mimics the input video), relighting the face, varying the camera position, and adding facial ornaments such as tattoos and scars.
Linear Subspaces for Illumination-Robust Face Recognition
Authors: Aziz Umit Batur, Monson H. Hayes III
Venue: Conference CVPR 2001
Year: 2001

In this paper, we present a segmented linear subspace model for face recognition that is robust under varying illumination conditions. The algorithm generalizes the 3D illumination subspace model by segmenting the image into regions that have surface normals whose directions are close to each other. This segmentation is performed using a K-means clustering algorithm and requires only a few training images under different illuminations. When the linear subspace model is applied to the segmented image, recognition is robust to attached and cast shadows, and the recognition rate is equal to that of computationally more complex systems that require constructing the 3D surface of the face.