OpenRaiser/Intern-Atlas huggingface dataset:
On the subjectivity and bias of web content credibility evaluations
Authors: Michal Kakol, Michal Jankowski-Lorek, Katarzyna Abramczuk, Adam Wierzbicki, Michele Catasta
Venue: Conference WWW 2013
Year: 2013

In this paper we describe the initial outcomes of the Reconcile1 study concerning Web content credibility evaluations. The study was run with a balanced sample of 1503 respondents who independently evaluated 154 web pages from several thematic categories. Users taking part in the study not only evaluated credibility, but also filled a questionnaire covering additional respondents' traits. Using the gathered information about socio-economic status and psychological features of the users, we studied the influence of subjectivity and bias in the credibility ratings. Subjectivity and bias, in fact, represent a key design issue for Web Credibility systems, to the extent that they could jeopardize the system performance if not taken into account.
Clinical trial retrieval via multi-grained similarity learning
Authors: Junyu Luo, Cheng Qian, Lucas Glass, Fenglong Ma
Venue: Conference SIGIR 2024
Year: 2024

Clinical trial analysis is one of the main business directions and services in IQVIA, and reviewing past similar studies is one of the most critical steps before starting a commercial clinical trial. The current review process is manual and time-consuming, requiring a clinical trial analyst to manually search through an extensive clinical trial database and then review all candidate studies. Therefore, it is of great interest to develop an automatic retrieval algorithm to select similar studies by giving new study information. To achieve this goal, we propose a novel group-based trial similarity learning network named GTSLNet, consisting of two kinds of similarity learning modules. The pair-wise section-level similarity learning module aims to compare the query trial and the candidate trial from the abstract semantic level via the proposed section transformer. Meanwhile, a word-level similarity learning module uses the word similarly matrix to capture the low-level similarity information. Additionally, an aggregation module combines these similarities. To address potential false negatives and noisy data, we introduce a variance-regularized group distance loss function. Experiment results show that the proposed GTSLNet significantly and consistently outperforms state-of-the-art baselines.
Science2cure: A clinical trial search prototype
Authors: Maciej Rybinski, Sarvnaz Karimi, Aleney Khoo
Venue: Conference SIGIR 2021
Year: 2021

With the advances in precision medicine, identifying clinical trials relevant to a specific patient profile becomes more challenging. Often very specific molecular-level patient features need to be matched for the trial to be deemed relevant. Clinical trials contain strict inclusion and exclusion criteria, often written in free-text. Patients profiles are also semi-structured, with some important information hidden in clinical notes. We present a search system that given a patient profile searches over clinical trials for potential matches. It enables the users to leverage the powerful querying language that comes with Apache Lucene query syntax in combination with state-of-the-art Divergence From Randomness retrieval coupled with a BERT-based neural ranking component. This system aims to assist in clinical decision making.
Causal modeling of climate activism on reddit
Authors: Jacopo Lenti, Luca Maria Aiello, Corrado Monti, Gianmarco De Francisci Morales
Venue: Conference WWW 2025
Year: 2025

Climate activism is crucial in stimulating collective societal and behavioral change towards sustainable practices through political pressure. Although multiple factors contribute to the participation in activism, their complex relationships and the scarcity of data on their interactions have restricted most prior research to studying them in isolation, thus preventing the development of a quantitative, causal understanding of why people approach activism. In this work, we develop a comprehensive causal model of how and why Reddit users engage with activist communities driving mass climate protests (mainly the 2019 Earth Strike, Fridays for Future, and Extinction Rebellion). Our framework, based on Stochastic Variational Inference applied to Bayesian Networks, learns the causal pathways over multiple time periods. Distinct from previous studies, our approach uses large-scale and fine-grained longitudinal data (2016 to 2022) to jointly model the roles of sociodemographic makeup, experience of extreme weather events, exposure to climate-related news, and social influence through online interactions. We find that among users interested in climate change, participation in online activist communities is indeed influenced by direct interactions with activists and largely by recent exposure to media coverage of climate protests. Among people aware of climate change, left-leaning people from lower socioeconomic backgrounds are particularly represented in online activist groups. Our findings offer empirical validation for theories of media influence and critical mass, and lay the foundations to inform interventions and future studies to foster public participation in collective action.
Hoaxes and Hidden agendas: A Twitter Conspiracy Theory Dataset: Data Paper
Authors: Samantha C. Phillips, Lynnette Hui Xian Ng, Kathleen M. Carley
Venue: Conference WWW 2022
Year: 2022

Hoaxes and hidden agendas make for compelling conspiracy theories. While many of these theories are ultimately innocuous, others have the potential to do real harm, instigating real-world support or disapproval of the theories. This is further fueled by social media which provides a platform for conspiracy theories to spread at unprecedented rates. Thus, there is a need for the development of automated models to detect conspiracy theories from the social media space in order to quickly and effectively identify the topics of the season and the prevailing stance. To support this development, we create ground truth data through human annotation. In this work, we collect and manually annotate a dataset from Twitter, comprising of four conspiracy theories. Each Tweet is annotated with one of the four topics {climate change, COVID-19 origin, COVID-19 vaccine, Epstein-Maxwell trial}, and its stance towards the conspiracy theory {support, neutral, against}. We perform experiments on this multi-topic dataset to demonstrate its usage in conspiracy-detection, stance-detection and topic-detection.
Over-penalization for extra information in neural ir models
Authors: Kota Usuha, Makoto P. Kato, Sumio Fujita
Venue: Conference CIKM 2024
Year: 2024

This paper presents our analysis of neural IR models, particularly focusing on over-penalization for extra information (OPEX) - a phenomenon where addition of a sentence to a document causes an unreasonable decline in the document rank. We found that neural IR models suffered from OPEX, especially when the added sentence is similar to the other sentences in the document. To mitigate OPEX, we propose to apply a window-based scoring approach that segments a document and aggregates scores of the segments to compute the overall document score. We theoretically proved that the window-based scoring approach fully suppressed OPEX in an extreme case where each segment contains only a single sentence, and empirically showed that this approach mitigated OPEX. The code is available at https://github.com/argonism/OPEX .
Mmlscu: A dataset for multi-modal multi-domain live streaming comment understanding
Authors: Zixiang Meng, Qiang Gao, Di Guo, Yunlong Li, Bobo Li, Hao Fei, Shengqiong Wu, Fei Li, Chong Teng, Donghong Ji
Venue: Conference WWW 2024
Year: 2024

With the increasing popularity of live streaming, the interactions from viewers during a live streaming can provide more specific and constructive feedback for both the streamer and platform. In such scenario, the primary and most direct feedback method from the audience is through comments. Thus, mining these live streaming comments to unearth the intentions behind them and, in turn, aiding streamers to enhance their live streaming quality is significant for the well development of live streaming ecosystem. To this end, we introduce the MMLSCU dataset, containing 50,129 intention-annotated comments across multiple modalities (text, images, vi-deos, audio) from eight streaming domains. Using multimodal pretrained large model and drawing inspiration from the Chain of Thoughts (CoT) concept, we implement an end-to-end model to sequentially perform the following tasks: viewer comment intent detection ➛ intent cause mining ➛ viewer comment explanation ➛ streamer policy suggestion. We employ distinct branches for video and audio to process their respective modalities. After obtaining the video and audio representations, we conduct a multimodal fusion with the comment. This integrated data is then fed into the large language model to perform inference across the four tasks following the CoT framework. Experimental results indicate that our model outperforms three multimodal classification baselines on comment intent detection and streamer policy suggestion, and one multimodal generation baselines on intent cause mining and viewer comment explanation. Compared to the models using only text, our multimodal setting yields superior outcomes. Moreover, incorporating CoT allows our model to enhance comment interpretation and more precise suggestions for the streamers. Our proposed dataset and model will bring new research attention on multimodal live streaming comment understanding.
Fact embedding through diffusion model for knowledge graph completion
Authors: Xiao Long, Liansheng Zhuang, Aodi Li, Houqiang Li, Shafei Wang
Venue: Conference WWW 2024
Year: 2024

Knowledge graph embedding (KGE) is an efficient and scalable method for knowledge graph completion tasks. Existing KGE models typically map entities and relations into a unified continuous vector space and define a score function to capture the connectivity patterns among the elements (entities and relations) of facts. The score on a fact measures its plausibility in a knowledge graph (KG). However, since the connectivity patterns are very complex in a real knowledge graph, it is difficult to define an explicit and efficient score function to capture them, which also limits their performance. This paper argues that plausible facts in a knowledge graph come from a distribution in the low-dimensional fact space. Inspired by this insight, this paper proposes a novel framework called Fact Embedding through Diffusion Model (FDM) to address the knowledge graph completion task. Instead of defining a score function to measure the plausibility of facts in a knowledge graph, this framework directly learns the distribution of plausible facts from the known knowledge graph and casts the entity prediction task into the conditional fact generation task. Specifically, we concatenate the elements embedding in a fact as a whole and take it as input. Then, we introduce a Conditional Fact Denoiser to learn the reverse denoising diffusion process and generate the target fact embedding from noised data. Extensive experiments demonstrate that FDM significantly outperforms existing state-of-the-art methods in three benchmark datasets.
Connecting domains and contrasting samples: A ladder for domain generalization
Authors: Tianxin Wei, Yifan Chen, Xinrui He, Wenxuan Bao, Jingrui He
Venue: Conference KDD 2025
Year: 2025

Distribution shifts between training and testing samples frequently occur in practice and impede model generalization performance. This crucial challenge thereby motivates studies on domain generalization (DG), which aim to predict the label on unseen target domain data by solely using data from source domains. It is intuitive to conceive the class-separated representations learned in contrastive learning (CL) are able to improve DG, while the reality is quite the opposite: users observe directly applying CL deteriorates the performance. We analyze the phenomenon with the insights from CL theory and discover lack of intra-class connectivity in the DG setting causes the deficiency. We thus propose a new paradigm, domain-connecting contrastive learning (DCCL), to enhance the conceptual connectivity across domains and obtain generalizable representations for DG. On the data side, more aggressive data augmentation and cross-domain positive samples are introduced to improve intra-class connectivity. On the model side, to better embed the unseen test domains, we propose model anchoring to exploit the intra-class connectivity in pre-trained representations and complement the anchoring with generative transformation loss. Extensive experiments on five standard DG benchmarks are performed. The results verify that DCCL outperforms state-of-the-art baselines even without domain supervision. The detailed model implementation and the code are provided through https://github.com/weitianxin/DCCL.
A mathematical word problem generator with structure planning and knowledge enhancement
Authors: Longhu Qin, Jiayu Liu, Zhenya Huang, Kai Zhang, Qi Liu, Binbin Jin, Enhong Chen
Venue: Conference SIGIR 2023
Year: 2023

Automatically generating controllable and diverse mathematical word problems (MWPs) which conform to equations and topics is a crucial task in information retrieval and natural language generation. Recent deep learning models mainly focus on improving the problem readability but overlook the mathematical logic coherence, which tends to generate unsolvable problems. In this paper, we draw inspiration from the human problem-designing process and propose a Mathematical structure Planning and Knowledge enhanced Generation model (MaPKG), following the "plan-then-generate" steps. Specifically, we propose a novel dynamic planning module to make sentence-level equation plans and a dual-attention mechanism for word-level generation, incorporating equation structure representation and external commonsense knowledge. Extensive experiments on two MWP datasets show our model can guarantee more solvable, high-quality, and diverse problems. Our code is available at https://github.com/KenelmQLH/MaPKG.git
Privacy-preserved neural graph databases
Authors: Qi Hu, Haoran Li 0003, Jiaxin Bai, Zihao Wang 0001, Yangqiu Song
Venue: Conference KDD 2024
Year: 2024

In the era of large language models (LLMs), efficient and accurate data retrieval has become increasingly crucial for the use of domain-specific or private data in the retrieval augmented generation (RAG). Neural graph databases (NGDBs) have emerged as a powerful paradigm that combines the strengths of graph databases (GDBs) and neural networks to enable efficient storage, retrieval, and analysis of graph-structured data which can be adaptively trained with LLMs. The usage of neural embedding storage and Complex neural logical Query Answering (CQA) provides NGDBs with generalization ability. When the graph is incomplete, by extracting latent patterns and representations, neural graph databases can fill gaps in the graph structure, revealing hidden relationships and enabling accurate query answering. Nevertheless, this capability comes with inherent trade-offs, as it introduces additional privacy risks to the domain-specific or private databases. Malicious attackers can infer more sensitive information in the database using well-designed queries such as from the answer sets of where Turing Award winners born before 1950 and after 1940 lived, the living places of Turing Award winner Hinton are probably exposed, although the living places may have been deleted in the training stage due to the privacy concerns. In this work, we propose a privacy-preserved neural graph database (P-NGDB) framework to alleviate the risks of privacy leakage in NGDBs. We introduce adversarial training techniques in the training stage to enforce the NGDBs to generate indistinguishable answers when queried with private information, enhancing the difficulty of inferring sensitive information through combinations of multiple innocuous queries. Extensive experimental results on three datasets show that our framework can effectively protect private information in the graph database while delivering high-quality public answers responses to queries. The code is available at https://github.com/HKUST-KnowComp/PrivateNGDB.
Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks
Authors: Alexander Braylan, Omar Alonso, Matthew Lease
Venue: Conference WWW 2022
Year: 2022

When annotators label data, a key metric for quality assurance is inter-annotator agreement (IAA): the extent to which annotators agree on their labels. Though many IAA measures exist for simple categorical and ordinal labeling tasks, relatively little work has considered more complex labeling tasks, such as structured, multi-object, and free-text annotations. Krippendorff's alpha, best known for use with simpler labeling tasks, does have a distance-based formulation with broader applicability, but little work has studied its efficacy and consistency across complex annotation tasks. We investigate the design and evaluation of IAA measures for complex annotation tasks, with evaluation spanning seven diverse tasks: image bounding boxes, image keypoints, text sequence tagging, ranked lists, free text translations, numeric vectors, and syntax trees. We identify the difficulty of interpretability and the complexity of choosing a distance function as key obstacles in applying Krippendorff's alpha generally across these tasks. We propose two novel, more interpretable measures, showing they yield more consistent IAA measures across tasks and annotation distance functions.
Escape from an echo chamber
Authors: Kuan-Chieh Lo, Shih-Chieh Dai, Aiping Xiong, Jing Jiang, Lun-Wei Ku
Venue: Conference WWW 2021
Year: 2021

An echo chamber effect refers to the phenomena that online users revealed selective exposure and ideological segregation on political issues. Prior studies indicate the connection between the spread of misinformation and online echo chambers. In this paper, to help users escape from an echo chamber, we propose a novel news-analysis platform that provides a panoramic view of stances towards a particular event from different news media sources. Moreover, to help users better recognize the stances of news sources which published these news articles, we adopt a news stance classification model to categorize their stances into “agree”, “disagree”, “discuss”, or “unrelated” to a relevant claim for specified events with political stances. Finally, we proposed two ways showing the echo chamber effects: 1) visualizing the event and the associated pieces of news; and 2) visualizing the stance distribution of news from news sources of different political ideology. By making the echo chamber effect explicit, we expect online users will become exposed to more diverse perspectives toward a specific event. The demo video of our platform is available on youtube1.
Example-based search: a new frontier for exploratory search
Authors: Matteo Lissandrini, Davide Mottin, Themis Palpanas, Yannis Velegrakis
Venue: Conference SIGIR 2019
Year: 2019

Exploration is one of the primordial ways to accrue knowledge about the world and its nature. As we accumulate, mostly automatically, data at unprecedented volumes and speed, our datasets have become complex and hard to understand. In this context, exploratory search provides a handy tool for progressively gather the necessary knowledge by starting from a tentative query that can provide cues about the next queries to issue. An exploratory query should be simple enough to avoid complicate declarative languages (such as SQL) and convoluted mechanism, and at the same time retain the flexibility and expressiveness required to express complex information needs. Recently, we have witnessed a rediscovery of the so called example-based methods, in which the user, or the analyst circumvent query languages by using examples as input. This shift in semantics has led to a number of methods receiving as query a set of example members of the answer set. The search system then infers the entire answer set based on the given examples and any additional information provided by the underlying database. In this tutorial, we present an excursus over the main example-based methods for exploratory analysis. We show how different data types require different techniques, and present algorithms that are specifically designed for relational, textual, and graph data. We conclude by providing a unifying view of this query-paradigm and identify new exciting research directions.
Few-shot llm synthetic data with distribution matching
Authors: Jiyuan Ren, Zhaocheng Du, Zhihao Wen, Qinglin Jia, Sunhao Dai, Chuhan Wu, Zhenhua Dong
Venue: Conference WWW 2025
Year: 2025

As large language models (LLMs) advance, their ability to perform in-context learning and few-shot language generation has improved significantly. This has spurred using LLMs to produce high-quality synthetic data to enhance the performance of smaller models like online retrievers or weak LLMs. However, LLM-generated synthetic data often differs from the real data in key language attributes (e.g., styles, tones, content proportions, etc.). As a result, mixing these synthetic data directly with real data may distort the original data distribution, potentially hindering performance improvements. To solve this, we introduce SynAlign: a synthetic data generation and filtering framework based on key attribute distribution matching. Before generation, SynAlign employs an uncertainty tracker surrogated by the Gaussian Process model to iteratively select data clusters distinct from selected ones as demonstrations for new data synthesis, facilitating the efficient exploration diversity of the real data. Then, a latent attribute reasoning method is employed: the LLM summarizes linguistic attributes of demonstrations and then synthesizes new data based on them. This approach facilitates synthesizing diverse data with linguistic attributes that appear in real data. After generation, the Maximum Mean Discrepancy is used as the objective function to learn the sampling weight of each synthetic data, ensuring distribution matching with the real data. Our experiments on multiple text prediction tasks show significant performance improvements. We also conducted an online A/B test on an online retriever to demonstrate SynAlign's effectiveness. Our code is available https://github.com/nighood/SynAlign here.