OpenRaiser/Intern-Atlas huggingface dataset:
Editable User Profiles for Controllable Text Recommendations
Authors: Sheshera Mysore, Mahmood Jasim, Andrew McCallum, Hamed Zamani
Venue: Conference SIGIR 2023
Year: 2023

Methods for making high-quality recommendations often rely on learning latent representations from interaction data. These methods, while performant, do not provide ready mechanisms for users to control the recommendation they receive. Our work tackles this problem by proposing LACE, a novel concept value bottleneck model for controllable text recommendations. LACE represents each user with a succinct set of human-readable concepts through retrieval given user-interacted documents and learns personalized representations of the concepts based on user documents. This concept based user profile is then leveraged to make recommendations. The design of our model affords control over the recommendations through a number of intuitive interactions with a transparent user profile. We first establish the quality of recommendations obtained from LACE in an offline evaluation on three recommendation tasks spanning six datasets in warm-start, cold-start, and zero-shot setups. Next, we validate the controllability of LACE under simulated user interactions. Finally, we implement LACE in an interactive controllable recommender system and conduct a user study to demonstrate that users are able to improve the quality of recommendations they receive through interactions with an editable user profile.
Dice: A dataset of italian crime event news
Authors: Giovanni Bonisoli, Maria Pia di Buono, Laura Po, Federica Rollo
Venue: Conference SIGIR 2023
Year: 2023

Extracting events from news stories as the aim of several Natural Language Processing (NLP) applications (e.g., question answering, news recommendation, news summarization) is not a trivial task, due to the complexity of natural language and the fact that news reporting is characterized by journalistic style and norms. Those aspects entail scattering an event description over several sentences within one document (or more documents), applying a mechanism of gradual specification of event-related information. This implies a widespread use of co-reference relations among the textual elements, conveying non-linear temporal information. In addition to this, despite the achievement of state-of-the-art results in several tasks, high-quality training datasets for non-English languages are rarely available. This paper presents our preliminary study to develop an annotated Dataset for Italian Crime Event news (DICE). The contribution of the paper are: (1) the creation of a corpus of 10,395 crime news; (2) the annotation schema; (3) a dataset of 10,395 news with automatic annotations; (4) a preliminary manual annotation using the proposed schema of 1000 documents. The first tests on DICE have compared the performance of a manual annotator with that of single-span and multi-span question answering models and shown there is still a gap in the models, especially when dealing with more complex annotation tasks and limited training data. This underscores the importance of investing in the creation of high-quality annotated datasets like DICE, which can provide a solid foundation for training and testing a wide range of NLP models.
Controlled analyses of social biases in Wikipedia bios
Authors: Anjalie Field, Chan Young Park, Kevin Z. Lin, Yulia Tsvetkov
Venue: Conference WWW 2022
Year: 2022

Social biases on Wikipedia, a widely-read global platform, could greatly influence public opinion. While prior research has examined man/woman gender bias in biography articles, possible influences of other demographic attributes limit conclusions. In this work, we present a methodology for analyzing Wikipedia pages about people that isolates dimensions of interest (e.g., gender), from other attributes (e.g., occupation). Given a target corpus for analysis (e.g. biographies about women), we present a method for constructing a comparison corpus that matches the target corpus in as many attributes as possible, except the target one. We develop evaluation metrics to measure how well the comparison corpus aligns with the target corpus and then examine how articles about gender and racial minorities (cis. women, non-binary people, transgender women, and transgender men; African American, Asian American, and Hispanic/Latinx American people) differ from other articles. In addition to identifying suspect social biases, our results show that failing to control for covariates can result in different conclusions and veil biases. Our contributions include methodology that facilitates further analyses of bias in Wikipedia articles, findings that can aid Wikipedia editors in reducing biases, and a framework and evaluation metrics to guide future work in this area.
Rag-ex: A generic framework for explaining retrieval augmented generation
Authors: Viju Sudhi, Sinchana Ramakanth Bhat, Max Rudat, Roman Teucher
Venue: Conference SIGIR 2024
Year: 2024

Owing to their size and complexity, large language models (LLMs) hardly explain why they generate a response. This effectively reduces the trust and confidence of end users in LLM-based applications, including Retrieval Augmented Generation (RAG) for Question Answering (QA) tasks. In this work, we introduce RAG-Ex, a model- and language-agnostic explanation framework that presents approximate explanations to the users revealing why the LLMs possibly generated a piece of text as a response, given the user input. Our framework is compatible with both open-source and proprietary LLMs. We report the significance scores of the approximated explanations from our generic explainer in both English and German QA tasks and also study their correlation with the downstream performance of LLMs. In the extensive user studies, our explainer yields an F1-score of 76.9% against the end user annotations and attains almost on-par performance with model-intrinsic approaches.
Online disinformation and generative language models: Motivations, challenges, and mitigations
Authors: Ziyi Guo
Venue: Conference WWW 2024
Year: 2024

Disinformation refers to the deliberate dissemination of fake or misleading information, which significantly threatens the modern social stability by undermining trust, intensifying polarization and manipulating public opinion. With the advances of generative AI, the landscape of modern disinformation is changing following the rise of Large Language Models. Recent studies has revealed the capability of generative language models to create convincing and misleading content against the truth and warned the availability of such models to be maliciously abused for deceptive generation. However, AI-driven disinformation is a human-centered societal issue in nature, the realization of which requires not only the in- depth discussion on the latest trends from both sides of generative AI and disinformation, but a critical analysis on the uncertainty of their potential interaction in practice as well. The paper introduces the new vision of AI-driven disinformation campaigns from the perspectives of human-centered AI, proposes a framework of core research questions based on the existing research gap, discusses the preliminary discovery in literature and initial experiments, and elaborates the main lines of research in the future work.
Multimodal sentiment analysis to explore the structure of emotions
Authors: Anthony Hu, Seth R. Flaxman
Venue: Conference KDD 2018
Year: 2018

We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual analysis and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the latent emotional state of the user. Thus, we focus on predicting the emotion word tags attached by users to their Tumblr posts, treating these as "self-reported emotions." We demonstrate that our multimodal model combining both text and image features outperforms separate models based solely on either images or text. Our model's results are interpretable, automatically yielding sensible word lists associated with emotions. We explore the structure of emotions implied by our model and compare it to what has been posited in the psychology literature, and validate our model on a set of images that have been used in psychology studies. Finally, our work also provides a useful tool for the growing academic study of images - both photographs and memes - on social networks.
Large scale sentiment learning with limited labels
Authors: Vasileios Iosifidis, Eirini Ntoutsi
Venue: Conference KDD 2017
Year: 2017

Sentiment analysis is an important task in order to gain insights over the huge amounts of opinions that are generated in the social media on a daily basis. Although there is a lot of work on sentiment analysis, there are no many datasets available which one can use for developing new methods and for evaluation. To the best of our knowledge, the largest dataset for sentiment analysis is TSentiment [8], a 1.6 millions machine-annotated tweets dataset covering a period of about 3 months in 2009. This dataset however is too short and therefore insufficient to study heterogeneous, fast evolving streams. Therefore, we annotated the Twitter dataset of 2015 (228 million tweets without retweets and 275 million with retweets) and we make it publicly available for research. For the annotation we leverage the power of unlabeled data, together with labeled data using semi-supervised learning and in particular, Self-Learning and Co-Training. Our main contribution is the provision of the TSentiment15 dataset together with insights from the analysis, which includes a batch and a stream-processing of the data. In the former, all labeled and unlabeled data are available to the algorithms from the beginning, whereas in the later, they are revealed gradually based on their arrival time in the stream.
Understanding conflicts in online conversations
Authors: Sharon Levy, Robert E. Kraut, Jane A. Yu, Kristen M. Altenburger, Yi-Chia Wang
Venue: Conference WWW 2022
Year: 2022

With the rise of social media, users from across the world are able to connect and converse with each other online. While these connections have facilitated a growth in knowledge, online discussions can also end in acrimonious conflict. Previous computational studies have focused on creating online conflict detection models from inferred labels, primarily examine disagreement but not acrimony, and do not examine the conflict’s emergence. Social science studies have investigated offline conflict, which can differ from its online form, and rarely examines its emergence. The current research aims to understand how online conflicts arise in online personal conversations. Our ground truth is a Facebook tool that allows group members to report conflict to administrators. We contrast discussions ending with a conflict report with paired non-conflict discussions from the same post. We study both user characteristics (e.g., historical user-to-user interactions) and conversation dynamics (e.g., changes in emotional intensity over the course of the conversation). We use logistic regression to identify the features that predict conflict. User characteristics such as the commenter’s gender and previous involvement in negative online activity are strong indicators of conflict. Conversational dynamics, such as an increase in person-oriented discussion, are also important signals of conflict. These results help us understand how conflicts emerge and suggest better detection models and ways to alert group administrators and members early on to mediate the conversation.
Moral emotions shape the virality of covid-19 misinformation on social media
Authors: Kirill Solovev, Nicolas Pröllochs
Venue: Conference WWW 2022
Year: 2022

While false rumors pose a threat to the successful overcoming of the COVID-19 pandemic, an understanding of how rumors diffuse in online social networks is – even for non-crisis situations – still in its infancy. Here we analyze a large sample consisting of COVID-19 rumor cascades from Twitter that have been fact-checked by third-party organizations. The data comprises N = 10,610 rumor cascades that have been retweeted more than 24 million times. We investigate whether COVID-19 misinformation spreads more viral than the truth and whether the differences in the diffusion of true vs. false rumors can be explained by the moral emotions they carry. We observe that, on average, COVID-19 misinformation is more likely to go viral than truthful information. However, the veracity effect is moderated by moral emotions: false rumors are more viral than the truth if the source tweets embed a high number of other-condemning emotion words, whereas a higher number of self-conscious emotion words is linked to a less viral spread. The effects are pronounced both for health misinformation and false political rumors. These findings offer insights into how true vs. false rumors spread and highlight the importance of considering emotions from the moral emotion families in social media content.
Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellings
Authors: Georgios Sidiropoulos, Evangelos Kanoulas
Venue: Conference SIGIR 2022
Year: 2022

Dense retrieval is becoming one of the standard approaches for document and passage ranking. The dual-encoder architecture is widely adopted for scoring question-passage pairs due to its efficiency and high performance. Typically, dense retrieval models are evaluated on clean and curated datasets. However, when deployed in real-life applications, these models encounter noisy user-generated text. That said, the performance of state-of-the-art dense retrievers can substantially deteriorate when exposed to noisy text. In this work, we study the robustness of dense retrievers against typos in the user question. We observe a significant drop in the performance of the dual-encoder model when encountering typos and explore ways to improve its robustness by combining data augmentation with contrastive learning. Our experiments on two large-scale passage ranking and open-domain question answering datasets show that our proposed approach outperforms competing approaches. Additionally, we perform a thorough analysis on robustness. Finally, we provide insights on how different typos affect the robustness of embeddings differently and how our method alleviates the effect of some typos but not of others.
Efok-cqa: Towards knowledge graph complex query answering beyond set operation
Authors: Hang Yin, Zihao Wang, Weizhi Fei, Yangqiu Song
Venue: Conference KDD 2025
Year: 2025

To answer complex queries on knowledge graphs, logical reasoning over incomplete knowledge needs learning-based methods because they are capable of generalizing over unobserved knowledge. Therefore, an appropriate dataset is fundamental to both obtaining and evaluating such methods under this paradigm. In this paper, we propose a comprehensive framework for data generation, model training, and method evaluation that covers the combinatorial space of Existential First-order Queries with multiple variables (EFOk). The combinatorial query space in our framework significantly extends those defined by set operations in the existing literature. Additionally, we construct a dataset, EFOk-CQA, with 741 query types for empirical evaluation, and our benchmark results provide new insights into how query hardness affects the results. Furthermore, we demonstrate that the existing dataset construction process is systematically biased and hinders the appropriate development of query-answering methods, highlighting the importance of our work. Our code and data are provided in https://github.com/HKUST-KnowComp/EFOK-CQA.
A feature-oriented sentiment rating for mobile app reviews
Authors: Washington Luiz, Felipe Viegas, Rafael Odon de Alencar, Fernando Mourão, Thiago Salles, Dárlinton B. F. Carvalho, Marcos André Gonçalves, Leonardo Rocha
Venue: Conference WWW 2018
Year: 2018

In this paper, we propose a general framework that allows developers to filter, summarize and analyze user reviews written about applications on App Stores. Our framework extracts automatically relevant features from reviews of apps (e.g., information about functionalities, bugs, requirements, etc) and analyzes the sentiment associated with each of them. Our framework has three main building blocks, namely, (i) topic modeling, (ii) sentiment analysis and (iii) summarization interface. The topic modeling block aims at finding semantic topics from textual comments, extracting the target features based on the most relevant words of each discovered topic. The sentiment analysis block detects the sentiment associated with each discovered feature. The summarization interface provides to developers an intuitive visualization of the features (i.e., topics) and their associated sentiment, providing richer information than a 'star rating' strategy. Our evaluation shows that the topic modeling block is able to organize information provided by users in subcategories that facilitate the understanding of which features more positively/negatively impact the overall evaluation of the application. Regarding user satisfaction, we can observe that, in spite of the star rating being a good measure of evaluation, the Sentiment Analysis technique is more accurate in capturing the sentiment transmitted by the user by means of a comment.
Corpuslm: Towards a unified language model on corpus for knowledge-intensive tasks
Authors: Xiaoxi Li, Zhicheng Dou, Yujia Zhou, Fangchao Liu
Venue: Conference SIGIR 2024
Year: 2024

Large language models (LLMs) have gained significant attention in various fields but prone to hallucination, especially in knowledge-intensive (KI) tasks. To address this, retrieval-augmented generation (RAG) has emerged as a popular solution to enhance factual accuracy. However, traditional retrieval modules often rely on large document index and disconnect with generative tasks. With the advent of generative retrieval (GR), language models can retrieve by directly generating document identifiers (DocIDs), offering superior performance in retrieval tasks. However, the potential relationship between GR and downstream tasks remains unexplored. In this paper, we propose \textbf{CorpusLM}, a unified language model that leverages external corpus to tackle various knowledge-intensive tasks by integrating generative retrieval, closed-book generation, and RAG through a unified greedy decoding process. We design the following mechanisms to facilitate effective retrieval and generation, and improve the end-to-end effectiveness of KI tasks: (1) We develop a ranking-oriented DocID list generation strategy, which refines GR by directly learning from a DocID ranking list, to improve retrieval quality. (2) We design a continuous DocIDs-References-Answer generation strategy, which facilitates effective and efficient RAG. (3) We employ well-designed unsupervised DocID understanding tasks, to comprehend DocID semantics and their relevance to downstream tasks. We evaluate our approach on the widely used KILT benchmark with two variants of backbone models, i.e., T5 and Llama2. Experimental results demonstrate the superior performance of our models in both retrieval and downstream tasks.
Leveraging llms for detecting and modeling the propagation of misinformation in social networks
Authors: Payel Santra
Venue: Conference SIGIR 2024
Year: 2024

Recent success in language generation capabilities of large language models (LLMs), such as GPT, Llama, etc., can potentially lead to concern about their possible misuse in inducing mass agitation and communal hatred via generating fake news and spreading misinformation. Traditional means of developing a misinformation ground-truth dataset do not scale well because of the extensive manual effort required to annotate the data. It is crucial to anticipate and counteract potential adversarial fake information to mitigate detrimental effects and promote societal harmony. To this end, this PhD proposal spans three main research directions. The first concerns investigating ways of developing unsupervised models for fake news identification leveraging retrieval augmented generation (RAG) approaches. In our second thread of work, we explore ways of creating synthetic datasets to eventually train supervised or few-shot example-based models. Another direction of research work involves tracking the propagation of fake information through social networks to develop preventive measures against it.
Timeline summarization in the era of llms
Authors: Daivik Sojitra, Raghav Jain, Sriparna Saha, Adam Jatowt, Manish Gupta
Venue: Conference SIGIR 2024
Year: 2024

Timeline summarization is the task of automatically generating concise overviews of documents that capture the key events and their progression on timelines. While this capability is useful for quickly comprehending event sequences without reading lengthy descriptions, timeline summarization remains a relatively underexplored area in recent years when compared to traditional document summarization task and their evolution. The advent of large language models (LLMs) has led some to presume summarization as a solved problem. However, timeline summarization poses unique challenges for LLMs. Our investigation is centered on evaluating the performance of LLMs, against state-of-the-art models in this field. We employed three different approaches: chunking, knowledge graph-based summarization, and TimeRanker. Each of these methods was systematically tested on three benchmark datasets for timeline summarization to assess their effectiveness in capturing and condensing key events and their evolution within timelines. Our findings reveal that while LLMs show promise, timeline summarization remains a complex task that is not yet fully resolved.
Proactive recommendation with iterative preference guidance
Authors: Shuxian Bi, Wenjie Wang, Hang Pan, Fuli Feng, Xiangnan He
Venue: Conference WWW 2024
Year: 2024

Recommender systems mainly tailor personalized recommendations according to user interests learned from user feedback. However, such recommender systems passively cater to user interests and even reinforce existing interests in the feedback loop, leading to problems like filter bubbles and opinion polarization. To counteract this, proactive recommendation actively steers users towards developing new interests in a target item or topic by strategically modulating recommendation sequences. Existing work for proactive recommendation faces significant hurdles: 1) overlooking the user feedback in the guidance process; 2) lacking explicit modeling of the guiding objective; and 3) insufficient flexibility for integration into existing industrial recommender systems. To address these issues, we introduce an Iterative Preference Guidance (IPG) framework. IPG performs proactive recommendation in a flexible post-processing manner by ranking items according to their IPG scores that consider both interaction probability and guiding value. These scores are explicitly estimated with iteratively updated user representation that considers the most recent user interactions. Extensive experiments validate that IPG can effectively guide user interests toward target interests with a reasonable trade-off in recommender accuracy. The code is available at https://github.com/GabyUSTC/IPG-Rec.