Reviewerly/exHarmony huggingface dataset:
MS-Shift: An Analysis of MS MARCO Distribution Shifts on Neural Retrieval

Pre-trained Language Models have recently emerged in Information Retrieval as providing the backbone of a new generation of neural systems that outperform traditional methods on a variety of tasks. However, it is still unclear to what extent such approaches generalize in zero-shot conditions. The recent BEIR benchmark provides partial answers to this question by comparing models on datasets and tasks that differ from the training conditions. We aim to address the same question by comparing models under more explicit distribution shifts. To this end, we build three query-based distribution shifts within MS MARCO (query-semantic, query-intent, query-length), which are used to evaluate the three main families of neural retrievers based on BERT: sparse, dense, and late-interaction – as well as a monoBERT re-ranker. We further analyse the performance drops between the train and test query distributions. In particular, we experiment with two generalization indicators: the first one based on train/test query vocabulary overlap, and the second based on representations of a trained bi-encoder. Intuitively, those indicators verify that the further away the test set is from the train one, the worse the drop in performance. We also show that models respond differently to the shifts – dense approaches being the most impacted. Overall, our study demonstrates that it is possible to design more controllable distribution shifts as a tool to better understand generalization of IR models. Finally, we release the MS MARCO query subsets, which provide an additional resource to benchmark zero-shot transfer in Information Retrieval.
Service Is Good, Very Good or Excellent? Towards Aspect Based Sentiment Intensity Analysis

Aspect-based sentiment analysis (ABSA) is a fast-growing research area in natural language processing (NLP) that provides more fine-grained information, considering the aspect as the fundamental item. The ABSA primarily measures sentiment towards a given aspect, but does not quantify the intensity of that sentiment. For example, intensity of positive sentiment expressed for service in service is good is comparatively weaker than in service is excellent. Thus, aspect sentiment intensity will assist the stakeholders in mining user preferences more precisely. Our current work introduces a novel task called aspect based sentiment intensity analysis (ABSIA) that facilitates research in this direction. An annotated review corpus for ABSIA is introduced by labelling the benchmark SemEval ABSA restaurant dataset with the seven (7) classes in a semi-supervised way. To demonstrate the effective usage of corpus, we cast ABSIA as a natural language generation task, where a natural sentence is generated to represent the output in order to utilize the pre-trained language models effectively. Further, we propose an effective technique for the joint learning where ABSA is used as a secondary task to assist the primary task, i.e. ABSIA. An improvement of 2 points is observed over the single task intensity model. To explain the actual decision process of the proposed framework, model explainability technique is employed that extracts the important opinion terms responsible for generation (Source code and the dataset has been made available on https://www.iitp.ac.in/~ai-nlp-ml/resources.html#ABSIA , https://github.com/20118/ABSIA )
Utilising Twitter Metadata for Hate Classification

Social media has become an essential daily feature of people's lives. Social media platforms provide individuals wishing to cause harm with an open, anonymous, and far-reaching channel. As a result, society is experiencing a crisis concerning hate and abuse on social media. This paper aims to provide a better method of identifying these instances of hate via a custom BERT classifier which leverages readily available metadata from Twitter alongside traditional text data. With Accuracy, F1, Recall and Precision scores of 0.85, 0.75, 0.76, and 0.74, the new model presents a competitive performance compared to similar state-of-the-art models. The increased performance of models within this domain can only benefit society as they provide more effective means to combat hate on social media.
Multimodal Geolocation Estimation of News Photos

The widespread growth of multimodal news requires sophisticated approaches to interpret content and relations of different modalities. Images are of utmost importance since they represent a visual gist of the whole news article. For example, it is essential to identify the locations of natural disasters for crisis management or to analyze political or social events across the world. In some cases, verifying the location(s) claimed in a news article might help human assessors or fact-checking efforts to detect misinformation, i.e., fake news. Existing methods for geolocation estimation typically consider only a single modality, e.g., images or text. However, news images can lack sufficient geographical cues to estimate their locations, and the text can refer to various possible locations. In this paper, we propose a novel multimodal approach to predict the geolocation of news photos. To enable this approach, we introduce a novel dataset called Multimodal Geolocation Estimation of News Photos (MMG-NewsPhoto). MMG-NewsPhoto is, so far, the largest dataset for the given task and contains more than half a million news texts with the corresponding image, out of which 3000 photos were manually labeled for the photo geolocation based on information from the image-text pairs. For a fair comparison, we optimize and assess state-of-the-art methods using the new benchmark dataset. Experimental results show the superiority of the multimodal models compared to the unimodal approaches.
CoLISA: Inner Interaction via Contrastive Learning for Multi-choice Reading Comprehension

Multi-choice reading comprehension (MC-RC) is supposed to select the most appropriate answer from multiple candidate options by reading and comprehending a given passage and a question. Recent studies dedicate to catching the relationships within the triplet of passage, question, and option. Nevertheless, one limitation in current approaches relates to the fact that confusing distractors are often mistakenly judged as correct, due to the fact that models do not emphasize the differences between the answer alternatives. Motivated by the way humans deal with multi-choice questions by comparing given options, we propose CoLISA (Contrastive Learning and In-Sample Attention), a novel model to prudently exclude the confusing distractors. In particular, CoLISA acquires option-aware representations via contrastive learning on multiple options. Besides, in-sample attention mechanisms are applied across multiple options so that they can interact with each other. The experimental results on QuALITY and RACE demonstrate that our proposed CoLISA pays more attention to the relation between correct and distractive options, and recognizes the discrepancy between them. Meanwhile, CoLISA also reaches the state-of-the-art performance on QuALITY (Our code is available at https://github.com/Walle1493/CoLISA. .).
Trigger or not Trigger: Dynamic Thresholding for Few Shot Event Detection

Recent studies in few-shot event trigger detection from text address the task as a word sequence annotation task using prototypical networks. In this context, the classification of a word is based on the similarity of its representation to the prototypes built for each event type and for the “non-event” class (also named null class). However, the “non-event” prototype aggregates by definition a set of semantically heterogeneous words, which hurts the discrimination between trigger and non-trigger words. We address this issue by handling the detection of non-trigger words as an out-of-domain (OOD) detection problem and propose a method for dynamically setting a similarity threshold to perform this detection. Our approach increases f-score by about 10 points on average compared to the state-of-the-art methods on three datasets.
What Is Your Cause for Concern? Towards Interpretable Complaint Cause Analysis

The abundance of information available on social media and the regularity with which complaints are posted online emphasizes the need for automated complaint analysis tools. Prior study has focused chiefly on complaint identification and complaint severity prediction: the former attempts to classify a piece of content as either complaint or non-complaint. The latter seeks to group complaints into various severity classes depending on the threat level that the complainant is prepared to accept. The complainant’s goal could be to express disapproval, seek compensation, or both. As a result, the complaint detection model should be interpretable or explainable. Recognizing the cause of a complaint in the text is a crucial yet untapped area of natural language processing research. We propose an interpretable complaint cause analysis model that is grounded on a dyadic attention mechanism. The model jointly learns complaint classification, emotion recognition, and polarity classification as the first sub-problem. Subsequently, the complaint cause extraction and the associated severity level prediction as the second sub-problem. We add the causal span annotation for the existing complaint classes in a publicly available complaint dataset to accomplish this. The results indicate that existing computational tools can be repurposed to tackle highly relevant novel tasks, thereby finding new research opportunities (Resources available at: https://bit.ly/Complaintcauseanalysis ).
De-biasing Relevance Judgements for Fair Ranking

The objective of this paper is to show that it is possible to significantly reduce stereotypical gender biases in neural rankers without modifying the ranking loss function, which is the current approach in the literature. We systematically de-bias gold standard relevance judgement datasets with a set of balanced and well-matched query pairs. Such a de-biasing process will expose neural rankers to comparable queries from across gender identities that have associated relevant documents with compatible degrees of gender bias. Therefore, neural rankers will learn not to associate varying degrees of bias to queries from certain gender identities. Our experiments show that our approach is able to (1) systematically reduces gender biases associated with different gender identities, and (2) at the same time maintain the same level of retrieval effectiveness.
Capturing Cross-Platform Interaction for Identifying Coordinated Accounts of Misinformation Campaigns

Recent years have witnessed the increasing abuse of coordinated accounts on multiple social media platforms. Such accounts are usually operated by misinformation campaigns to manipulate the public opinions on different platforms jointly. However, existing methods mainly focus on detecting such accounts by capturing the coordinated activities within a single platform. As a result, their performances are limited as they can not make use of the information from other platforms. In this work, we propose that capturing cross-platform coordinated activities can bring a significant boost to identifying the accounts operated by misinfromation campaigns. To leverage such information in a practical way, we design a novel Conditional Gaussian-distribution Basis to extract cross-platform correlation from Coordinated Activity Set, which can be easily acquired. Experimental results indicate that our methodology outperform baselines and its own variants that can not leverage cross-platform information.
Augmenting Graph Convolutional Networks with Textual Data for Recommendations

Graph Convolutional Networks have recently shown state-of-the-art performance for collaborative filtering-based recommender systems. However, many systems use a pure user-item bipartite interaction graph, ignoring available additional information about the items and users. This paper proposes an effective and general method, TextGCN, that utilizes rich textual information about the graph nodes, specifically user reviews and item descriptions, using pre-trained text embeddings. We integrate those reviews and descriptions into item recommendations to augment graph embeddings obtained using LightGCN, a SOTA graph network. Our model achieves a 7–23% statistically significant improvement over this SOTA baseline when evaluated on several diverse large-scale review datasets. Furthermore, our method captures semantic signals from the text, which are not available when using graph connections alone.
Evaluating Humorous Response Generation to Playful Shopping Requests

AI assistants are gradually becoming embedded in our lives, utilized for everyday tasks like shopping or music. In addition to the everyday utilization of AI assistants, many users engage them with playful shopping requests, gauging their ability to understand – or simply seeking amusement. However, these requests are often not being responded to in the same playful manner, causing dissatisfaction and even trust issues. In this work, we focus on equipping AI assistants with the ability to respond in a playful manner to irrational shopping requests. We first evaluate several neural generation models, which lead to unsuitable results – showing that this task is non-trivial. We devise a simple, yet effective, solution, that utilizes a knowledge graph to generate template-based responses grounded with commonsense. While the commonsense-aware solution is slightly less diverse than the generative models, it provides better responses to playful requests. This emphasizes the gap in commonsense exhibited by neural language models.
Time-Dependent Next-Basket Recommendations

There are various real-world applications for next-basket recommender systems. One of them is guiding a website user who wants to buy anything toward a collection of items. Recent works demonstrate that methods based on the frequency of prior purchases outperform other deep learning algorithms in terms of performance. These techniques, however, do not consider timestamps and time intervals between interactions. Additionally, they often miss the time period that passes between the last known basket and the prediction time. In this study, we explore whether such knowledge could improve current state-of-the-art next-basket recommender systems. Our results on three real-world datasets show how such enhancement may increase prediction quality. These findings might pave the way for important research directions in the field of next-basket recommendations.
Learning Query-Space Document Representations for High-Recall Retrieval

Recent studies have shown that significant performance improvements reported by neural rankers do not necessarily extend to a diverse range of queries. There is a large set of queries that cannot be effectively addressed by neural rankers primarily because relevant documents to these queries are not identified by first-stage retrievers. In this paper, we propose a novel document representation approach that represents documents within the query space, and hence increases the likelihood of recalling a higher number of relevant documents. Based on experiments on the MS MARCO dataset as well as the hardest subset of its queries, we find that the proposed approach shows synergistic behavior to existing neural rankers and is able to increase recall both on MS MARCO dev set queries as well as the hardest queries of MS MARCO.
Domain-Aligned Data Augmentation for Low-Resource and Imbalanced Text Classification

Data Augmentation approaches often use Language Models, pretrained on large quantities of unlabeled generic data, to conditionally generate examples. However, the generated data can be of subpar quality and struggle to maintain the same characteristics as the original dataset. To this end, we propose a Data Augmentation method for low-resource and imbalanced datasets, by aligning Language Models to in-domain data prior to generating synthetic examples. In particular, we propose the alignment of existing generic models in task-specific unlabeled data, in order to create better synthetic examples and boost performance in Text Classification tasks. We evaluate our approach on three diverse and well-known Language Models, four datasets, and two settings (i.e. imbalance and low-resource) in which Data Augmentation is usually deployed, and study the correlation between the amount of data required for alignment, model size, and its effects in downstream in-domain and out-of-domain tasks. Our results showcase that in-domain alignment helps create better examples and increase the performance in Text Classification. Furthermore, we find a positive connection between the number of training parameters in Language Models, the volume of fine-tuning data, and their effects in downstream tasks.
Towards Effective Paraphrasing for Information Disguise

Information Disguise (ID), a part of computational ethics in Natural Language Processing (NLP), is concerned with best practices of textual paraphrasing to prevent the non-consensual use of authors' posts on the Internet. Research on ID becomes important when authors' written online communication pertains to sensitive domains, e.g., mental health. Over time, researchers have utilized AI-based automated word spinners (e.g., SpinRewriter, WordAI) for paraphrasing content. However, these tools fail to satisfy the purpose of ID as their paraphrased content still leads to the source when queried on search engines. There is limited prior work on judging the effectiveness of paraphrasing methods for ID on search engines or their proxies, neural retriever (NeurIR) models. We propose a framework where, for a given sentence from an author's post, we perform iterative perturbation on the sentence in the direction of paraphrasing with an attempt to confuse the search mechanism of a NeurIR system when the sentence is queried on it. Our experiments involve the subreddit 'r/AmItheAsshole' as the source of public content and Dense Passage Retriever as a NeurIR system-based proxy for search engines. Our work introduces a novel method of phrase-importance rankings using perplexity scores and involves multi-level phrase substitutions via beam search. Our multi-phrase substitution scheme succeeds in disguising sentences 82% of the time and hence takes an essential step towards enabling researchers to disguise sensitive content effectively before making it public. We also release the code of our approach.
Topic-Enhanced Personalized Retrieval-Based Chatbot

Building a personalized chatbot has drawn much attention recently. A personalized chatbot is considered to have a consistent personality. There are two types of methods to learn the personality. The first mainly model the personality from explicit user profiles (e.g., manually created persona descriptions). The second learn implicit user profiles from the user's dialogue history, which contains rich, personalized information. However, a user's dialogue history can be long and noisy as it contains long-time, multi-topic historical dialogue records. Such data noise and redundancy impede the model's ability to thoroughly and faithfully learn a consistent personality, especially when applied with models that have an input length limit (e.g., BERT). In this paper, we propose deconstructing the long and noisy dialogue history into topic-dependent segments. We only use the topically related dialogue segment as context to learn the topic-aware user personality. Specifically, we design a Topic-enhanced personalized Retrieval-based Chatbot, TopReC. It first deconstructs the dialogue history into topic-dependent dialogue segments and filters out irrelevant segments to the current query via a Heter-Merge-Reduce framework. It then measures the matching degree between the response candidates and the current query conditioned on each topic-dependent segment. We consider the matching degree between the response candidate and the cross-topic user personality. The final matching score is obtained by combining the topic-dependent and cross-topic matching scores. Experimental results on two large dataset show that TopReC outperforms all previous state-of-the-art methods.
Exploiting Graph Structured Cross-Domain Representation for Multi-domain Recommendation

Multi-domain recommender systems benefit from cross-domain representation learning and positive knowledge transfer. Both can be achieved by introducing a specific modeling of input data (i.e. disjoint history) or trying dedicated training regimes. At the same time, treating domains as separate input sources becomes a limitation as it does not capture the interplay that naturally exists between domains. In this work, we efficiently learn multi-domain representation of sequential users’ interactions using graph neural networks. We use temporal intra- and inter-domain interactions as contextual information for our method called MAGRec (short for Multi-dom Ain Graph-based Recommender). To better capture all relations in a multi-domain setting, we learn two graph-based sequential representations simultaneously: domain-guided for recent user interest, and general for long-term interest. This approach helps to mitigate the negative knowledge transfer problem from multiple domains and improve overall representation. We perform experiments on publicly available datasets in different scenarios where MAGRec consistently outperforms state-of-the-art methods. Furthermore, we provide an ablation study and discuss further extensions of our method.
On the Current State of Reproducibility and Reporting of Uncertainty for Aspect-Based Sentiment Analysis

Abstract For the latter part of the past decade, Aspect-Based Sentiment Analysis has been a field of great interest within Natural Language Processing. Supported by the Semantic Evaluation Conferences in 2014–2016, a variety of methods has been developed competing in improving performances on benchmark data sets. Exploiting the transformer architecture behind BERT, results improved rapidly and efforts in this direction still continue today. Our contribution to this body of research is a holistic comparison of six different architectures which achieved (near) state-of-the-art results at some point in time. We utilize a broad spectrum of five publicly available benchmark data sets and introduce a fixed setting with respect to the pre-processing, the train/validation splits, the performance measures and the quantification of uncertainty. Overall, our findings are two-fold: First, we find that the results reported in the scientific articles are hardly reproducible, since in our experiments the observed performance most of the time fell short of the reported one. Second, the results are burdened with notable uncertainty, depending on the data splits, which is why a reporting of uncertainty measures is crucial.