Reviewerly/exHarmony huggingface dataset:
Towards Detecting Interesting Ideas Expressed in Text

In recent years, product and project ideas are often sourced from public competitions, where anyone can enter their own solutions to an open-ended question. While copious ideas can be gathered in this way, it becomes difficult to find the most promising results among all entries. This paper explores the potential of automating the detection of interesting ideas and studies the effect of various features of ideas on the prediction task. A BERT-based model is built to rank ideas by their predicted interestingness, using text embeddings from idea descriptions and the concreteness, novelty as well as the uniqueness of ideas. The model is trained on a dataset of OpenIDEO idea competitions. The results show that language models can be used to speed up finding promising ideas, but care must be taken in choosing a suitable dataset.
CPR: Cross-Domain Preference Ranking with User Transformation

Data sparsity is a well-known challenge in recommender systems. One way to alleviate this problem is to leverage knowledge from relevant domains. In this paper, we focus on an important real-world scenario in which some users overlap two different domains but items of the two domains are distinct. Although several studies leverage side information (e.g., user reviews) for cross-domain recommendation, side information is not always available or easy to obtain in practice. To this end, we propose cross-domain preference ranking (CPR) with a simple yet effective user transformation that leverages only user interactions with items in the source and target domains to transform the user representation. Given the proposed user transformation, CPR not only successfully enhances recommendation performance for users having interactions with target-domain items but also yields superior performance for cold-start users in comparison with state-of-the-art cross-domain recommendation approaches. Extensive experiments conducted on three pairs of cross-domain recommendation datasets demonstrate the effectiveness of the proposed method in comparison with existing cross-domain recommendation approaches. Our codes are available at https://github.com/cnclabs/codes.crossdomain.rec .
Towards Linguistically Informed Multi-objective Transformer Pre-training for Natural Language Inference

We introduce a linguistically enhanced combination of pre-training methods for transformers. The pre-training objectives include POS-tagging, synset prediction based on semantic knowledge graphs, and parent prediction based on dependency parse trees. Our approach achieves competitive results on the Natural Language Inference task, compared to the state of the art. Specifically for smaller models, the method results in a significant performance boost, emphasizing the fact that intelligent pre-training can make up for fewer parameters and help building more efficient models. Combining POS-tagging and synset prediction yields the overall best results.
Neural Approaches to Multilingual Information Retrieval

Providing access to information across languages has been a goal of Information Retrieval (IR) for decades. While progress has been made on Cross Language IR (CLIR) where queries are expressed in one language and documents in another, the multilingual (MLIR) task to create a single ranked list of documents across many languages is considerably more challenging. This paper investigates whether advances in neural document translation and pretrained multilingual neural language models enable improvements in the state of the art over earlier MLIR techniques. The results show that although combining neural document translation with neural ranking yields the best Mean Average Precision (MAP), 98% of that MAP score can be achieved with an 84% reduction in indexing time by using a pretrained XLM-R multilingual language model to index documents in their native language, and that 2% difference in effectiveness is not statistically significant. Key to achieving these results for MLIR is to fine-tune XLM-R using mixed-language batches from neural translations of MS MARCO passages.
Bootstrapped nDCG Estimation in the Presence of Unjudged Documents

Retrieval studies often reuse TREC collections after the corresponding tracks have passed. Yet, a fair evaluation of new systems that retrieve documents outside the original judgment pool is not straightforward. Two common ways of dealing with unjudged documents are to remove them from a ranking (condensed lists), or to treat them as non- or highly relevant (naïve lower and upper bounds). However, condensed list-based measures often overestimate the effectiveness of a system, and naïve bounds are often very “loose”—especially for nDCG when some top-ranked documents are unjudged. As a new alternative, we employ bootstrapping to generate a distribution of nDCG scores by sampling judgments for the unjudged documents using run-based and/or pool-based priors. Our evaluation on four TREC collections with real and simulated cases of unjudged documents shows that bootstrapped nDCG scores yield more accurate predictions than condensed lists, and that they are able to strongly tighten upper bounds at a negligible loss of accuracy.
Topics in Contextualised Attention Embeddings

Contextualised word vectors obtained via pre-trained language models encode a variety of knowledge that has already been exploited in applications. Complementary to these language models are probabilistic topic models that learn thematic patterns from the text. Recent work has demonstrated that conducting clustering on the word-level contextual representations from a language model emulates word clusters that are discovered in latent topics of words from Latent Dirichlet Allocation. The important question is how such topical word clusters are automatically formed, through clustering, in the language model when it has not been explicitly designed to model latent topics. To address this question, we design different probe experiments. Using BERT and DistilBERT, we find that the attention framework plays a key role in modelling such word topic clusters. We strongly believe that our work paves way for further research into the relationships between probabilistic topic models and pre-trained language models.
A Study on FGSM Adversarial Training for Neural Retrieval

Neural retrieval models have acquired significant effectiveness gains over the last few years compared to term-based methods. Nevertheless, those models may be brittle when faced to typos, distribution shifts or vulnerable to malicious attacks. For instance, several recent papers demonstrated that such variations severely impacted models performances, and then tried to train more resilient models. Usual approaches include synonyms replacements or typos injections – as data-augmentation – and the use of more robust tokenizers (characterBERT, BPE-dropout). To further complement the literature, we investigate in this paper adversarial training as another possible solution to this robustness issue. Our comparison includes the two main families of BERT-based neural retrievers, i.e. dense and sparse, with and without distillation techniques. We then demonstrate that one of the most simple adversarial training techniques – the Fast Gradient Sign Method (FGSM) – can improve first stage rankers robustness and effectiveness. In particular, FGSM increases models performances on both in-domain and out-of-domain distributions, and also on queries with typos, for multiple neural retrievers.
Sentence Retrieval for Open-Ended Dialogue Using Dual Contextual Modeling

We address the task of retrieving sentences for an open domain dialogue that contain information useful for generating the next turn. We propose several novel neural retrieval architectures based on dual contextual modeling: the dialogue context and the context of the sentence in its ambient document. The architectures utilize contextualized language models (BERT), fine-tuned on a large-scale dataset constructed from Reddit. We evaluate the models using a recently published dataset. The performance of our most effective model is substantially superior to that of strong baselines.
Leveraging Comment Retrieval for Code Summarization

Open-source code often suffers from mismatched or missing comments, leading to difficult code comprehension, and burdening software development and maintenance. In this paper, we design a novel code summarization model CodeFiD to address this laborious challenge. Inspired by retrieval-augmented methods for open-domain question answering, CodeFiD first retrieves a set of relevant comments from code collections for a given code, and then aggregates presentations of code and these comments to produce a natural language sentence that summarizes the code behaviors. Different from current code summarization works that focus on improving code representations, our model resorts to external knowledge to enhance code summarizing performance. Extensive experiments on public code collections demonstrate the effectiveness of CodeFiD by outperforming state-of-the-art counterparts across all programming languages.
A Study of Term-Topic Embeddings for Ranking

Contextualized representations from transformer models have significantly improved the performance of neural ranking models. Late interactions popularized by ColBERT and recently compressed with clustering in ColBERTv2 deliver state-of-the-art quality on many benchmarks. ColBERTv2 uses centroids along with occurrence-specific delta vectors to approximate contextualized embeddings without reducing ranking effectiveness. Analysis of this work suggests that these centroids are “term-topic embeddings”. We examine whether term-topic embeddings can be created in a differentiable end-to-end way, finding that this is a viable strategy for removing the separate clustering step. We investigate the importance of local context for contextualizing these term-topic embeddings, analogous to refining centroids with delta vectors. We find this end-to-end approach is sufficient for matching the effectiveness of the original contextualized embeddings.
Consumer Health Question Answering Using Off-the-Shelf Components

In this paper, we address the task of open-domain health question answering (QA). The quality of existing QA systems heavily depends on the annotated data that is often difficult to obtain, especially in the medical domain. To tackle this issue, we opt for PubMed and Wikipedia as trustworthy document collections to retrieve evidence. The questions and retrieved passages are passed to off-the-shelf question answering models, whose predictions are then aggregated into a final score. Thus, our proposed approach is highly data-efficient. Evaluation on 113 health-related yes/no question and answer pairs demonstrates good performance achieving AUC of 0.82.
Extractive Summarization of Financial Earnings Call Transcripts

To date, automatic summarization methods have been mostly developed for (and applied to) general news articles, whereas other document types have been neglected. In this paper, we introduce the task of summarizing financial earnings call transcripts, and we present a method for summarizing this text type essential for the financial industry. Earnings calls are briefing events common for public companies in many countries, typically in the form of conference calls held between company executives and analysts that consist of a spoken monologue part followed by moderated questions and answers. We show that traditional methods work less well in this domain, we present a method suitable for summarizing earnings calls. Our large-scale evaluation on a new human-annotated corpus of summary-worthy sentences shows that this method outperforms a set of strong baselines, including a new one that we propose specifically for earnings calls. To the best of our knowledge, this is the first application of summarization to financial earnings calls transcripts, a primary source of information for financial professionals.
New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches

In evaluation campaigns, participants often explore variations of popular, state-of-the-art baselines as a low-risk strategy to achieve competitive results. While effective, this can lead to local “hill climbing” rather than a more radical and innovative departure from standard methods. Moreover, if many participants build on similar baselines, the overall diversity of approaches considered may be limited. In this work, we propose a new class of IR evaluation metrics intended to promote greater diversity of approaches in evaluation campaigns. Whereas traditional IR metrics focus on user experience, our two “innovation” metrics instead reward exploration of more divergent, higher-risk strategies finding relevant documents missed by other systems. Experiments on four TREC collections show that our metrics do change system rankings by rewarding systems that find such rare, relevant documents. This result is further supported by a controlled, synthetic data experiment, and a qualitative analysis. In addition, we show that our metrics achieve higher evaluation stability and discriminative power than the standard metrics we modify. To support reproducibility, we share our source code.
Feature Differentiation and Fusion for Semantic Text Matching

Semantic Text Matching (STM for short) stands for the task of automatically determining the semantic similarity for a pair of texts. It has been widely applied in a variety of downstream tasks, e.g., information retrieval and question answering. The most recent works of STM leverage Pre-trained Language Models (abbr., PLMs) due to their remarkable capacity for representation learning. Accordingly, significant improvements have been achieved. However, our findings show that PLMs fail to capture task-specific features that signal hardly-perceptible changes in semantics. To overcome the issue, we propose a two-channel Feature Differentiation and Fusion network (FDF). It utilizes a PLM-based encoder to extract features separately from the unabridged texts and those abridged by deduplication. On this basis, gated feature fusion and interaction are conducted across the channels to expand text representations with attentive and distinguishable features. Experiments on the benchmarks QQP, MRPC and BQ show that FDF obtains substantial improvements compared to the baselines and outperforms the state-of-the-art STM models.
Generating Topic Pages for Scientific Concepts Using Scientific Publications

In this paper, we describe Topic Pages, an inventory of scientific concepts and information around them extracted from a large collection of scientific books and journals. The main aim of Topic Pages is to provide all the necessary information to the readers to understand scientific concepts they come across while reading scholarly content in any scientific domain. Topic Pages are a collection of automatically generated information pages using NLP and ML, each corresponding to a scientific concept. Each page contains three pieces of information: a definition, related concepts, and the most relevant snippets, all extracted from scientific peer-reviewed publications. In this paper, we discuss the details of different components to extract each of these elements. The collection of pages in production contains over 360, 000 Topic Pages across 20 different scientific domains with an average of 23 million unique visits per month, constituting it a popular source for scientific information.
The Impact of a Popularity Punishing Hyperparameter on ItemKNN Recommendation Performance

Collaborative filtering techniques have a tendency to amplify popularity biases present in the training data if no countermeasures are taken. The ItemKNN algorithm with conditional probability-inspired similarity function has a hyperparameter $$\alpha $$ that allows one to counteract this popularity bias. In this work, we perform a deep dive into the effects of this hyperparameter in both online and offline experiments, with regard to both accuracy metrics and equality of exposure. Our experiments show that the hyperparameter can indeed counteract popularity bias in a dataset. We also find that there exists a trade-off between countering popularity bias and the quality of the recommendations: Reducing popularity bias too much results in a decrease in click-through rate, but some counteracting of popularity bias is required for optimal online performance.
Don’t Raise Your Voice, Improve Your Argument: Learning to Retrieve Convincing Arguments

The Information Retrieval community has made strides in developing neural rankers, which have show strong retrieval effectiveness on large-scale gold standard datasets. The focus of existing neural rankers has primarily been on measuring the relevance of a document or passage to the user query. However, other considerations such as the convincingness of the content are not taken into account when retrieving content. We present a large gold standard dataset, referred to as CoRe, which focuses on enabling researchers to explore the integration of the concepts of convincingness and relevance to allow for the retrieval of relevant yet persuasive content. Through extensive experiments on this dataset, we report that there is a close association between convincingness and relevance that can have practical value in how convincing content are presented and retrieved in practice.
Joint Extraction and Classification of Danish Competences for Job Matching

The matching of competences, such as skills, occupations or knowledges, is a key desiderata for candidates to be fit for jobs. Automatic extraction of competences from CVs and Jobs can greatly promote recruiters’ productivity in locating relevant candidates for job vacancies. This work presents the first model that jointly extracts and classifies competence from Danish job postings. Different from existing works on skill extraction and skill classification, our model is trained on a large volume of annotated Danish corpora and is capable of extracting a wide range of danish competences, including skills, occupations and knowledges of different categories. More importantly, as a single BERT-like architecture for joint extraction and classification, our model is lightweight and efficient at inference. On a real-scenario job matching dataset, our model beats the state-of-the-art models in the overall performance of Danish competence extraction and classification, and saves over 50% time at inference.
Fragmented Visual Attention in Web Browsing: Weibull Analysis of Item Visit Times

Abstract Users often browse the web in an exploratory way, inspecting what they find interesting without a specific goal. However, the temporal dynamics of visual attention during such sessions, emerging when users gaze from one item to another, are not well understood. In this paper, we examine how people distribute visual attention among content items when browsing news. Distribution of visual attention is studied in a controlled experiment, wherein eye-tracking data and web logs are collected for 18 participants exploring newsfeeds in a single- and multi-column layout. Behavior is modeled using Weibull analysis of item (article) visit times, which describes these visits via quantities like durations and frequencies of switching focused item. Bayesian inference is used to quantify uncertainty. The results suggest that visual attention in browsing is fragmented, and affected by the number, properties and composition of the items visible on the viewport. We connect these findings to previous work explaining information-seeking behavior through cost-benefit judgments.
Joint Span Segmentation and Rhetorical Role Labeling with Data Augmentation for Legal Documents

Segmentation and Rhetorical Role Labeling of legal judgements play a crucial role in retrieval and adjacent tasks, including case summarization, semantic search, argument mining etc. Previous approaches have formulated this task either as independent classification or sequence labeling of sentences. In this work, we reformulate the task at span level as identifying spans of multiple consecutive sentences that share the same rhetorical role label to be assigned via classification. We employ semi-Markov Conditional Random Fields (CRF) to jointly learn span segmentation and span label assignment. We further explore three data augmentation strategies to mitigate the data scarcity in the specialized domain of law where individual documents tend to be very long and annotation cost is high. Our experiments demonstrate improvement of span-level prediction metrics with a semi-Markov CRF model over a CRF baseline. This benefit is contingent on the presence of multi sentence spans in the document.
Comparison Analysis of LSTM and CNN Variants with Embedding Word Methods for Sentiment Analysis on Food Consumption Behavior

Lockdowns, working from home, staying at home, and physical distance are expected to significantly impact consumer attitudes and behaviors during the COVID-19 pandemic. During the implementation of the Movement Control Order, Malaysians’ food preferences are already shifting away, influencing new consumption behavior. Since it has played a significant role in many areas of natural language, mainly using social media data from Twitter, there has been increased interest in sentiment analysis in recent years. However, research on the performance of various sentiment analysis methodologies such as n-gram ranges, lexicon techniques, deep learning, word embedding, and hybrid methods within this domain-specific sentiment is limited. This study evaluates several approaches to determine the best approach for tweets on food consumption behavior in Malaysia during the COVID-19 pandemic. This study combined unigram and bigram ranges with two lexicon-based techniques, TextBlob and VADER, and three deep learning classi-fiers, Long Short-Term Memory Network (LSTM), Convolutional Neural Networks (CNN), and their hybridization. Word2Vector and GloVe are two-word embedding approaches used by LSTM-CNN. The embedding GloVe on TextBlob approach with a combination of Unigram + Bigram [1,2] range produced the best results, with 85.79% accuracy and 85.30% F1-score. According to these findings, LSTM outperforms other classifiers because it achieves the highest scores for both performance metrics. The classification performance can be improved in future studies if the dataset is more evenly distributed across each positive and negative label.
The Characterization of Rainfall Data Set Using Persistence Diagram and Its Relation to Extreme Events: Case Study of Three Locations in Kemaman, Terengganu

Floods are recurring phenomena at certain locations because of excessive rainfall, resulting in the overflow of lakes, drains, and rivers. In this work, we employ Persistence Homology (PH) to investigate the relationship between rainfall and flood that occurred from 1997 to 2018. Three stations in Kemaman, Terengganu, have been chosen to study this relationship. Persistence Diagram (PD) is one of the most powerful tools available under the umbrella of PH for detecting topological signatures in high dimension points cloud. In this paper, we use the rainfall time series dataset and express it in higher dimensions by selecting the embedded dimension, $$M = 5$$ , , and manipulating the time delay τ to obtain the maximum persistence. Then, we compared with past flood events which are labelled based on water level and PD’s max score to identify its suitability for flood identification. The area under the curve of receiver operation characteristics (ROC) have been used to measure the performance with three different thresholds for station 4131001, 4232002, and 4332001. The results clearly show PD’s significance to characterize the rainfall dataset as normal and flood events. The employed maximum persistence is robust despite missing data values.