Reviewerly/exHarmony huggingface dataset:
Exploiting Abstract Meaning Representation for Open-Domain Question Answering

The Open-Domain Question Answering (ODQA) task involves retrieving and subsequently generating answers from fine-grained relevant passages within a database.Current systems leverage Pretrained Language Models (PLMs) to model the relationship between questions and passages.However, the diversity in surface form expressions can hinder the model's ability to capture accurate correlations, especially within complex contexts.Therefore, we utilize Abstract Meaning Representation (AMR) graphs to assist the model in understanding complex semantic information.We introduce a method known as Graph-as-Token (GST) to incorporate AMRs into PLMs.Results from Natural Questions (NQ) and TriviaQA (TQ) demonstrate that our GST method can significantly improve performance, resulting in up to 2.44/3.17Exact Match score improvements on NQ/TQ respectively.Furthermore, our method enhances robustness and outperforms alternative Graph Neural Network (GNN) methods for integrating AMRs.To the best of our knowledge, we are the first to employ semantic graphs in ODQA.
Making Pre-trained Language Models Better Learn Few-Shot Spoken Language Understanding in More Practical Scenarios

Most previous few-shot Spoken Language Understanding (SLU) models typically need to be trained on a set of data-rich source domains and adapt to the target domain with a few examples.In this paper, we explore a more practical scenario for few-shot SLU, in which we only assume access to a pre-trained language model and a few labeled examples without any other source domain data.We concentrate on understanding how far the few-shot SLU could be pushed in this setting.To this end, we develop a prompt-based intent detection model in fewshot settings, which leverages the BERT original pre-training next sentence prediction task and the prompt template to detect the user's intent.For slot filling, we propose an approach of reconstructing slot labels, which reduces the training complexity by reducing the number of slot labels in few-shot settings.To evaluate the few-shot SLU for a more practical scenario, we present two benchmarks, FewShotATIS and FewShotSNIPS.And a dynamic sampling strategy is designed to construct the two datasets according to the learning difficulty of each intent and slot.Experiments on FewShotATIS and FewShotSNIPS demonstrate that our proposed model achieves state-of-the-art performance.
Towards Distribution-shift Robust Text Classification of Emotional Content

Supervised models based on Transformers have been shown to achieve impressive performances in many natural language processing tasks.However, besides requiring a large amount of costly manually annotated data, supervised models tend to adapt to the characteristics of the training dataset, which are usually created ad-hoc and whose data distribution often differs from the one in real applications, showing significant performance degradation in real-world scenarios.We perform an extensive assessment of the out-of-distribution performances of supervised models for classification in the emotion and hate-speech detection tasks and show that NLI-based zeroshot models often outperform them, making task-specific annotation useless when the characteristics of final-user data are not known in advance.To benefit from both supervised and zero-shot approaches, we propose to finetune an NLI-based model on the task-specific dataset.The resulting model often outperforms all available supervised models both in distribution and out of distribution, with only a few thousand training samples.Recent zero-shot models (Yin et al., 2019;Liu et al., 2021) have gained popularity thanks to their ability to reduce the dependency on task-specific annotated data by enabling models to predict previously unseen labels.For instance, models trained for Next Sentence Prediction (NLP) or Natural Language Inference (NLI) tasks can be applied to infer
Can Language Models Make Fun? A Case Study in Chinese Comical Crosstalk

Language is the principal tool for human communication, in which humor is one of the most attractive parts.Producing natural language like humans using computers, a.k.a,Natural Language Generation (NLG), has been widely used for dialogue systems, chatbots, text summarization, as well as AI-Generated Content (AIGC), e.g., idea generation, and scriptwriting.However, the humor aspect of natural language is relatively under-investigated, especially in the age of pre-trained language models.In this work, we aim to preliminarily test whether NLG can generate humor as humans do.We build the largest dataset consisting of numerous Chinese Comical Crosstalk scripts (called C 3 in short), which is for a popular Chinese performing art called 'Xiangsheng' or '相 声' since 1800s 1 .We benchmark various generation approaches including training-from-scratch Seq2seq, fine-tuned middle-scale PLMs, and large-scale PLMs with and without fine-tuning.Moreover, we also conduct a human assessment, showing that 1) large-scale pretraining largely improves crosstalk generation quality; and 2) even the scripts generated from the best PLM is far from what we expect.We conclude humor generation could be largely improved using large-scale PLMs, but it is still in its infancy.The data and benchmarking code are publicly available in https://github.com/anonNo2/crosstalk-generation.* Benyou is the corresponding author. 1 For convenience for non-Chinese speakers, we called 'crosstalk' for 'Xiangsheng' in this paper.
AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation

Crowdsourced dialogue corpora are usually limited in scale and topic coverage due to the expensive cost of data curation.This would hinder the generalization of downstream dialogue models to open-domain topics.In this work, we leverage large language models for dialogue augmentation in the task of emotional support conversation (ESC).By treating dialogue augmentation as a dialogue completion task, we prompt a fine-tuned language model to complete full dialogues from available dialogue posts of various topics, which are then postprocessed based on heuristics.Applying this approach, we construct AUGESC, an augmented dataset for the ESC task, which largely extends the scale and topic coverage of the crowdsourced ESConv corpus.Through comprehensive human evaluation, we demonstrate that our approach is superior to strong baselines of dialogue augmentation and that AUGESC has comparable dialogue quality to the crowdsourced corpus.We also conduct human interactive evaluation and prove that post-training on AUGESC improves downstream dialogue models' generalization ability to open-domain topics.These results suggest the utility of AUGESC and highlight the potential of large language models in improving data-scarce dialogue generation tasks.
M3: A Multi-View Fusion and Multi-Decoding Network for Multi-Document Reading Comprehension

Multi-document reading comprehension task requires collecting evidences from different documents for answering questions.Previous research works either use the extractive modeling method to naively integrate the scores from different documents on the encoder side or use the generative modeling method to collect the clues from different documents on the decoder side individually.However, any single modeling method cannot make full of the advantages of both.In this work, we propose a novel method that tries to employ a multi-view fusion and multi-decoding mechanism to achieve it.For one thing, our approach leverages question-centered fusion mechanism and cross-attention mechanism to gather finegrained fusion of evidence clues from different documents in the encoder and decoder concurrently.For another, our method simultaneously employs both the extractive decoding approach and the generative decoding method to effectively guide the training process.Compared with existing methods, our method can perform both extractive decoding and generative decoding independently and optionally.Our experiments on two mainstream multi-document reading comprehension datasets (Natural Questions and Triv-iaQA) demonstrate that our method can provide consistent improvements over previous state-of-the-art methods.
Deploying a Retrieval based Response Model for Task Oriented Dialogues

Task-oriented dialogue systems in industry settings need to have high conversational capability, be easily adaptable to changing situations and conform to business constraints. This paper describes a 3-step procedure to develop a conversational model that satisfies these criteria and can efficiently scale to rank a large set of response candidates. First, we provide a simple algorithm to semi-automatically create a high-coverage template set from historic conversations without any annotation. Second, we propose a neural architecture that encodes the dialogue context and applicable business constraints as profile features for ranking the next turn. Third, we describe a two-stage learning strategy with self-supervised training, followed by supervised fine-tuning on limited data collected through a human-in-the-loop platform. Finally, we describe offline experiments and present results of deploying our model with human-in-the-loop to converse with live customers online.
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Recent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment 1 .However, they either perform turn-level evaluation or look at a single dialogue quality dimension.One would expect a good evaluation metric to assess multiple quality dimensions at the dialogue level.To this end, we are motivated to propose a multi-dimensional dialogue-level metric, which consists of three sub-metrics with each targeting a specific dimension.The submetrics are trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimensions.Moreover, we explore two approaches to combine the sub-metrics: metric ensemble and multitask learning.Both approaches yield a holistic metric that significantly outperforms individual sub-metrics.Compared to the existing state-of-the-art metric, the combined metrics achieve around 16% relative improvement on average across three high-quality dialoguelevel evaluation benchmarks.
Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study

Imbalanced data distribution is a practical and common challenge in building production-level machine learning (ML) models in industry, where data usually exhibits long-tail distributions. For instance, in virtual AI Assistants, such as Google Assistant, Amazon Alexa and Apple Siri, the “play music” or “set timer” utterance is exposed to an order of magnitude more traffic than other skills. This can easily cause trained models to overfit to the majority classes, categories or intents, lead to model miscalibration. The uncalibrated models output unreliable (mostly overconfident) predictions, which are at high risk of affecting downstream decision-making systems. In this work, we study the calibration of production models in the industry use-case of predicting product return reason codes in customer service conversations of an online retail store; The returns reasons also exhibit class imbalance.To alleviate the resulting miscalibration in the production ML model, we streamline the model development and deployment using focal loss (CITATION).We empirically show the effectiveness of model training with focal loss in learning better calibrated models, as compared to standard cross-entropy loss. Better calibration, in turn, enables better control of the precision-recall trade-off for the models deployed in production.
A Major Obstacle for NLP Research: Let’s Talk about Time Allocation!

The field of natural language processing (NLP) has grown over the last few years: conferences have become larger, we have published an incredible amount of papers, and state-of-the-art research has been implemented in a large variety of customer-facing products.However, this paper argues that we have been less successful than we should have been and reflects on where and how the field fails to tap its full potential.Specifically, we demonstrate that, in recent years, subpar time allocation has been a major obstacle for NLP research.We outline multiple concrete problems together with their negative consequences and, importantly, suggest remedies to improve the status quo.We hope that this paper will be a starting point for discussions around which common practices are -or are not -beneficial for NLP research.
RL with KL penalties is better viewed as Bayesian inference

Reinforcement learning (RL) is frequently employed in fine-tuning large language models (LMs) to penalize them for undesirable features of generated sequences, such as offensiveness or harmfulness.In this paper, we analyze challenges associated with treating language models as RL policies and show how avoiding those challenges requires moving beyond the RL paradigm.We start by observing that naïve maximisation of the standard RL objective unavoidably leads to distribution collapse: turning the LM into a degenerate distribution.Then, we analyze KL-regularised RL, a widely used recipe for fine-tuning LMs, which additionally constrains the fine-tuned LM to stay close to its original distribution.We show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function.Based on these observations, we sketch a Bayesian perspective that provides a firstprinciples derivation for KL-regularised RL.The Bayesian perspective also separates the modelling problem (defining a target distribution specifying the desired behaviour of an LM) and the inference problem (approximating that target distribution).Finally, it suggests that RL is not a good formal framework for thinking about fine-tuning LMs.
QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation

Large Language Models (LLMs) have shown impressive results on a variety of text understanding tasks.Search queries though pose a unique challenge, given their short-length and lack of nuance or context.Complicated feature engineering efforts do not always lead to downstream improvements as their performance benefits may be offset by increased complexity of knowledge distillation.Thus, in this paper we make the following contributions: (1) We demonstrate that Retrieval Augmentation of queries provides LLMs with valuable additional context enabling improved understanding.While Retrieval Augmentation typically increases latency of LMs (thus hurting distillation efficacy), (2) we provide a practical and effective way of distilling Retrieval Augmentation LLMs.Specifically, we use a novel two-stage distillation approach that allows us to carry over the gains of retrieval augmentation, without suffering the increased compute typically associated with it.(3) We demonstrate the benefits of the proposed approach (QUILL) on a billion-scale, real-world query understanding system resulting in huge gains.Via extensive experiments, including on public benchmarks, we believe this work offers a recipe for practical use of retrievalaugmented query understanding.
Dictionary-Assisted Supervised Contrastive Learning

Text analysis in the social sciences often involves using specialized dictionaries to reason with abstract concepts, such as perceptions about the economy or abuse on social media.These dictionaries allow researchers to impart domain knowledge and note subtle usages of words relating to a concept(s) of interest.We introduce the dictionary-assisted supervised contrastive learning (DASCL) objective, allowing researchers to leverage specialized dictionaries when fine-tuning pretrained language models.The text is first keyword simplified: a common, fixed token replaces any word in the corpus that appears in the dictionary(ies) relevant to the concept of interest.During fine-tuning, a supervised contrastive objective draws closer the embeddings of the original and keyword-simplified texts of the same class while pushing further apart the embeddings of different classes.The keyword-simplified texts of the same class are more textually similar than their original text counterparts, which additionally draws the embeddings of the same class closer together.Combining DASCL and crossentropy improves classification performance metrics in few-shot learning settings and social science applications compared to using crossentropy alone and alternative contrastive and data augmentation methods. 1
Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers

Large pre-trained language models have shown remarkable performance over the past few years.These models, however, sometimes learn superficial features from the dataset and cannot generalize to the distributions that are dissimilar to the training scenario.There have been several approaches proposed to reduce model's reliance on these bias features which can improve model robustness in the out-ofdistribution setting.However, existing methods usually use a fixed low-capacity model to deal with various bias features, which ignore the learnability of those features.In this paper, we analyze a set of existing bias features and demonstrate there is no single model that works best for all the cases.We further show that by choosing an appropriate bias model, we can obtain a better robustness result than baselines with a more sophisticated model design.
A World Without Email: Reimagining Work in an Age of Communication Overload <b>A World Without Email: Reimagining Work in an Age of Communication Overload</b> , by Cal Newport, Portfolio, 2021, 320pp., $27.00, ISBN 9780525536550

"A World Without Email: Reimagining Work in an Age of Communication Overload,." Journal of Electronic Resources in Medical Libraries, ahead-of-print(ahead-of-print), pp. 1–2
Chapter 13. Class Expression Learning with Multiple Representations

Knowledge bases are now first-class citizens of the Web. Circa 50% of the 3.2 billion websites in the 2022 crawl of Web Data Commons contains knowledge base fragments in RDF. The 82 billion assertions known to exist in these websites are complemented by a roughly comparable number of triples available in dumps. As this data is now the backbone of a number of applications, it stands to reason that machine learning approaches able to exploit the explicit semantics exposed by RDF knowledge bases must scale to large knowledge bases. In this chapter, we present approaches based on continuous and symbolic representations that aim to achieve this goal by addressing some of the main scalability bottlenecks of existing class expression learning approaches. While we focus on the description logic ALC , the approaches we present are far from being limited to this particular expressiveness.
Chapter 23. Combining Symbolic and Deep Learning Approaches for Sentiment Analysis

Deep learning approaches have become popular in sentiment analysis because of their competitive performance. The downside of this approach is that they do not provide understandable explanations on how the sentiment values are calculated. Previous approaches that used sentiment lexicons for sentiment analysis can do that, but their performance is lower than deep learning approaches. Therefore, it is natural to wonder if the two approaches can be combined to exploit their advantages. In this chapter, we present a neuro-symbolic approach that combines both symbolic and deep learning approaches for sentiment analysis tasks. The symbolic approach exploits sentiment lexicon and shifter patterns—which cover the operations of inversion/reversal, intensification, and attenuation/downtoning. The deep learning approach used a pre-trained language model (PLM) to construct sentiment lexicon. Our experimental result shows that the proposed approach leads to promising results, substantially better than the results of a pure lexicon-based approach. Although the results did not reach the level of the deep learning approach, a great advantage is that sentiment prediction can be accompanied by understandable explanations. For some users, it is very important to see how sentiment is derived, even if performance is a little lower.
Improve Writing Skills In Junior High Students Via Whatsapp Messenger

This study aims to evaluate the pros and cons of using WhatsApp Messenger to help students improve their writing abilities as well as to identify any gaps in the current literature. This study used a classroom action research methodology. To get the data, the researcher used tests, interviews, and observation. The researcher used the steps of data reduction, data display, and conclusion drafting when assessing the data. The study shows that using a scientific method helped students write recall texts more effectively. The proportion of scores for cycle I encompasses the rhetorical stages (67.50%), grammar (71.25%), vocabulary (66.25%), meaning clarity (67.50%), and relationship between concepts (63.75%). In cycle II, the percentage of scores encompasses vocabulary (76.25%), clarity of meaning (87.50%), grammar (78.75%), and link between ideas (81.25%). The students engaged in creative teaching and learning activities, showed interest in the teaching and learning processes, and openly expressed their opinions when associating and networking, according to this study. On the other hand, it also experienced difficulties with the teaching and learning procedures. The difficulties were that the majority of students lacked enthusiasm for studying WhatsApp. They must be directed to be disciplined in the classroom and encouraged to ask questions during the questioning phase.
AI-powered Resume-Job matching

This study focuses on the importance of well-designed online matching systems for job seekers and employers. We treat resumes and job descriptions as documents. Then, calculate their similarity to determine the suitability of applicants, and rank a set of resumes based on their similarity to a specific job description. We employ Siamese Neural Networks, comprised of identical sub-network components, to evaluate the semantic similarity between documents. Our novel architecture integrates various neural network architectures, where each sub-network incorporates multiple layers such as CNN, LSTM and attention layers to capture sequential, local and global patterns within the data. The LSTM and CNN components are applied concurrently and merged together. The resulting output is then fed into a multi-head attention layer. These layers extract features and capture document representations. The extracted features are then combined to form a unified representation of the document. We leverage pre-trained language models to obtain embeddings for each document, which serve as a lower-dimensional representation of our input data. The model is trained on a private dataset of 268,549 real resumes and 4,198 job descriptions from twelve industry sectors, resulting in a ranked list of matched resumes. We performed a comparative analysis involving our model, Siamese CNN (S-CNNs), Siamese LSTM with Manhattan distance, and a BERT-based sentence transformer model. By combining the power of language models and the novel Siamese architecture, this approach leverages both strengths to improve document ranking accuracy and enhance the matching process between job descriptions and resumes. Our experimental results demonstrate that our model outperforms other models in terms of performance.