Reviewerly/exHarmony huggingface dataset:
Bidirectional Generative Framework for Cross-domain Aspect-based Sentiment Analysis

Cross-domain aspect-based sentiment analysis (ABSA) aims to perform various fine-grained sentiment analysis tasks on a target domain by transferring knowledge from a source domain.Since labeled data only exists in the source domain, a model is expected to bridge the domain gap for tackling cross-domain ABSA.Though domain adaptation methods have proven to be effective, most of them are based on a discriminative model, which needs to be specifically designed for different ABSA tasks.To offer a more general solution, we propose a unified bidirectional generative framework to tackle various cross-domain ABSA tasks.Specifically, our framework trains a generative model in both text-to-label and label-to-text directions.The former transforms each task into a unified format to learn domain-agnostic features, and the latter generates natural sentences from noisy labels for data augmentation, with which a more accurate model can be trained.To investigate the effectiveness and generality of our framework, we conduct extensive experiments on four cross-domain ABSA tasks and present new state-of-the-art results on all tasks.
Learning Joint Structural and Temporal Contextualized Knowledge Embeddings for Temporal Knowledge Graph Completion

Temporal knowledge graph completion that predicts missing links for incomplete temporal knowledge graphs (TKG) is gaining increasing attention.Most existing works have achieved good results by incorporating time information into static knowledge graph embedding methods.However, they ignore the contextual nature of the TKG structure, i.e., queryspecific subgraph contains both structural and temporal neighboring facts.This paper presents the SToKE, a novel method that employs the pre-trained language model (PLM) to learn joint Structural and Temporal Contextualized Knowledge Embeddings.Specifically, we first construct an event evolution tree (EET) for each query to enable PLMs to handle the TKG, which can be seen as a structured event sequence recording query-relevant structural and temporal contexts.We then propose a novel temporal embedding and structural matrix to learn the time information and structural dependencies of facts in EET.Finally, we formulate TKG completion as a mask prediction problem by masking the missing entity of the query to fine-tune pre-trained language models.Experimental results on three widely used datasets show the superiority of our model.
Combo of Thinking and Observing for Outside-Knowledge VQA

Outside-knowledge visual question answering is a challenging task that requires both the acquisition and the use of open-ended real-world knowledge.Some existing solutions draw external knowledge into the cross-modality space which overlooks the much vaster textual knowledge in natural-language space, while others transform the image into a text that further fuses with the textual knowledge into the natural-language space and completely abandons the use of visual features.In this paper, we are inspired to constrain the cross-modality space into the same space of natural-language space which makes the visual features preserved directly, and the model still benefits from the vast knowledge in natural-language space.To this end, we propose a novel framework consisting of a multimodal encoder, a textual encoder and an answer decoder.Such structure allows us to introduce more types of knowledge including explicit and implicit multimodal and textual knowledge.Extensive experiments validate the superiority of the proposed method which outperforms the state-ofthe-art by 6.17% accuracy.We also conduct comprehensive ablations of each component, and systematically study the roles of varying types of knowledge.Codes and knowledge data can be found at https://github.com/PhoebusSi/Thinking-while-Observing.
Knowledgeable Parameter Efficient Tuning Network for Commonsense Question Answering

Commonsense question answering is important for making decisions about everyday matters.Although existing commonsense question answering works based on fully fine-tuned PLMs have achieved promising results, they suffer from prohibitive computation costs as well as poor interpretability.Some works improve the PLMs by incorporating knowledge to provide certain evidence, via elaborately designed GNN modules which require expertise.In this paper, we propose a simple knowledgeable parameter efficient tuning network to couple PLMs with external knowledge for commonsense question answering.Specifically, we design a trainable parameter-sharing adapter attached to a parameter-freezing PLM to incorporate knowledge at a small cost.The adapter is equipped with both entity-and query-related knowledge via two auxiliary knowledge-related tasks (i.e., span masking and relation discrimination).To make the adapter focus on the relevant knowledge, we design gating and attention mechanisms to respectively filter and fuse the query information from the PLM.Extensive experiments on two benchmark datasets show that KPE is parameter-efficient and can effectively incorporate knowledge for improving commonsense question answering.
Japanese Lexical Complexity for Non-Native Readers: A New Dataset

Lexical complexity prediction (LCP) is the task of predicting the complexity of words in a text on a continuous scale.It plays a vital role in simplifying or annotating complex words to assist readers.To study lexical complexity in Japanese, we construct the first Japanese LCP dataset.Our dataset provides separate complexity scores for Chinese/Korean annotators and others to address the readers' L1-specific needs.In the baseline experiment, we demonstrate the effectiveness of a BERT-based system for Japanese LCP.
LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming

Open-domain dialogue systems have made promising progress in recent years.While the state-of-the-art dialogue agents are built upon large-scale text-based social media data and large pre-trained models, there is no guarantee these agents could also perform well in fast-growing scenarios, such as live streaming, due to the bounded transferability of pretrained models and biased distributions of public datasets from Reddit and Weibo, etc.To improve the essential capability of responding and establish a benchmark in the live opendomain scenario, we introduce the LiveChat dataset, composed of 1.33 million real-life Chinese dialogues with almost 3800 average sessions across 351 personas and fine-grained profiles for each persona.LiveChat is automatically constructed by processing numerous live videos on the Internet and naturally falls within the scope of multi-party conversations, where the issues of Who says What to Whom should be considered.Therefore, we target two critical tasks of response modeling and addressee recognition and propose retrieval-based baselines grounded on advanced techniques.Experimental results have validated the positive effects of leveraging persona profiles and larger average sessions per persona.In addition, we also benchmark the transferability of advanced generation-based models on LiveChat and pose some future directions for current challenges.
ExASAG: Explainable Framework for Automatic Short Answer Grading

As in other NLP tasks, Automatic Short Answer Grading (ASAG) systems have evolved from using rule-based and interpretable machine learning models to utilizing deep learning architectures to boost accuracy.Since proper feedback is critical to student assessment, explainability will be crucial for deploying ASAG in real-world applications.This paper proposes a framework to generate explainable outcomes for assessing question-answer pairs of a Data Mining course in a binary manner.Our framework utilizes a fine-tuned Transformer-based classifier and an explainability module using SHAP or Integrated Gradients to generate language explanations for each prediction.We assess the outcome of our framework by calculating accuracy-based metrics for classification performance.Furthermore, we evaluate the quality of the explanations by measuring their agreement with human-annotated justifications using Intersection-Over-Union at a token level to derive a plausibility score.Despite the relatively limited sample, results show that our framework derives explanations that are, to some degree, aligned with domain-expert judgment.Furthermore, both explainability methods perform similarly in their agreement with human-annotated explanations.A natural progression of our work is to analyze the use of our explainable ASAG framework on a larger sample to determine the feasibility of implementing a pilot study in a real-world setting.
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations

Evaluating multi-document summarization (MDS) quality is difficult.This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across different documents.Prior work has shown that rather than performing the task, models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE.Better automated evaluation metrics are needed, but few resources exist to assess metrics when they are proposed.Therefore, we introduce a dataset of human-assessed summary quality facets and pairwise preferences to encourage and support the development of better automated evaluation methods for literature review MDS.We take advantage of community submissions to the Multi-document Summarization for Literature Review (MSLR) shared task to compile a diverse and representative sample of generated summaries.We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality.We find that not only do automated metrics fail to capture aspects of quality as assessed by humans, in many cases the system rankings produced by these metrics are anti-correlated with rankings according to human annotators. 1
End-to-end clinical temporal information extraction with multi-head attention

Understanding temporal relationships in text from electronic health records can be valuable for many important downstream clinical applications. Since Clinical TempEval 2017, there has been little work on end-to-end systems for temporal relation extraction, with most work focused on the setting where gold standard events and time expressions are given. In this work, we make use of a novel multi-headed attention mechanism on top of a pre-trained transformer encoder to allow the learning process to attend to multiple aspects of the contextualized embeddings. Our system achieves state of the art results on the THYME corpus by a wide margin, in both the in-domain and cross-domain settings.
Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?

The CoNLL-2003 English named entity recognition (NER) dataset has been widely used to train and evaluate NER models for almost 20 years.However, it is unclear how well models that are trained on this 20-year-old data and developed over a period of decades using the same test set will perform when applied on modern data.In this paper, we evaluate the generalization of over 20 different models trained on CoNLL-2003, and show that NER models have very different generalization.Surprisingly, we find no evidence of performance degradation in pre-trained Transformers, such as RoBERTa and T5, even when fine-tuned using decades-old data.We investigate why some models generalize well to new data while others do not, and attempt to disentangle the effects of temporal drift and overfitting due to test reuse.Our analysis suggests that most deterioration is due to temporal mismatch between the pre-training corpora and the downstream test sets.We found that four factors are important for good generalization: model architecture, number of parameters, time period of the pre-training corpus, in addition to the amount of fine-tuning data.We suggest current evaluation methods have, in some sense, underestimated progress on NER over the past 20 years, as NER models have not only improved on the original CoNLL-2003 test set, but improved even more on modern data.Our datasets can be found at https:// github.com/ShuhengL/acl2023_conllpp.
Learning to Generate Equitable Text in Dialogue from Biased Training Data

The ingrained principles of fairness in a dialogue system's decision-making process and generated responses are crucial for user engagement, satisfaction, and task achievement.Absence of equitable and inclusive principles can hinder the formation of common ground, which in turn negatively impacts the overall performance of the system.For example, misusing pronouns in a user interaction may cause ambiguity about the intended subject.Yet, there is no comprehensive study of equitable text generation in dialogue.Aptly, in this work, we use theories of computational learning to study this problem.We provide formal definitions of equity in text generation, and further, prove formal connections between learning humanlikeness and learning equity: algorithms for improving equity ultimately reduce to algorithms for improving human-likeness (on augmented data).With this insight, we also formulate reasonable conditions under which text generation algorithms can learn to generate equitable text without any modifications to the biased training data on which they learn.To exemplify our theory in practice, we look at a group of algorithms for the GuessWhat?! visual dialogue game and, using this example, test our theory empirically.Our theory accurately predicts relative-performance of multiple algorithms in generating equitable text as measured by both human and automated evaluation.
Exploiting Rich Textual User-Product Context for Improving Personalized Sentiment Analysis

User and product information associated with a review is useful for sentiment polarity prediction.Typical approaches incorporating such information focus on modeling users and products as implicitly learned representation vectors.Most do not exploit the potential of historical reviews, or those that currently do require unnecessary modifications to model architecture or do not make full use of user/product associations.The contribution of this work is twofold: i) a method to explicitly employ historical reviews belonging to the same user/product in initializing representations, and ii) efficient incorporation of textual associations between users and products via a user-product cross-context module.Experiments on the IMDb, Yelp-2013 and Yelp-2014 English benchmarks with BERT, SpanBERT and Longformer pretrained language models show that our approach substantially outperforms previous state-of-the-art.
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks

Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition.In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape.We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance.In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will release a new benchmark suite, including for each task (i) curated annotations for a relatively small fine-tuning set, (ii) reproducible pipeline (speech recognizer + text model) and end-to-end baseline models and evaluation metrics, (iii) baseline model performance in various types of systems for easy comparisons.We present the details of data collection and annotation and the performance of the baseline models.We also analyze the sensitivity of pipeline models' performance to the speech recognition accuracy, using more than 20 publicly available speech recognition models.
Cross-Domain Data Augmentation with Domain-Adaptive Language Modeling for Aspect-Based Sentiment Analysis

Cross-domain Aspect-Based Sentiment Analysis (ABSA) aims to leverage the useful knowledge from a source domain to identify aspectsentiment pairs in sentences from a target domain.To tackle the task, several recent works explore a new unsupervised domain adaptation framework, i.e., Cross-Domain Data Augmentation (CDDA), aiming to directly generate much labeled target-domain data based on the labeled source-domain data.However, these CDDA methods still suffer from several issues: 1) preserving many source-specific attributes such as syntactic structures; 2) lack of fluency and coherence; 3) limiting the diversity of generated data.To address these issues, we propose a new cross-domain Data Augmentation approach based on Domain-Adaptive Language Modeling named DA 2 LM, which contains three stages: 1) assigning pseudo labels to unlabeled target-domain data; 2) unifying the process of token generation and labeling with a Domain-Adaptive Language Model (DALM) to learn the shared context and annotation across domains; 3) using the trained DALM to generate labeled target-domain data.Experiments show that DA 2 LM consistently outperforms previous feature adaptation and CDDA methods on both ABSA and Aspect Extraction tasks.
Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue

Factual correctness is often the limiting factor in practical applications of natural language generation in high-stakes domains such as healthcare.An essential requirement for maintaining factuality is the ability to deal with rare tokens.This paper focuses on rare tokens that appear in both the source and the reference sequences, and which, when missed during generation, decrease the factual correctness of the output text.For high-stake domains that are also knowledge-rich, we show how to use knowledge to (a) identify which rare tokens that appear in both source and reference are important and (b) uplift their conditional probability.We introduce the "utilization rate" that encodes knowledge and serves as a regularizer by maximizing the marginal probability of selected tokens.We present a study in a knowledgerich domain of healthcare, where we tackle the problem of generating after-visit care instructions based on patient-doctor dialogues.We verify that, in our dataset, specific medical concepts with high utilization rates are underestimated by conventionally trained sequence-tosequence models.We observe that correcting this with our approach to knowledge injection reduces the uncertainty of the model as well as improves factuality and coherence without negatively impacting fluency.
Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations

Natural language video localization(NLVL) task involves the semantic matching of a text query with a moment from an untrimmed video.Previous methods primarily focus on improving performance with the assumption of independently identical data distribution while ignoring the out-of-distribution data.Therefore, these approaches often fail when handling the videos and queries in novel scenes, which is inevitable in real-world scenarios.In this paper, we, for the first time, formulate the scene-robust NLVL problem and propose a novel generalizable NLVL framework utilizing data in multiple available scenes to learn a robust model.Specifically, our model learns a group of generalizable domain-invariant representations by alignment and decomposition.First, we propose a comprehensive intra-and inter-sample distance metric for complex multimodal feature space, and an asymmetric multimodal alignment loss for different information densities of text and vision.Further, to alleviate the conflict between domain-invariant features for generalization and domain-specific information for reasoning, we introduce domainspecific and domain-agnostic predictors to decompose and refine the learned features by dynamically adjusting the weights of samples.Based on the original video tags, we conduct extensive experiments on three NLVL datasets with different-grained scene shifts to show the effectiveness of our proposed methods.
SERENGETI: Massively Multilingual Language Models for Africa

Multilingual pretrained language models (mPLMs) acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning.To date, only ∼ 31 out of ∼ 2, 000 African languages are covered in existing language models.We ameliorate this limitation by developing SERENGETI, a massively multilingual language model that covers 517 African languages and language varieties.We evaluate our novel models on eight natural language understanding tasks across 20 datasets, comparing to 4 mPLMs that cover 4 -23 African languages.SERENGETI outperforms other models on 11 datasets across the eights tasks, achieving 82.27 average F 1 .We also perform analyses of errors from our models, which allows us to investigate the influence of language genealogy and linguistic similarity when the models are applied under zero-shot settings.We will publicly release our models for research.
Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning

Massively multilingual pre-trained language models (MMPLMs) are developed in recent years demonstrating superpowers and the pre-knowledge they acquire for downstream tasks.This work investigates whether MMPLMs can be applied to clinical domain machine translation (MT) towards entirely unseen languages via transfer learning.We carry out an experimental investigation using Meta-AI's MMPLMs "wmt21-dense-24-wide-en-X and X-en (WMT21fb)" which were pre-trained on 7 language pairs and 14 translation directions including English to Czech, German, Hausa, Icelandic, Japanese, Russian, and Chinese, and the opposite direction.We fine-tune these MMPLMs towards English-Spanish language pair which did not exist at all in their original pre-trained corpora both implicitly and explicitly.We prepare carefully aligned clinical domain data for this fine-tuning, which is different from their original mixed domain knowledge.Our experimental result shows that the fine-tuning is very successful using just 250k well-aligned in-domain EN-ES segments for three sub-task translation testings: clinical cases, clinical terms, and ontology concepts.It achieves very close evaluation scores to another MMPLM NLLB from Meta-AI, which included Spanish as a high-resource setting in the pre-training.To the best of our knowledge, this is the first work on using MMPLMs towards clinical domain transferlearning NMT successfully for totally unseen languages during pre-training.
INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation

Neural machine translation has achieved promising results on many translation tasks.However, previous studies have shown that neural models induce a non-smooth representation space, which harms its generalization results.Recently, kNN-MT has provided an effective paradigm to smooth the prediction based on neighbor representations during inference.Despite promising results, kNN-MT usually requires large inference overhead.We propose an effective training framework INK to directly smooth the representation space via adjusting representations of kNN neighbors with a small number of new parameters.The new parameters are then used to refresh the whole representation datastore to get new kNN knowledge asynchronously.This loop keeps running until convergence.Experiments on four benchmark datasets show that INK achieves average gains of 1.99 COMET and 1.0 BLEU, outperforming the state-of-the-art kNN-MT system with 0.02× memory space and 1.9× inference speedup 1 .