Reviewerly/exHarmony huggingface dataset:
Understanding In-Context Learning via Supportive Pretraining Data

In-context learning (ICL) improves language models' performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time.It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstrations.Unlike prior work that explores implicit mechanisms behind ICL, we study ICL via investigating the pretraining data.Specifically, we first adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL.We observe that a continued pretraining on this small subset significantly improves the model's ICL ability, by up to 18%.We then compare the supportive subset constrastively with random subsets of pretraining data and discover: (1) The supportive pretraining data to ICL do not have a higher domain relevance to downstream tasks.(2) The supportive pretraining data have a higher mass of rarely occurring, long-tail tokens.(3) The supportive pretraining data are challenging examples where the information gain from long-range context is below average, indicating learning to incorporate difficult long-range context encourages ICL.Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data.Our insights have a potential to enhance the ICL ability of language models by actively guiding the construction of pretraining data in the future.* Work done during an internship at Meta AI.Pretraining data I can't think of any scenario where the Chiefs don't win that game if Charles doesn't go down.What's that?Need to chew clock with the run game?How convenient that we have an All Pro running back!While I agree that Charles going down definitely affected the outcome of the game, it's not like their back-up crapped the bed either.Knile Davis did end up with 2 TDs, so while he's not going to be mistaken for Charles, he played a great game ...
NLP-LISAC at SemEval-2023 Task 12: Sentiment Analysis for Tweets expressed in African languages via Transformer-based Models

This paper presents our systems and findings for SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages.The main objective of this task was to determine the polarity of a tweet (positive, negative, or neutral).Our submitted models (highest rank is 1 and lowest rank is 21 depending on the target Track) consist of various Transformer-based approaches.
Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework

Factuality is important to dialogue summarization.Factual error correction (FEC) of modelgenerated summaries is one way to improve factuality.Current FEC evaluation that relies on factuality metrics is not reliable and detailed enough.To address this problem, we are the first to manually annotate a FEC dataset for dialogue summarization containing 4000 items and propose FERRANTI, a fine-grained evaluation framework based on reference correction that automatically evaluates the performance of FEC models on different error categories.Using this evaluation framework, we conduct sufficient experiments with FEC approaches under a variety of settings and find the best training modes and significant differences in the performance of the existing approaches on different factual error categories. 1
BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering

Unsupervised commonsense reasoning (UCR) is becoming increasingly popular as the construction of commonsense reasoning datasets is expensive, and they are inevitably limited in their scope.A popular approach to UCR is to fine-tune language models with external knowledge (e.g., knowledge graphs), but this usually requires a large number of training examples.In this paper, we propose to transform the downstream multiple choice question answering task into a simpler binary classification task by ranking all candidate answers according to their reasonableness.To this end, for training the model, we convert the knowledge graph triples into reasonable and unreasonable texts.Extensive experimental results show the effectiveness of our approach on various multiple choice question answering benchmarks.Furthermore, compared with existing UCR approaches using KGs, ours is less data hungry.Our code is available at https://github.com/probe2/BUCAJohn wanted to be a better dancer.
Importance of Synthesizing High-quality Data for Text-to-SQL Parsing

There has been increasing interest in synthesizing data to improve downstream text-to-SQL tasks.In this paper, we examined the existing synthesized datasets and discovered that state-of-the-art text-to-SQL algorithms did not further improve on popular benchmarks when trained with augmented synthetic data.We observed three shortcomings: illogical synthetic SQL queries from independent column sampling, arbitrary table joins, and language gaps between the synthesized SQL and natural language question (NLQ) pair.To address these issues, we propose a novel synthesis framework that imposes strong typing constraints, incorporates key relationships from schema, and conducts schema-distance-weighted column sampling.We also adopt an intermediate representation (IR) for the SQL-to-text task to further improve the quality of the generated NLQ.When existing powerful text-to-SQL parsers are pretrained on our high-quality synthesized data, these models have significant accuracy boosts and achieve new state-of-the-art performance on Spider.We also demonstrate the effectiveness of our techniques with ablation studies.
Cross-Domain Argument Quality Estimation

Argumentation is one of society's foundational pillars, and, sparked by advances in NLP, and the vast availability of text data, automated mining of arguments receives increasing attention.A decisive property of arguments is their strength or quality.While there are works on the automated estimation of argument strength, their scope is narrow: they focus on isolated datasets and neglect the interactions with related argument-mining tasks, such as argument identification and evidence detection.In this work, we close this gap by approaching argument quality estimation from multiple different angles: Grounded on rich results from thorough empirical evaluations, we assess the generalization capabilities of argument quality estimation across diverse domains and the interplay with related argument mining tasks.We find that generalization depends on a sufficient representation of different domains in the training part.In zero-shot transfer and multi-task experiments, we reveal that argument quality is among the more challenging tasks but can improve others.
StoryARG: a corpus of narratives and personal experiences in argumentative texts

Humans are storytellers, even in communication scenarios which are assumed to be more rationality-oriented, such as argumentation.Indeed, supporting arguments with narratives or personal experiences (henceforth, stories) is a very natural thing to do -and yet, this phenomenon is largely unexplored in computational argumentation.Which role do stories play in an argument?Do they make the argument more effective?What are their narrative properties?To address these questions, we collected and annotated StoryARG, a dataset sampled from well-established corpora in computational argumentation (ChangeMyView and RegulationRoom), and the Social Sciences (Europolis), as well as comments to New York Times articles.StoryARG contains 2451 textual spans annotated at two levels.At the argumentative level, we annotate the function of the story (e.g., clarification, disclosure of harm, search for a solution, establishing speaker's authority), as well as its impact on the effectiveness of the argument and its emotional load.At the level of narrative properties, we annotate whether the story has a plot-like development, is factual or hypothetical, and who the protagonist is.What makes a story effective in an argument?Our analysis of the annotations in StoryARG uncover a positive impact on effectiveness for stories which illustrate a solution to a problem, and in general, annotator-specific preferences that we investigate with regression analysis.
Towards Open Environment Intent Prediction

Out-of-Domain (OOD) Intent Classification and New Intent Discovery are as two basic and critical tasks in the Task-Oriented Dialogue System, which are typically treated as two independent tasks.Classification focuses on identifying intents beyond the predefined set of the dialog system, but it will not further differentiate detected OOD intents in fine granularity.Discovery focuses on how to cluster unlabeled samples according to their semantic representation, which relies heavily on prior knowledge and can not provide label information for the formed clusters.To be closer to the real userfacing scenarios, we strengthen a combined generative task paradigm to extend Classification with Discovery referred to as Open Environment Intent Prediction, which is to make a further fine-grained discovery of OOD based on OOD Intent Classification.Using various widely-used generative models as an archetype, we propose a general scheme for Open Environment Intent Prediction.In a nutshell, we first perform intent detection to identify the Indomain (IND) samples and then generate labels for those identified as OOD.With these generated labels, we can discover new general intents and provide label information for them.We develop a suite of benchmarks on the existing intent datasets and present a simple yet effective implementation.Extensive experiments demonstrate that our method establishes substantial improvement compared to the baselines.Codes is publicly available.
Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models

Pre-trained language models (PLMs) are known to be overly parameterized and have significant redundancy, indicating a small degree of freedom of the PLMs.Motivated by the observation, in this paper, we study the problem of re-parameterizing and fine-tuning PLMs from a new perspective: Discovery of intrinsic task-specific subspace.Specifically, by exploiting the dynamics of the fine-tuning process for a given task, the parameter optimization trajectory is learned to uncover its intrinsic task-specific subspace.A key finding is that PLMs can be effectively fine-tuned in the subspace with a small number of free parameters.Beyond, we observe some outlier dimensions emerging during fine-tuning in the subspace.Disabling these dimensions degrades the model performance significantly.This suggests that these dimensions are crucial to induce task-specific knowledge to downstream tasks.
Howard University Computer Science at SemEval-2023 Task 12: A 2-Step System Design for Multilingual Sentiment Classification with Language Identification

The recent release of the AfriSenti-SemEval shared Task 12 has made available 14 new datasets annotated for sentiment analysis on African Languages.We proposed and evaluated two approaches to this task, Delta TF-IDF, and a proposed Language-Specific Model Fusion System using Language Identification.Both produced comparable or better classification performance than the current state-of-art models on this task: AfriBERTa, AfroXLMR, and AfroLM.
The ADAIO System at the BEA-2023 Shared Task: Shared Task Generating AI Teacher Responses in Educational Dialogues

This paper presents the ADAIO team's system entry in the Building Educational Applications (BEA) 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues.The task aims to assess the performance of state-of-the-art generative models as AI teachers in producing suitable responses within a student-teacher dialogue.Our system comprises evaluating various baseline models using OpenAI GPT-3 and designing diverse prompts to prompt the OpenAI models for teacher response generation.After the challenge, our system achieved second place by employing a few-shot prompt-based approach with the Ope-nAI text-davinci-003 model.The results highlight the few-shot learning capabilities of largelanguage models, particularly OpenAI's GPT-3, in the role of AI teachers.
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering

Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.Specifically, most of the wellperformed metrics are required to train on evaluation datasets of specific NLG tasks and evaluation dimensions, which may cause over-fitting to task-specific datasets.Furthermore, existing metrics only provide an evaluation score for each dimension without revealing the evidence to interpret how this score is obtained.To deal with these challenges, we propose a simple yet effective metric called DecompEval.This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models (PLMs) without training on evaluation datasets, aiming to enhance the generalization ability.To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence.The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result.Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability 1 .
Explanation Graph Generation via Generative Pre-training over Synthetic Graphs

The generation of explanation graphs is a significant task that aims to produce explanation graphs in response to user input, revealing the internal reasoning process.This task is challenging due to the significant discrepancy between unstructured user queries and structured explanation graphs.Current research commonly fine-tunes a text-based pre-trained language model on a small downstream dataset that is annotated with labeled graphs.However, due to the limited scale of available datasets, this approach may prove to be insufficient in bridging the gap between natural language text and structured graphs.In this paper, to alleviate the above limitations, we propose a novel pre-trained framework EG 3 P(for Explanation Graph Generation via Generative Pre-training over synthetic graphs) for the explanation graph generation task.Specifically, we first propose a text-to-graph generative task to pre-train the model with the goal of bridging the text-graph gap.Additionally, we propose an automatic corpus synthesis strategy for synthesizing a large scale of high-quality corpus, reducing the reliance on costly manual annotation methods.Experimental results on ExplaGraphs show the effectiveness of EG 3 P that our model surpasses all baseline systems with remarkable margins.Besides, further analysis demonstrates that EG 3 P is able to generate better explanation graphs on actual reasoning tasks such as CommonsenseQA and OpenbookQA. 1
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization

A series of datasets and models have been proposed for summaries generated for wellformatted documents such as news articles.Dialogue summaries, however, have been under explored.In this paper, we present the first dataset with fine-grained factual error annotations named DIASUMFACT.We define finegrained factual error detection as a sentencelevel multi-label classification problem, and we evaluate two state-of-the-art (SOTA) models on our dataset.Both models yield sub-optimal results, with a macro-averaged F1 score of around 0.25 over 6 error classes.We further propose an unsupervised model ENDERANKER via candidate ranking using pretrained encoder-decoder models.Our model performs on par with the SOTA models while requiring fewer resources.These observations confirm the challenges in detecting factual errors from dialogue summaries, which call for further studies, for which our dataset and results offer a solid foundation. 1Lilly: Wanna go out tonight?
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation

Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent.In this paper, we introduce the MMDialog dataset to facilitate multi-modal conversation better.MMDialog is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics.MMDialog has two main and unique advantages.First, it is the largest multi-modal conversation dataset by the number of dialogues by 88x.Second, it contains massive topics to generalize the open domain.To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios.In addition, we build two baselines for the above tasks with state-of-theart techniques and report their experimental performance.We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses.
PMI-Align: Word Alignment With Point-Wise Mutual Information Without Requiring Parallel Training Data

Word alignment has many applications including cross-lingual annotation projection, bilingual lexicon extraction, and the evaluation or analysis of translation outputs.Recent studies show that using contextualized embeddings from pre-trained multilingual language models could give us high quality word alignments without the need of parallel training data.In this work, we propose PMI-Align which computes and uses the point-wise mutual information between source and target tokens to extract word alignments, instead of the cosine similarity or dot product which is mostly used in recent approaches.Our experiments show that our proposed PMI-Align approach could outperform the rival methods on five out of six language pairs.Although our approach requires no parallel training data, we show that this method could also benefit the approaches using parallel data to fine-tune pre-trained language models on word alignments.Our code and data are publicly available 1 .
An (unhelpful) guide to selecting the best ASR architecture for your under-resourced language

Advances in deep neural models for automatic speech recognition (ASR) have yielded dramatic improvements in ASR quality for resource-rich languages, with English ASR now achieving word error rates comparable to that of human transcribers.The vast majority of the world's languages, however, lack the quantity of data necessary to approach this level of accuracy.In this paper we use four of the most popular ASR toolkits to train ASR models for eleven languages with limited ASR training resources: eleven widely spoken languages of Africa, Asia, and South America, one endangered language of Central America, and three critically endangered languages of North America.We find that no single architecture consistently outperforms any other.These differences in performance so far do not appear to be related to any particular feature of the datasets or characteristics of the languages.These findings have important implications for future research in ASR for under-resourced languages.ASR systems for languages with abundant existing media and available speakers may derive the most benefit simply by collecting large amounts of additional acoustic and textual training data.Communities using ASR to support endangered language documentation efforts, who cannot easily collect more data, might instead focus on exploring multiple architectures and hyperparameterizations to optimize performance within the constraints of their available data and resources.
Complementary Explanations for Effective In-Context Learning

Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective.This work aims to better understand the mechanisms by which explanations are used for in-context learning.We first study the impact of two different factors on the performance of prompts with explanations: the computation trace (the way the solution is decomposed) and the natural language used to express the prompt.By perturbing explanations on three controlled tasks, we show that both factors contribute to the effectiveness of explanations.We further study how to form maximally effective sets of explanations for solving a given test query.We find that LLMs can benefit from the complementarity of the explanation set: diverse reasoning skills shown by different exemplars can lead to better performance.Therefore, we propose a maximal marginal relevance-based exemplar selection approach for constructing exemplar sets that are both relevant as well as complementary, which successfully improves the incontext learning performance across three realworld tasks on multiple LLMs.* Work done during an internship at Meta AI.
Continual Contrastive Finetuning Improves Low-Resource Relation Extraction

Relation extraction (RE), which has relied on structurally annotated corpora for model training, has been particularly challenging in lowresource scenarios and domains.Recent literature has tackled low-resource RE by selfsupervised learning, where the solution involves pretraining the entity pair embedding by RE-based objective and finetuning on labeled data by classification-based objective.However, a critical challenge to this approach is the gap in objectives, which prevents the RE model from fully utilizing the knowledge in pretrained representations.In this paper, we aim at bridging the gap and propose to pretrain and finetune the RE model using consistent objectives of contrastive learning.Since in this kind of representation learning paradigm, one relation may easily form multiple clusters in the representation space, we further propose a multi-center contrastive loss that allows one relation to form multiple clusters to better align with pretraining.Experiments on two document-level RE datasets, BioRED and Re-DocRED, demonstrate the effectiveness of our method.Particularly, when using 1% end-task training data, our method outperforms PLMbased RE classifier by 10.5% and 6.1% on the two datasets, respectively.
Together We Make Sense–Learning Meta-Sense Embeddings

Sense embedding learning methods learn multiple vectors for a given ambiguous word, corresponding to its different word senses.For this purpose, different methods have been proposed in prior work on sense embedding learning that use different sense inventories, sensetagged corpora and learning methods.However, not all existing sense embeddings cover all senses of ambiguous words equally well due to the discrepancies in their training resources.To address this problem, we propose the first-ever meta-sense embedding method -Neighbour Preserving Meta-Sense Embeddings, which learns meta-sense embeddings by combining multiple independently trained source sense embeddings such that the sense neighbourhoods computed from the source embeddings are preserved in the meta-embedding space.Our proposed method can combine source sense embeddings that cover different sets of word senses.Experimental results on Word Sense Disambiguation (WSD) and Word-in-Context (WiC) tasks show that the proposed meta-sense embedding method consistently outperforms several competitive baselines.