Reviewerly/exHarmony huggingface dataset:
Investigating the Benefits of Free-Form Rationales

Free-form rationales aim to aid model interpretability by supplying background knowledge that can help understand model decisions.Popular commonsense QA datasets such as CoS-E and ECQA provide crowdsourced freeform rationales for instances, but their utility remains under-investigated.We present studies which show that 88% of ECQA rationales indeed provide humans additional background information to understand a decision, while 93% of CoS-E rationales do not.Inspired by this finding, we ask: can the additional context provided by free-form rationales benefit models, similar to their effect on human users?We investigate the usefulness of rationales as an additional training signal, by varying the quantity and quality of rationales during training.After controlling for instances where rationales leak the correct answer while not providing additional background knowledge, we find that incorporating only 5% of rationales during training can boost model performance by 47.22% for CoS-E and 57.14% for ECQA during inference.Moreover, we also show that rationale quality matters: compared to crowdsourced rationales, T5-generated rationales provide not only a weaker training signal, but are also not helpful for humans in aiding model interpretability.* Work done prior to JM joining Amazon. 1 We use the terms "rationale" and "explanation" interchangeably.Please see Wiegreffe
Improved Evaluation of Automatic Source Code Summarisation

Source code summarisation is a vital tool for the understanding and maintenance of source code as summarisations can be used to explain code in simple terms.However, source code with missing, incorrect, or outdated summaries is a common occurrence in production code.Automatic source code summarisation seeks to solve these issues by generating up-to-date summaries of source code methods.Recent work in automatically generating source code summaries uses neural networks for generating summaries; commonly Sequence-to-Sequence or Transformer models, pretrained on methodsummary pairs.The most common method of evaluating the quality of these summaries is comparing the machine-generated summaries against human-written summaries.Summaries can be evaluated using n-gram-based translation metrics such as BLEU, METEOR, or ROUGE-L.However, these metrics alone can be unreliable and new Natural Language Generation metrics based on large pretrained language models provide an alternative.In this paper, we propose a method of improving the evaluation of a model by improving the preprocessing of the data used to train it, as well as proposing evaluating the model with a metric based off a language model, pretrained on a Natural Language (English) alongside traditional metrics.Our evaluation suggests our model has been improved by cleaning and preprocessing the data used in model training.The addition of a pretrained language model metric alongside traditional metrics shows that both produce results which can be used to evaluate neural source code summarisation.
Utilizing Weak Supervision to Create S3D: A Sarcasm Annotated Dataset

Sarcasm is prevalent in all corners of social media, posing many challenges within Natural Language Processing (NLP), particularly for sentiment analysis.Sarcasm detection remains a largely unsolved problem in many NLP tasks due to its contradictory and typically derogatory nature as a figurative language construct.With recent strides in NLP, many pre-trained language models exist that have been trained on data from specific social media platforms, i.e., Twitter.In this paper, we evaluate the efficacy of multiple sarcasm detection datasets using machine and deep learning models.We create two new datasets -a manually annotated gold standard Sarcasm Annotated Dataset (SAD) and a Silver-Standard Sarcasm-annotated Dataset (S3D).Using a combination of existing sarcasm datasets with SAD, we train a sarcasm detection model over a social-media domain pre-trained language model, BERTweet, which yields an F1-score of 78.29%.Using an Ensemble model with an underlying majority technique, we further label S3D to produce a weakly supervised dataset containing over 100, 000 tweets.We publicly release all the code, our manually annotated and weakly supervised datasets, and fine-tuned models for further research.
Tracing and Manipulating intermediate values in Neural Math Problem Solvers

How language models process complex input that requires multiple steps of inference is not well understood.Previous research has shown that information about intermediate values of these inputs can be extracted from the activations of the models, but it is unclear where that information is encoded and whether that information is indeed used during inference.We introduce a method for analyzing how a Transformer model processes these inputs by focusing on simple arithmetic problems and their intermediate values.To trace where information about intermediate values is encoded, we measure the correlation between intermediate values and the activations of the model using principal component analysis (PCA).Then, we perform a causal intervention by manipulating model weights.This intervention shows that the weights identified via tracing are not merely correlated with intermediate values, but causally related to model predictions.Our findings show that the model has a locality to certain intermediate values, and this is useful for enhancing the interpretability of the models.
Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models

A large body of recent work highlights the fallacies of zero-shot cross-lingual transfer (ZS-XLT) with large multilingual language models.Namely, their performance varies substantially for different target languages and is the weakest where needed the most: for low-resource languages distant to the source language.One remedy is few-shot transfer (FS-XLT), where leveraging only a few task-annotated instances in the target language(s) may yield sizable performance gains.However, FS-XLT also succumbs to large variation, as models easily overfit to the small datasets.In this work, we present a systematic study focused on a spectrum of FS-XLT fine-tuning regimes, analyzing key properties such as effectiveness, (in)stability, and modularity.We conduct extensive experiments on both higher-level (NLI, paraphrasing) and lowerlevel tasks (NER, POS), presenting new FS-XLT strategies that yield both improved and more stable FS-XLT across the board.Our findings challenge established FS-XLT methods: e.g., we propose to replace sequential fine-tuning with joint fine-tuning on source and target language instances, offering consistent gains with different number of shots (including resourcerich scenarios).We also show that further gains can be achieved with multi-stage FS-XLT training in which joint multilingual fine-tuning precedes the bilingual source-target specialization.
DORE: Document Ordered Relation Extraction based on Generative Framework

In recent years, there is a surge of generationbased information extraction work, which allows a more direct use of pre-trained language models and efficiently captures output dependencies.However, previous generative methods using lexical representation do not naturally fit document-level relation extraction (DocRE) where there are multiple entities and relational facts.In this paper, we investigate the root cause of the underwhelming performance of the existing generative DocRE models and discover that the culprit is the inadequacy of the training paradigm, instead of the capacities of the models.We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn.Moreover, we design a parallel row generation method to process overlong target sequences.Besides, we introduce several negative sampling strategies to improve the performance with balanced signals.Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.We have released our code at
AdaK-NER: An Adaptive Top-K Approach for Named Entity Recognition with Incomplete Annotations

State-of-the-art Named Entity Recognition (NER) models rely heavily on large amounts of fully annotated training data.However, accessible data are often incompletely annotated since the annotators usually lack comprehensive knowledge in the target domain.Normally the unannotated tokens are regarded as non-entities by default, while we underline that these tokens could either be non-entities or part of any entity.Here, we study NER modeling with incomplete annotated data where only a fraction of the named entities are labeled, and the unlabeled tokens are equivalently multi-labeled by every possible label.Taking multi-labeled tokens into account, the numerous possible paths can distract the training model from the gold path (ground truth label sequence), and thus hinders the learning ability.In this paper, we propose AdaK-NER, named the adaptive top-K approach, to help the model focus on a smaller feasible region where the gold path is more likely to be located.We demonstrate the superiority of our approach through extensive experiments on both English and Chinese datasets, averagely improving 2% in F-score on the CoNLL-2003 and over 10% on two Chinese datasets compared with the prior state-of-the-art works.
Answerability: A custom metric for evaluating chatbot performance

Most commercial conversational AI products in domains spanning e-commerce, health care, finance, and education involve a hierarchy of NLP models that perform a variety of tasks such as classification, entity recognition, question-answering, sentiment detection, semantic text similarity, and so on.Despite our understanding of each of the constituent models, we often do not have a clear view as to how these models affect the overall platform metrics.To bridge this gap, we define a metric known as answerability, which penalizes not only irrelevant or incorrect chatbot responses but also unhelpful responses that do not serve the chatbot's purpose despite being correct or relevant.Additionally, we describe a formula-based mathematical framework to relate individual model metrics to the answerability metric.We also describe a modeling approach for predicting a chatbot's answerability to a user question and its corresponding chatbot response.
Adversarial Perturbations Augmented Language Models for Euphemism Identification

Euphemisms are mild words or expressions used instead of harsh or direct words while talking to someone to avoid discussing something unpleasant, embarrassing, or offensive.However, they are often ambiguous, thus making it a challenging task.The Third Workshop on Figurative Language Processing colocated with EMNLP 2022 organized a shared task on Euphemism Detection to better understand euphemisms.We have used the adversarial augmentation technique to construct new data.This augmented data was then trained using two language models, namely, BERT and Longformer.To further enhance the overall performance, various combinations of the results obtained using Longformer and BERT were passed through a voting ensembler.We were able to achieve an F1 score of 71.5 using the combination of two adversarial Longformers, two ad-versarial BERT, 1 non adversarial BERT.
Automatic Document Selection for Efficient Encoder Pretraining

Building pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance?We propose an alternative to larger training sets by automatically identifying smaller yet domain-representative subsets.We extend Cynical Data Selection, a statistical sentence scoring method that conditions on a representative target domain corpus.As an example, we treat the OntoNotes corpus as a target domain and pretrain a RoBERTa-like encoder from a cynically selected subset of the Pile.On both perplexity and across several downstream tasks in the target domain, it consistently outperforms random selection with 20x less data, 3x fewer training iterations, and 2x less estimated cloud compute cost, validating the recipe of automatic document selection for LM pretraining.
Look Ma, Only 400 Samples! Revisiting the Effectiveness of Automatic N-Gram Rule Generation for Spelling Normalization in Filipino

With 84.75 million Filipinos online, the ability for models to process online text is crucial for developing Filipino NLP applications.To this end, spelling correction is a crucial preprocessing step for downstream processing.However, the lack of data prevents the use of language models for this task.In this paper, we propose an N-Gram + Damerau-Levenshtein distance model with automatic rule extraction.We train the model on 300 samples, and show that despite limited training data, it achieves good performance and outperforms other deep learning approaches in terms of accuracy and edit distance.Moreover, the model (1) requires little compute power, (2) trains in little time, thus allowing for retraining, and (3) is easily interpretable, allowing for direct troubleshooting, highlighting the success of traditional approaches over more complex deep learning models in settings where data is unavailable.
Legal Named Entity Recognition with Multi-Task Domain Adaptation

Named Entity Recognition (NER) is a wellexplored area from Information Retrieval and Natural Language Processing with an extensive research community.Despite that, few languages, such as English and German, are well-resourced, whereas many other languages, such as Romanian, have scarce resources, especially in domain-specific applications.In this work, we address the NER problem in the legal domain from both Romanian and German languages and evaluate the performance of our proposed method based on domain adaptation.We employ multi-task learning to jointly train a neural network on two legal and general domains and perform adaptation among them.The results show that domain adaptation increase performances by a small amount, under 1%, while considerable improvements are in the recall metric.
The challenges of temporal alignment on Twitter during crises

Language use changes over time, and this impacts the effectiveness of NLP systems.This phenomenon is even more prevalent in social media data during crisis events where meaning and frequency of word usage may change over the course of days.Contextual language models fail to adapt temporally, emphasizing the need for temporal adaptation in models which need to be deployed over an extended period of time.While existing approaches consider data spanning large periods of time (from years to decades), shorter time spans are critical for crisis data.We quantify temporal degradation for this scenario and propose methods to cope with performance loss by leveraging techniques from domain adaptation.To the best of our knowledge, this is the first effort to explore effects of rapid language change driven by adversarial adaptations, particularly during natural and human-induced disasters.Through extensive experimentation on diverse crisis datasets, we analyze under what conditions our approaches outperform strong baselines while highlighting the current limitations of temporal adaptation methods in scenarios where access to unlabeled data is scarce. 1
Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models

The ability to extrapolate, i.e., to make predictions on sequences that are longer than those presented as training examples, is a challenging problem for current deep learning models.Recent work shows that this limitation persists in state-of-the-art Transformer-based models.Most solutions to this problem use specific architectures or training methods that do not generalize to other tasks.We demonstrate that large language models can succeed in extrapolation without modifying their architecture or training procedure.Our experimental results show that generating step-by-step rationales and introducing marker tokens are both required for effective extrapolation.First, we induce a language model to produce step-bystep rationales before outputting the answer to effectively communicate the task to the model.However, as sequences become longer, we find that current models struggle to keep track of token positions.To address this issue, we interleave output tokens with markup tokens that act as explicit positional and counting symbols.Our findings show how these two complementary approaches enable remarkable sequence extrapolation and highlight a limitation of current architectures to effectively generalize without explicit surface form guidance.Code available at https://github.com/MirelleB/induced-rationales-markup-tokens
Punctuation and case restoration in code mixed Indian languages

Automatic Speech Recognition (ASR) systems are taking over in different industries starting from producing video subtitles to interactive digital assistants.ASR output can be used in automatic indexing, categorizing, searching along with normal human readability.Raw transcripts from ASR systems are difficult to interpret since it usually produces text without punctuation and case information (all lower, all upper, camel case etc.), thus limiting the performance of downstream NLP tasks.We proposed an approach to restore the punctuation and case for both English and Hinglish (i.e Hindi vocabulary in Latin script) languages.We have performed a classification task using encoder-based transformers which is a mini BERT consisting of 4 encoder layers for punctuation and case restoration instead of the traditional Seq2Seq model considering the latency constraint in real world use cases.It consists of a total number of 15 distinct classes for the model which includes 5 punctuations i.e Period(.),Comma(,), Single Quote('), Double Quote(") & Question Mark(?) with different combinations of casing.The model is benchmarked on an internal dataset which was based on user conversation with the voice assistant and it achieves a F1(macro) score of 91.52% on the test set.
Syntactically Robust Training on Partially-Observed Data for Open Information Extraction

Open Information Extraction models have shown promising results with sufficient supervision.However, these models face a fundamental challenge that the syntactic distribution of training data is partially observable in comparison to the real world.In this paper, we propose a syntactically robust training framework that enables models to be trained on a syntacticabundant distribution based on diverse paraphrase generation.To tackle the intrinsic problem of knowledge deformation of paraphrasing, two algorithms based on semantic similarity matching and syntactic tree walking are used to restore the expressionally transformed knowledge.The training framework can be generally applied to other syntactic partial observable domains.Based on the proposed framework, we build a new evaluation set called CaRB-AutoPara, a syntactically diverse dataset consistent with the real-world setting for validating the robustness of the models.Experiments including a thorough analysis show that the performance of the model degrades with the increase of the difference in syntactic distribution, while our framework gives a robust boundary.
Investigating Transformer-Guided Chaining for Interpretable Natural Logic Reasoning

Natural logic reasoning has received increasing attention lately, with several datasets and neural models proposed, though with limited success. More recently, a new class of works have emerged adopting a Neuro-Symbolic approach, called transformer guided chaining, whereby the idea is to iteratively perform 1-step neural inferences and chain together the results to generate a multi-step reasoning trace. Several works have adapted variants of this central idea and reported significantly high accuracies compared to vanilla LLM’s. In this paper, we perform a critical empirical investigation of the chaining approach on a multi-hop First-Order Logic (FOL) reasoning benchmark. In particular, we develop a reference implementation, called Chainformer, and conduct several experiments to analyze the accuracy, generalization, interpretability, and performance over FOLs. Our findings highlight key strengths and possible current limitations and suggest potential areas for future research in logic reasoning.
DCU-ML at the FinNLP-2022 ERAI Task: Investigating the Transferability of Sentiment Analysis Data for Evaluating Rationales of Investors

In this paper, we describe our system for the FinNLP-2022 shared task: Evaluating the Rationales of Amateur Investors (ERAI).The ERAI shared tasks focuses on mining profitable information from financial texts by predicting the possible Maximal Potential Profit (MPP) and Maximal Loss (ML) based on the posts from amateur investors.There are two subtasks in ERAI: Pairwise Comparison and Unsupervised Rank, both target on the prediction of MPP and ML.To tackle the two tasks, we frame this task as a text-pair classification task where the input consists of two documents and the output is the label of whether the first document will lead to higher MPP or lower ML.Specifically, we propose to take advantage of the transferability of Sentiment Analysis data with an assumption that a more positive text will lead to higher MPP or higher ML to facilitate the prediction of MPP and ML.In experiment on the ERAI blind test set, our systems trained on Sentiment Analysis data and ERAI training data ranked 1st and 8th in ML and MPP pairwise comparison respectively.
Sentence-Incremental Neural Coreference Resolution

We propose a sentence-incremental neural coreference resolution system which incrementally builds clusters after marking mention boundaries in a shift-reduce method.The system is aimed at bridging two recent approaches at coreference resolution: (1) state-of-the-art non-incremental models that incur quadratic complexity in document length with high computational cost, and (2) memory networkbased models which operate incrementally but do not generalize beyond pronouns.For comparison, we simulate an incremental setting by constraining non-incremental systems to form partial coreference chains before observing new sentences.In this setting, our system outperforms comparable state-of-the-art methods by 2 F1 on OntoNotes and 6.8 F1 on the CODI-CRAC 2021 corpus.In a conventional coreference setup, our system achieves 76.3 F1 on OntoNotes and 45.5 F1 on CODI-CRAC 2021, which is comparable to state-of-the-art baselines.We also analyze variations of our system and show that the degree of incrementality in the encoder has a surprisingly large effect on the resulting performance. 11 Code is available at: https://github.com/mgrenander/sentence-incremental-coref
A Survey on Zero Pronoun Translation

Zero pronouns (ZPs) are frequently omitted in pro-drop languages (e.g.Chinese, Hungarian, and Hindi), but should be recalled in nonpro-drop languages (e.g.English).This phenomenon has been studied extensively in machine translation (MT), as it poses a significant challenge for MT systems due to the difficulty in determining the correct antecedent for the pronoun.This survey paper highlights the major works that have been undertaken in zero pronoun translation (ZPT) after the neural revolution so that researchers can recognize the current state and future directions of this field.We provide an organization of the literature based on evolution, dataset, method, and evaluation.In addition, we compare and analyze competing models and evaluation metrics on different benchmarks.We uncover a number of insightful findings such as: 1) ZPT is in line with the development trend of large language model; 2) data limitation causes learning bias in languages and domains; 3) performance improvements are often reported on single benchmarks, but advanced methods are still far from realworld use; 4) general-purpose metrics are not reliable on nuances and complexities of ZPT, emphasizing the necessity of targeted metrics; 5) apart from commonly-cited errors, ZPs will cause risks of gender bias.