Reviewerly/exHarmony huggingface dataset:
DISTANT: Distantly Supervised Entity Span Detection and Classification

We propose a distantly supervised pipeline NER which executes entity span detection and entity classification in sequence named DISTANT (DIstantly Supervised enTity spAN deTection and classification).The former entity span detector extracts possible entity mention spans by the distant supervision.Then the later entity classifier assigns each entity span to one of the positive entity types or none by employing a positive and unlabeled (PU) learning framework.Two models were built based on the pre-trained SciBERT model and fine-tuned with the silver corpus generated by the distant supervision.Experimental results on BC5CDR and NCBI-Disease datasets show that our method outperforms the end-to-end NER baselines without PU learning by a large margin.In particular, it increases the recall score effectively.
Local Temperature Beam Search: Avoid Neural Text DeGeneration via Enhanced Calibration

Previous studies have constantly observed that a language model repeats itself, creating repetitions in an output sequence.To cope with the issue, stochastic decoding schemes have been the de facto approaches; the strategies add randomness in inference, hence avoiding the "self-loop".However, the remedy comes at the cost of sacrificing output quality due to the randomness involved.In this work, we introduce a deterministic decoding scheme, local temperature beam search.This inference algorithm is an embarrassingly simple variant of beam search, yet it reduces repetition, whose level is superior to that of a sampling-based decoding algorithm, while maintaining the level of coherence as in beam search.Our idea is rooted in the concept of model calibration; we view a repetition as a casualty from overconfidence in a model.Therefore, our work mitigates the miscalibration present in the course of inference with a post-calibration approach applied in beam-specific manner.Our inference scheme is validated on text completion tasks, in which the repetition problem is seen most clearly, and is exhaustively compared with existing inference schemes.
PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts

A key component of modern conversational systems is the Dialogue State Tracker (or DST), which models a user's goals and needs.Toward building more robust and reliable DSTs, we introduce a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models.Two key characteristics of this approach are: (i) it only needs the output of the DST with no need for model parameters, and (ii) it can learn to generate natural language utterances that can target any DST.Through experiments over state-ofthe-art DSTs, the proposed framework leads to the greatest reduction in accuracy and the best attack success rate while maintaining good fluency and a low perturbation ratio.We also show how much the generated adversarial examples can bolster a DST through adversarial training.These results indicate the strength of prompt-based attacks on DSTs and leave open avenues for continued refinement.
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods

A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component.A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement -if multiple methods agree on an explanation, its credibility increases.However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods.In our work, we demonstrate that rank correlation is sensitive to small perturbations when evaluating agreement and argue that Pearson-r could be a better-suited alternative.We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods.By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances.Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods. 1
PruMUX: Augmenting Data Multiplexing with Model Compression

As language models increase in size by the day, methods for efficient inference are critical to leveraging their capabilities for various applications.Prior work has investigated techniques like model pruning, knowledge distillation, and data multiplexing to increase model throughput without sacrificing accuracy.In this paper, we combine two such methods -structured pruning and data multiplexing -to compound the speedup gains obtained by either method.Our approach, PruMUX, obtains up to 7.5-29.5Xthroughput improvement over BERTbase model with accuracy threshold from 80% to 74%.We further study various combinations of parameters (such as sparsity and multiplexing factor) in the two techniques to provide a comprehensive analysis of the tradeoff between accuracy and throughput in the resulting models.We then propose Auto-PruMUX, a meta-level model that can predict the highperformance parameters for pruning and multiplexing given a desired accuracy loss budget, providing a practical method to leverage the combination effectively. 1
Adaptation Approaches for Nearest Neighbor Language Models

Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, by leveraging large-scale neighborhood retrieval over external memory datastores. However, there has been little investigation into adapting such models for new domains. This work attempts to fill that gap and suggests the following approaches for adapting kNN-LMs — 1) adapting the underlying LM (using Adapters), 2) expanding neighborhood retrieval over an additional adaptation datastore, and 3) adapting the weights (scores) of retrieved neighbors using a learned Rescorer module. We study each adaptation strategy separately, as well as the combined performance improvement through ablation experiments and an extensive set of evaluations run over seven adaptation domains. Our combined adaptation approach consistently outperforms purely parametric adaptation and zero-shot (kNN-LM) baselines that construct datastores from the adaptation data. On average, we see perplexity improvements of 17.1% and 16% for these respective baselines, across domains.
CAIR-NLP at SemEval-2023 Task 2: A Multi-Objective Joint Learning System for Named Entity Recognition

This paper describes the NER system designed by the CAIR-NLP team for submission to Multilingual Complex Named Entity Recognition (MultiCoNER II) shared task, which presented a novel challenge of recognizing complex, ambiguous, and fine-grained entities in lowcontext, multi-lingual, multi-domain dataset and further evaluation on the noisy subset.We propose a Multi-Objective Joint Learning System (MOJLS) for NER, which aims to enhance the representation of entities and improve label predictions through joint implementation of a set of learning objectives.Our official submission MOJLS implements four objectives.These include, representation of the named entities should be close to its entity type definition, low-context inputs should have representation close to their augmented context, and also minimization of two label prediction errors, one based on CRF and another biaffine based predictions, where both are producing distributions over the output labels.The official results ranked our system 2 nd in five tracks (Multilingual, Spanish, Swedish, Ukrainian, and Farsi) and 3 rd in three tracks (French, Italian, and Portuguese) out of 13 tracks.Also evaluation on the noisy subset, our model achieved relatively better ranks.Official ranks indicate the effectiveness of the proposed MOJLS in dealing with the contemporary challenges of NER.
Can In-context Learners Learn a Reasoning Concept from Demonstrations?

Large language models show an emergent ability to learn a new task from a small number of input-output demonstrations.However, recent work shows that in-context learners largely rely on their pre-trained knowledge, such as the sentiment of the labels, instead of finding new associations in the input.However, the commonly-used few-shot evaluation settings using a random selection of in-context demonstrations can not disentangle models' ability to learn a new skill from demonstrations, as most of the randomly-selected demonstrations do not present relations informative for prediction beyond exposing the new task distribution.To disentangle models' in-context learning ability independent of models' memory, we introduce a Conceptual few-shot learning method selecting the demonstrations sharing a possiblyinformative concept with the predicted sample.We extract a set of such concepts from annotated explanations and measure how much can models benefit from presenting these concepts in few-shot demonstrations.We find that smaller models are more sensitive to the presented concepts.While some of the models are able to benefit from conceptpresenting demonstrations for each assessed concept, we find that none of the assessed incontext learners can benefit from all presented reasoning concepts consistently, leaving the incontext concept learning an open challenge.
A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches

EXtremely Weakly Supervised Text Classification (XWS-TC) refers to text classification based on minimal high-level human guidance, such as a few label-indicative seed words or classification instructions.There are two mainstream approaches for XWS-TC, however, never being rigorously compared: (1) training classifiers based on pseudo-labels generated by (softly) matching seed words (SEED) and ( 2) prompting (and calibrating) language models using classification instruction (and raw texts) to decode label words (PROMPT).This paper presents the first XWS-TC benchmark to compare the two approaches on fair grounds, where the datasets, supervisions, and hyperparameter choices are standardized across methods.Our benchmarking results suggest that (1) Both SEED and PROMPT approaches are competitive and there is no clear winner; (2) SEED is empirically more tolerant than PROMPT to human guidance (e.g., seed words, classification instructions, and label words) changes; (3) SEED is empirically more selective than PROMPT to the pre-trained language models; (4) Recent SEED and PROMPT methods have close connections and a clustering post-processing step based on raw in-domain texts is a strong performance booster to both.We hope this benchmark serves as a guideline in selecting XWS-TC methods in different scenarios and stimulate interest in developing guidance-and modelrobust XWS-TC methods 1 .
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference

Large-scale language model pretraining is a very successful form of self-supervised learning in natural language processing, but it is increasingly expensive to perform as the models and pretraining corpora have become larger over time.We propose NarrowBERT, a modified transformer encoder that increases the throughput for masked language model pretraining by more than 2×.NarrowBERT sparsifies the transformer model such that the selfattention queries and feedforward layers only operate on the masked tokens of each sentence during pretraining, rather than all of the tokens as with the usual transformer encoder.We also show that NarrowBERT increases the throughput at inference time by as much as 3.5× with minimal (or no) performance degradation on sentence encoding tasks like MNLI.Finally, we examine the performance of NarrowBERT on the IMDB and Amazon reviews classification and CoNLL NER tasks and show that it is also comparable to standard BERT performance.
Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models

Transformer-based language models trained on large natural language corpora have been very useful in downstream entity extraction tasks.However, they often result in poor performances when applied to domains that are different from those they are pretrained on.Continued pretraining using unlabeled data from target domains can help improve the performances of these language models on the downstream tasks.However, using all of the available unlabeled data for pretraining can be time-intensive; also, it can be detrimental to the performance of the downstream tasks, if the unlabeled data is not aligned with the data distribution for the target tasks.Previous works employed external supervision in the form of ontologies for selecting appropriate data samples for pretraining, but external supervision can be quite hard to obtain in low-resource domains.In this paper, we introduce effective ways to select data from unlabeled corpora of target domains for language model pretraining to improve the performances in target entity extraction tasks.Our data selection strategies do not require any external supervision.We conduct extensive experiments for the task of named entity recognition (NER) on seven different domains and show that language models pretrained on target domain unlabeled data obtained using our data selection strategies achieve better performances compared to those using data selection strategies in previous works that use external supervision.We also show that these pretrained language models using our data selection strategies outperform those pretrained on all of the available unlabeled target domain data.
Competency-Aware Neural Machine Translation: Can Machine Translation Know its Own Translation Quality?

Neural machine translation (NMT) is often criticized for failures that happen without awareness.The lack of competency awareness makes NMT untrustworthy.This is in sharp contrast to human translators who give feedback or conduct further investigations whenever they are in doubt about predictions.To fill this gap, we propose a novel competency-aware NMT by extending conventional NMT with a selfestimator, offering abilities to translate a source sentence and estimate its competency.The selfestimator encodes the information of the decoding procedure and then examines whether it can reconstruct the original semantics of the source sentence.Experimental results on four translation tasks demonstrate that the proposed method not only carries out translation tasks intact but also delivers outstanding performance on quality estimation.Without depending on any reference or annotated data typically required by state-of-the-art metric and quality estimation methods, our model yields an even higher correlation with human quality judgments than a variety of aforementioned methods, such as BLEURT, COMET, and BERTScore.Quantitative and qualitative analyses show better robustness of competency awareness in our model. 1
Retrieval Augmentation for Commonsense Reasoning: A Unified Approach

A common thread of retrieval-augmented methods in the existing literature focuses on retrieving encyclopedic knowledge, such as Wikipedia, which facilitates well-defined entity and relation spaces that can be modeled.However, applying such methods to commonsense reasoning tasks faces two unique challenges, i.e., the lack of a general large-scale corpus for retrieval and a corresponding effective commonsense retriever.In this paper, we systematically investigate how to leverage commonsense knowledge retrieval to improve commonsense reasoning tasks.We proposed a unified framework of Retrieval-Augmented Commonsense reasoning (called RACO), including a newly constructed commonsense corpus with over 20 million documents and novel strategies for training a commonsense retriever.We conducted experiments on four different commonsense reasoning tasks.Extensive evaluation results showed that our proposed RACO can significantly outperform other knowledgeenhanced method counterparts, achieving new SoTA performance on the CommonGen 1 and CREAK 2 leaderboards.Our code is available at https://github.com/wyu97/RACo.
Event-Based Knowledge MLM for Arabic Event Detection

With the fast pace of reporting around the globe from various sources, event extraction has increasingly become an important task in NLP.The use of pre-trained language models (PTMs) has become popular to provide contextual representation for downstream tasks.This work aims to pre-train language models that enhance event extraction accuracy.To this end, we propose an Event-Based Knowledge (EBK) masking approach to mask the most significant terms in the event detection task.These significant terms are based on an external knowledge source that is curated for the purpose of event detection for the Arabic language.The proposed approach improves the classification accuracy of all the 9 event types.The experimental results demonstrate the effectiveness of the proposed masking approach and encourage further exploration.
Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates

Calibration strengthens the trustworthiness of black-box models by producing better accurate confidence estimates on given examples.However, little is known about if model explanations can help confidence calibration.Intuitively, humans look at important features attributions and decide whether the model is trustworthy.Similarly, the explanations can tell us when the model may or may not know.Inspired by this, we propose a method named CME that leverages model explanations to make the model less confident with non-inductive attributions.The idea is that when the model is not highly confident, it is difficult to identify strong indications of any class, and the tokens accordingly do not have high attribution scores for any class and vice versa.We conduct extensive experiments on six datasets with two popular pre-trained language models in the in-domain and out-of-domain settings.The results show that CME improves calibration performance in all settings.The expected calibration errors are further reduced when combined with temperature scaling.Our findings highlight that model explanations can help calibrate posterior estimates.
Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue

Video-grounded Dialogue (VGD) aims to decode an answer sentence to a question regarding a given video and dialogue context.Despite the recent success of multi-modal reasoning to generate answer sentences, existing dialogue systems still suffer from a text hallucination problem, which denotes indiscriminate text-copying from input texts without an understanding of the question.This is due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts, thus the VGD system excessively relies on copying words from input texts by hoping those words to overlap with ground-truth texts.Hence, we design Text Hallucination Mitigating (THAM) framework, which incorporates Text Hallucination Regularization (THR) loss derived from the proposed information-theoretic text hallucination measurement approach.Applying THAM with current dialogue systems validates the effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows enhanced interpretability.
IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces

The ability to extract high-quality translation dictionaries from monolingual word embedding spaces depends critically on the geometric similarity of the spaces-their degree of "isomorphism."We address the root-cause of faulty cross-lingual mapping: that word embedding training resulted in the underlying spaces being non-isomorphic.We incorporate global measures of isomorphism directly into the Skip-gram loss function, successfully increasing the relative isomorphism of trained word embedding spaces and improving their ability to be mapped to a shared crosslingual space.The result is improved bilingual lexicon induction in general data conditions, under domain mismatch, and with training algorithm dissimilarities.We release IsoVec at https://github.com/kellymarchisio/isovec.
A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques

Authorship obfuscation techniques have commonly been evaluated based on their ability to hide the author's identity (evasion) while preserving the content of the original text.However, to avoid overstating the systems' effectiveness, evasion detection must be evaluated using competitive identification techniques in settings that mimic real-life scenarios, and the outcomes of the content-preservation evaluation have to be interpretable by potential users of these obfuscation tools.Motivated by recent work on cross-topic authorship identification and content preservation in summarization, we re-evaluate different authorship obfuscation techniques on detection evasion and content preservation.Furthermore, we propose a new information-theoretic measure to characterize the misattribution harm that can be caused by detection evasion.Our results 1 reveal key weaknesses in state-of-the-art obfuscation techniques and a surprisingly competitive effectiveness from a back-translation baseline in all evaluation aspects.
Improving compositional generalization for multi-step quantitative reasoning in question answering

Quantitative reasoning is an important aspect of question answering, especially when numeric and verbal cues interact to indicate sophisticated, multi-step programs.In this paper, we demonstrate how modeling the compositional nature of quantitative text can enhance the performance and robustness of QA models, allowing them to capture arithmetic logic that is expressed verbally.Borrowing from the literature on semantic parsing, we propose a method that encourages the QA models to adjust their attention patterns and capture input/output alignments that are meaningful to the reasoning task.We show how this strategy improves program accuracy and renders the models more robust against overfitting as the number of reasoning steps grows.Our approach is designed as a standalone module which can be prepended to many existing models and trained in an endto-end fashion without the need for additional supervisory signal.As part of this exercise, we also create a unified dataset building on four previously released numerical QA datasets over tabular data 1 .
IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach

In this paper, we describe our participation in the subtask 1 of CASE-2022, Event Causality Identification with Casual News Corpus.We address the Causal Relation Identification (CRI) task by exploiting a set of simple yet complementary techniques for fine-tuning language models (LMs) on a small number of annotated examples (i.e., a few-shot configuration).We follow a prompt-based prediction approach for fine-tuning LMs in which the CRI task is treated as a masked language modeling problem (MLM).This approach allows LMs natively pre-trained on MLM problems to directly generate textual responses to CRI-specific prompts.We compare the performance of this method against ensemble techniques trained on the entire dataset.Our bestperforming submission was fine-tuned with only 256 instances per class, 15.7% of the all available data, and yet obtained the second-best precision (0.82), third-best accuracy (0.82), and an F1-score (0.85) very close to what was reported by the winner team (0.86). 1