Reviewerly/exHarmony huggingface dataset:
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning.Yet, they fail miserably on the recently proposed Winoground dataset (Thrush et al., 2022), which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug").By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images.In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset.Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding.We release our annotation and code at https://github.com/ajd12342/why-winoground-hard.
Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts

Anaphora resolution is an important task, which traditionally has required costly supervised training datasets for each new language, text genre, and domain.Meanwhile, prompting large language models with a few in-context examples has emerged as a promising approach to reduce labeling costs, however there are a number of challenges in applying in-context learning to resolve anaphora.In this paper, we present MICE (Mixtures of In-Context Experts), which we demonstrate is effective for few-shot anaphora resolution in the domain of scientific protocols (Tamari et al., 2021).Given only a handful of training examples, MICE combines the predictions of hundreds of in-context experts, yielding a 30% increase in F 1 score over a competitive prompt retrieval baseline.Furthermore, we show MICE can be used to train compact student models without sacrificing performance.As far as we are aware, this is the first work to present experimental results demonstrating the effectiveness of in-context learning on the task of few-shot anaphora resolution in scientific protocols. 1of transfer learning with a unified text-to-text transformer.
Reconciliation of Pre-trained Models and Prototypical Neural Networks in Few-shot Named Entity Recognition

Incorporating large-scale pre-trained models with the prototypical neural networks is a de-facto paradigm in few-shot named entity recognition.Existing methods, unfortunately, are not aware of the fact that embeddings from pre-trained models contain a prominently large amount of information regarding word frequencies, biasing prototypical neural networks against learning word entities.This discrepancy constrains the two models' synergy.Thus, we propose a one-line-code normalization method to reconcile such a mismatch with empirical and theoretical grounds.Our experiments based on nine benchmark datasets show the superiority of our method over the counterpart models and are comparable to the stateof-the-art methods.In addition to the model enhancement, our work also provides an analytical viewpoint for addressing the general problems in few-shot name entity recognition or other tasks that rely on pre-trained models or prototypical neural networks. 1
Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translation

Simultaneous machine translation (SiMT) outputs the translation while receiving the source inputs, and hence needs to balance the received source information and translated target information to make a reasonable decision between waiting for inputs or outputting translation.Previous methods always balance source and target information at the token level, either directly waiting for a fixed number of tokens or adjusting the waiting based on the current token.In this paper, we propose a Wait-info Policy to balance source and target at the information level.We first quantify the amount of information contained in each token, named info.Then during simultaneous translation, the decision of waiting or outputting is made based on the comparison results between the total info of previous target outputs and received source inputs.Experiments show that our method outperforms strong baselines under and achieves better balance via the proposed info 1 .
Third-Party Aligner for Neural Word Alignments

Word alignment is to find translationally equivalent words between source and target sentences.Previous work has demonstrated that self-training can achieve competitive word alignment results.In this paper, we propose to use word alignments generated by a third-party word aligner to supervise the neural word alignment training.Specifically, source word and target word of each word pair aligned by the third-party aligner are trained to be close neighbors to each other in the contextualized embedding space when fine-tuning a pre-trained cross-lingual language model.Experiments on the benchmarks of various language pairs show that our approach can surprisingly do self-correction over the third-party supervision by finding more accurate word alignments and deleting wrong word alignments, leading to better performance than various third-party word aligners, including the currently best one.When we integrate all supervisions from various third-party aligners, we achieve state-of-the-art word alignment performances, with averagely more than two points lower alignment error rates than the best third-party aligner.We released our code at https://github.com/sdongchuanqi/Third-Party-Supervised-Aligner.
Revisiting Grammatical Error Correction Evaluation and Beyond

Pretraining-based (PT-based) automatic evaluation metrics (e.g., BERTScore and BARTScore) have been widely used in several sentence generation tasks (e.g., machine translation and text summarization) due to their better correlation with human judgments over traditional overlapbased methods.Although PT-based methods have become the de facto standard for training grammatical error correction (GEC) systems, GEC evaluation still does not benefit from pretrained knowledge.This paper takes the first step towards understanding and improving GEC evaluation with pretraining.We first find that arbitrarily applying PT-based metrics to GEC evaluation brings unsatisfactory correlation results because of the excessive attention to inessential systems outputs (e.g., unchanged parts).To alleviate the limitation, we propose a novel GEC evaluation metric to achieve the best of both worlds, namely PT-M 2 , which only uses PT-based metrics to score those corrected parts.Experimental results on the CoNLL14 evaluation task show that PT-M 2 significantly outperforms existing methods, achieving a new state-of-the-art result of 0.949 Pearson correlation.Further analysis reveals that PT-M 2 is robust to evaluate competitive GEC systems.
SUMMARY WORKBENCH: Unifying Application and Evaluation of Text Summarization Models

This paper presents SUMMARY WORKBENCH, a new tool for developing and evaluating text summarization models.New models and evaluation measures can be easily integrated as Docker-based plugins, allowing to examine the quality of their summaries against any input and to evaluate them using various evaluation measures.Visual analyses combining multiple measures provide insights into the models' strengths and weaknesses.The tool is hosted at https://tldr.demo.webis.deand also supports local deployment for private resources.
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence

Question answering models can use rich knowledge sources -up to one hundred retrieved passages and parametric knowledge in the large-scale language model (LM).Prior work assumes information in such knowledge sources is consistent with each other, paying little attention to how models blend information stored in their LM parameters with that from retrieved evidence documents.In this paper, we simulate knowledge conflicts (i.e., where parametric knowledge suggests one answer and different passages suggest different answers) and examine model behaviors.We find retrieval performance heavily impacts which sources models rely on, and current models mostly rely on non-parametric knowledge in their best-performing settings.We discover a troubling trend that contradictions among knowledge sources affect model confidence only marginally.To address this issue, we present a new calibration study, where models are discouraged from presenting any single answer when presented with multiple conflicting answer candidates in retrieved evidences.
Generating Coherent Narratives with Subtopic Planning to Answer How-to Questions

Answering how-to questions remains a major challenge in question answering research.A vast number of narrow, long-tail questions cannot be readily answered using a search engine.Moreover, there is little to no annotated data available to help develop such systems.This paper makes a first attempt at generating coherent, long-form answers for how-to questions.We propose new architectures, consisting of passage retrieval, subtopic planning and narrative generation, to consolidate multiple relevant passages into a coherent, explanatory answer.Our subtopic planning module aims to produce a set of relevant, diverse subtopics that serve as the backbone for answer generation to improve topic coherence.We present extensive experiments on a WikiHow dataset repurposed for long-form question answering.Empirical results demonstrate that generating narratives to answer how-to questions is a challenging task.Nevertheless, our architecture incorporated with subtopic planning can produce highquality, diverse narratives evaluated using automatic metrics and human assessment. 1 1 We have made our dataset and source code available at https://github.com/pengshancai/how-to-QAHow to Stop the Spread of Covid-19 Method 1 Method 2 Practicing Healthy Habits Stay at home if you can.Although it's a simple solution, avoiding contact with other people is an effective way to prevent the spread of COVID-19… Wear a mask or face covering before going out in public.Slip on a medical mask or face covering before you go grocery shopping, or enter any public area…How to Naturally Reduce Blood Pressure? 1. Changing your diet Eat a balanced diet.Drink lots of water.Cut back on salt intake.Avoid alcohol and smoking... 2. Exercise regularly Keep the habit of working out.Lose weight if you are overweight... 1. Changing your diet.2. Getting medical help… 1. Changing your habits.2. Making dietary changes…
Mitigating Covertly Unsafe Text within Natural Language Systems

An increasingly prevalent problem for intelligent technologies is text safety, as uncontrolled systems may generate recommendations to their users that lead to injury or life-threatening consequences.However, the degree of explicitness of a generated statement that can cause physical harm varies.In this paper, we distinguish types of text that can lead to physical harm and establish one particularly underexplored category: covertly unsafe text.Then, we further break down this category with respect to the system's information and discuss solutions to mitigate the generation of text in each of these subcategories.Ultimately, our work defines the problem of covertly unsafe language that causes physical harm and argues that this subtle yet dangerous issue needs to be prioritized by stakeholders and regulators.We highlight mitigation strategies to inspire future researchers to tackle this challenging problem and help improve safety within smart systems.
Conditional Language Models for Community-Level Linguistic Variation

Community-level linguistic variation is a core concept in sociolinguistics.In this paper, we use conditioned neural language models to learn vector representations for 510 online communities.We use these representations to measure linguistic variation between communities and investigate the degree to which linguistic variation corresponds with social connections between communities.We find that our sociolinguistic embeddings are highly correlated with a social network-based representation that does not use any linguistic input.
Generating Natural Language Proofs with Verifier-Guided Search

Reasoning over natural language is a challenging problem in NLP.In this work, we focus on proof generation: Given a hypothesis and a set of supporting facts, the model generates a proof tree indicating how to derive the hypothesis from supporting facts.Compared to generating the entire proof in one shot, stepwise generation can better exploit the compositionality and generalize to longer proofs but has achieved limited success on real-world data.Existing stepwise methods struggle to generate proof steps that are both logically valid and relevant to the hypothesis.Instead, they tend to hallucinate invalid steps given the hypothesis.In this paper, we present a novel stepwise method, NLProofS (Natural Language Proof Search), which learns to generate relevant steps conditioning on the hypothesis.At the core of our approach, we train an independent verifier to check the validity of the proof steps to prevent hallucination.Instead of generating steps greedily, we search for proofs maximizing a global proof score judged by the verifier.NL-ProofS achieves state-of-the-art performance on EntailmentBank and RuleTaker.Specifically, it improves the correctness of predicted proofs from 27.7% to 33.3% in the distractor setting of EntailmentBank, demonstrating the effectiveness of NLProofS in generating challenging human-authored proofs. 1
Multilingual SubEvent Relation Extraction: A Novel Dataset and Structure Induction Method

Subevent Relation Extraction (SRE) is a task in Information Extraction that aims to recognize spatial and temporal containment relations between event mentions in text.Recent methods have utilized pre-trained language models to represent input texts for SRE.However, a key issue in existing SRE methods is the employment of sequential order of words in texts to feed into representation learning methods, thus unable to explicitly focus on important context words and their interactions to enhance representations.In this work, we introduce a new method for SRE that learns to induce effective graph structures for input texts to boost representation learning.Our method features a word alignment framework with dependency paths and optimal transport to identify important context words to form effective graph structures for SRE.In addition, to enable SRE research on non-English languages, we present a new multilingual SRE dataset for five typologically different languages.Extensive experiments reveal the state-of-the-art performance for our method on different datasets and languages.
Low-resource Neural Machine Translation with Cross-modal Alignment

How to achieve neural machine translation with limited parallel data?Existing techniques often rely on large-scale monolingual corpora, which is impractical for some low-resource languages.In this paper, we turn to connect several low-resource languages to a particular high-resource one by additional visual modality.Specifically, we propose a cross-modal contrastive learning method to learn a shared space for all languages, where both a coarsegrained sentence-level objective and a finegrained token-level one are introduced.Experimental results and further analysis show that our method can effectively learn the crossmodal and cross-lingual alignment with a small amount of image-text pairs and achieves significant improvements over the text-only baseline under both zero-shot and few-shot scenarios.Our code could be found at https: //github.com/ictnlp/LNMT-CA.
A Corpus and Evaluation for Predicting Semi-Structured Human Annotations

A wide variety of tasks have been framed as text-to-text tasks to allow processing by sequence-to-sequence models.We propose a new task of generating a semi-structured interpretation of a source document.The interpretation is semi-structured in that it contains mandatory and optional fields with free-text information.This structure is surfaced by human annotations, which we standardize and convert to text format.We then propose an evaluation technique that is generally applicable to any such semi-structured annotation, called equivalence classes evaluation.The evaluation technique is efficient and scalable; it creates a large number of evaluation instances from a comparably cheap clustering of the free-text information by domain experts.For our task, we release a dataset about the monetary policy of the Federal Reserve.On this corpus, our evaluation shows larger differences between pretrained models than standard text generation metrics.
Modeling Complex Dialogue Mappings via Sentence Semantic Segmentation Guided Conditional Variational Auto-Encoder

Complex dialogue mappings (CDM), including one-to-many and many-to-one mappings, tend to make dialogue models generate incoherent or dull responses, and modeling these mappings remains a huge challenge for neural dialogue systems.To alleviate these problems, methods like introducing external information, reconstructing the optimization function, and manipulating data samples are proposed, while they primarily focus on avoiding training with CDM, inevitably weakening the model's ability of understanding CDM in human conversations and limiting further improvements in model performance.This paper proposes a Sentence Semantic Segmentation guided Conditional Variational Auto-Encoder (SegCVAE) method which can model and take advantages of the CDM data.Specifically, to tackle the incoherent problem caused by one-tomany, SegCVAE uses response-related prominent semantics to constrained the latent variable.To mitigate the non-diverse problem brought by many-to-one, SegCVAE segments multiple prominent semantics to enrich the latent variables.Three novel components, Internal Separation, External Guidance, and Semantic Norms, are proposed to achieve SegCVAE.On dialogue generation tasks, both the automatic and human evaluation results show that SegCVAE achieves new state-of-the-art performance.
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging

Knowledge Distillation (KD) is a commonly used technique for improving the generalization of compact Pre-trained Language Models (PLMs) on downstream tasks.However, such methods impose the additional burden of training a separate teacher model for every new dataset.Alternatively, one may directly work on the improvement of the optimization procedure of the compact model towards better generalization.Recent works observe that the flatness of the local minimum correlates well with better generalization.In this work, we adapt Stochastic Weight Averaging (SWA), a method encouraging convergence to a flatter minimum, to fine-tuning PLMs.We conduct extensive experiments on various NLP tasks (text classification, question answering, and generation) and different model architectures and demonstrate that our adaptation improves the generalization without extra computation cost.Moreover, we observe that this simple optimization technique is able to outperform the state-of-the-art KD methods for compact models.
JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation

Transformer-based autoregressive and nonautoregressive models have played an essential role in sequence generation tasks.The autoregressive model can obtain excellent performance, while the non-autoregressive model brings fast decoding speed for inference.In this paper, we propose JANUS, a Joint Autoregressive and Non-autoregressive training method using aUxiliary losS to enhance the model performance in both AR and NAR manner simultaneously and effectively alleviate the problem of distribution discrepancy.Further, we pre-train BART with JANUS on a large corpus with minimal cost (16 GPU days) and make the BART-JANUS capable of nonautoregressive generation, demonstrating that our approach can transfer the AR knowledge to NAR.Empirically, we show our approach and BART-JANUS can achieve significant improvement on multiple generation tasks, including machine translation and GLGE benchmarks.Our code is available at Github 1 .
PIE-QG: Paraphrased Information Extraction for Unsupervised Question Generation from Small Corpora

Supervised Question Answering systems (QA systems) rely on domain-specific humanlabeled data for training.Unsupervised QA systems generate their own question-answer training pairs, typically using secondary knowledge sources to achieve this outcome.Our approach (called PIE-QG) uses Open Information Extraction (OpenIE) to generate synthetic training questions from paraphrased passages and uses the question-answer pairs as training data for a language model for a state-of-the-art QA system based on BERT.Triples in the form of <subject, predicate, object> are extracted from each passage, and questions are formed with subjects (or objects) and predicates while objects (or subjects) are considered as answers.Experimenting on five extractive QA datasets demonstrates that our technique achieves onpar performance with existing state-of-the-art QA systems with the benefit of being trained on an order of magnitude fewer documents and without any recourse to external reference data sources.
A Rule based Modified Quadratic Mean for the Elimination of Outliers in Images

A Rule based Modified Quadratic Mean based for the Elimination of Outliers in Natural Images is proposed. The algorithm employs series of decision that checks for fixed valued outliers and noisy information about the vicinity that is being processed. The Experimental results suggests that the Modified quadratic Mean fairs well when compared to existing algorithms and Conventional Quadratic Mean. The Algorithm restored the image by keeping the information intact.
Stanford Parser and OpenNLP in Analyzing the Literary Text “The Alchemist” by Paulo Coelho

Natural Language Processing (NLP) tools have gained considerable attention in recent years due to their ability to analyze and understand human language. In this study, we explore the effectiveness of using two popular NLP tools, Stanford Parser and OpenNLP, in analyzing 50 statements from the novel “The Alchemist” by Paulo Coelho. The statements were analyzed using three NLP techniques: tokenization, part-of-speech tagging, and dependency parsing. Tokenization is the process of dividing a text into individual tokens, or words. Part-of-speech tagging is the process of assigning a grammatical category, such as noun or verb, to each word. Dependency parsing is the process of identifying the grammatical relationships between words in a sentence, such as subject-verb-object. The study aims to demonstrate the effectiveness of using NLP tools such as Stanford Parser and OpenNLP in analyzing literary texts, particularly in identifying the grammatical structure and relationships between words in a sentence. These tools can provide valuable insights into the text that may not be immediately apparent through manual analysis alone. Furthermore, the study highlights the need for careful consideration when selecting an NLP tool, as different tools may excel in different tasks depending on the nature of the text being analyzed. The analysis of literary texts using NLP tools has several applications, including literary analysis, sentiment analysis, and authorship attribution. By analyzing the grammatical structure of a text, NLP tools can help to identify the author’s writing style, the tone of the text, and the author’s use of literary devices. Additionally, NLP tools can be used to analyze the sentiment of a text, identifying positive or negative emotions expressed in the language.