Reviewerly/exHarmony huggingface dataset:
Leveraging Synthetic Targets for Machine Translation

In this work, we provide a recipe for training machine translation models in a limited resource setting by leveraging synthetic target data generated using a large pre-trained model.We show that consistently across different benchmarks in bilingual, multilingual, and speech translation setups, training models on synthetic targets outperforms training on the actual ground-truth data.This performance gap grows bigger with increasing limits on the amount of available resources in the form of the size of the dataset and the number of parameters in the model.We also provide preliminary analysis into whether this boost in performance is linked to ease of optimization or more deterministic nature of the predictions, and whether this paradigm leads to better out-of-distribution performance across different testing domains.
Grammatical Error Correction for Sentence-level Assessment in Language Learning

The paper presents experiments on using a Grammatical Error Correction (GEC) model to assess the correctness of answers that language learners give to grammar exercises.We empirically check the hypothesis that the GEC model corrects only errors and leaves correct answers unchanged.We perform a test on assessing learner answers in a real but constrained language-learning setup: the learners answer only fill-in-the-blank and multiple-choice exercises.For this purpose, we use ReLCo, a publicly available manually annotated learner dataset in Russian (Katinskaia et al., 2022).In this experiment, we fine-tune a large-scale T5 language model for the GEC task and estimate its performance on the RULEC-GEC dataset (Rozovskaya and Roth, 2019) to compare with top-performing models.We also release an updated version of the RULEC-GEC test set, manually checked by native speakers.Our analysis shows that the GEC model performs reasonably well in detecting erroneous answers to grammar exercises, and potentially can be used in a real learning setting for the best-performing error types.However, it struggles to assess answers which were tagged by human annotators as alternative-correct using the aforementioned hypothesis.This is in large part due to a still low recall in correcting errors, and the fact that the GEC model may modify even correct words-it may generate plausible alternatives, which are hard to evaluate against the gold-standard reference.
What is the Real Intention behind this Question? Dataset Collection and Intention Classification

Asking and answering questions are inseparable parts of human social life.The primary purposes of asking questions are to gain knowledge or request help which has been the subject of question-answering studies.However, questions can also reflect negative intentions and include implicit offenses, such as highlighting one's lack of knowledge or bolstering an alleged superior knowledge, which can lead to conflict in conversations; yet has been scarcely researched.This paper is the first study to introduce a dataset (Question Intention Dataset) that includes questions with positive/neutral and negative intentions and the underlying intention categories within each group.We further conduct a meta-analysis to highlight tacit and apparent intents.We also propose a classification method using Transformers augmented by TF-IDF-based features and report the results of several models for classifying the main intention categories.We aim to highlight the importance of taking intentions into account, especially implicit and negative ones, to gain insight into conflict-evoking questions and better understand human-human communication on the web for NLP applications.
One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning

Fine-tuning pre-trained language models for multiple tasks tends to be expensive in terms of storage.To mitigate this, parameter-efficient transfer learning (PETL) methods have been proposed to address this issue, but they still require a significant number of parameters and storage when being applied to broader ranges of tasks.To achieve even greater storage reduction, we propose PROPETL, a novel method that enables efficient sharing of a single PETL module which we call prototype network (e.g., adapter, LoRA, and prefix-tuning) across layers and tasks.We then learn binary masks to select different sub-networks from the shared prototype network and apply them as PETL modules into different layers.We find that the binary masks can determine crucial information from the network, which is often ignored in previous studies.Our work can also be seen as a type of pruning method, where we find that overparameterization also exists in the seemingly small PETL modules.We evaluate PROPETL on various downstream tasks and show that it can outperform other PETL methods with approximately 10% of the parameter storage required by the latter. 1
Exploring the Compositional Generalization in Context Dependent Text-to-SQL Parsing

In the context-dependent Text-to-SQL task, the generated SQL statements are refined iteratively based on the user input utterance from each interaction.The input text from each interaction can be viewed as component modifications to the previous SQL statements, which could be further extracted as the modification patterns.Since these modification patterns could also be combined with other SQL statements, the models are supposed to have the compositional generalization to these novel combinations.This work is the first exploration of compositional generalization in context-dependent Text-to-SQL scenarios.To facilitate related studies, we constructed two challenging benchmarks named COSQL-CG and SPARC-CG by recombining the modification patterns and existing SQL statements.The following experiments show that all current models struggle on our proposed benchmarks.Furthermore, we found that better aligning the previous SQL statements with the input utterance could give models better compositional generalization ability.Based on these observations, we propose a method named p-align to improve the compositional generalization of Text-to-SQL models.
What Makes Pre-trained Language Models Better Zero-shot Learners?

Current methods for prompt learning in zeroshot scenarios widely rely on a development set with sufficient human-annotated data to select the best-performing prompt template a posteriori.This is not ideal because in a real-world zero-shot scenario of practical relevance, no labelled data is available.Thus, we propose a simple yet effective method for screening reasonable prompt templates in zero-shot text classification: Perplexity Selection (Perplection).We hypothesize that language discrepancy can be used to measure the efficacy of prompt templates, and thereby develop a substantiated perplexity-based scheme allowing for forecasting the performance of prompt templates in advance.Experiments show that our method leads to improved prediction performance in a realistic zero-shot setting, eliminating the need for any labelled examples.Dataset 1. [very/not] pleased.2.
Predicting Human Translation Difficulty Using Automatic Word Alignment

Translation difficulty arises when translators are required to resolve translation ambiguity from multiple possible translations.Translation difficulty can be measured by recording the diversity of responses provided by human translators and the time taken to provide these responses, but these behavioral measures are costly and do not scale.In this work, we use word alignments computed over large scale bilingual corpora to develop predictors of lexical translation difficulty.We evaluate our approach using behavioural data from translations provided both in and out of context, and report results that improve on a previous embeddingbased approach (Thompson et al., 2020).Our work can therefore contribute to a deeper understanding of cross-lingual differences and of causes of translation difficulty.
Backpack Language Models

We present Backpacks: a new neural architecture that marries strong modeling performance with an interface for interpretability and control.Backpacks learn multiple non-contextual sense vectors for each word in a vocabulary, and represent a word in a sequence as a contextdependent, non-negative linear combination of sense vectors in this sequence.We find that, after training, sense vectors specialize, each encoding a different aspect of a word.We can interpret a sense vector by inspecting its (non-contextual, linear) projection onto the output space, and intervene on these interpretable hooks to change the model's behavior in predictable ways.We train a 170M-parameter Backpack language model on OpenWebText, matching the loss of a GPT-2 small (124Mparameter) Transformer.On lexical similarity evaluations, we find that Backpack sense vectors outperform even a 6B-parameter Transformer LM's word embeddings.Finally, we present simple algorithms that intervene on sense vectors to perform controllable text generation and debiasing.For example, we can edit the sense vocabulary to tend more towards a topic, or localize a source of gender bias to a sense vector and globally suppress that sense.
A Question Answering Benchmark Database for Hungarian

Within the research presented in this article, we created a new question answering benchmark database for Hungarian called MILQA. When creating the dataset, we basically followed the principles of the English SQuAD 2.0, however, like in some more recent English question answering datasets, we introduced a number of innovations beyond SQuAD: e.g., yes/no-questions, list-like answers consisting of several text spans, long answers, questions requiring calculation and other question types where you cannot simply copy the answer from the text. For all these non-extractive question types, the pragmatically adequate form of the answer was also added to make the training of generative models possible. We implemented and evaluated a set of baseline retrieval and answer span extraction models on the dataset. BM25 performed better than any vector-based solution for retrieval. Cross-lingual transfer from English significantly improved span extraction models.
Deep Model Compression Also Helps Models Capture Ambiguity

Natural language understanding (NLU) tasks face a non-trivial amount of ambiguous samples where veracity of their labels is debatable among annotators.NLU models should thus account for such ambiguity, but they approximate the human opinion distributions quite poorly and tend to produce over-confident predictions.To address this problem, we must consider how to exactly capture the degree of relationship between each sample and its candidate classes.In this work, we propose a novel method with deep model compression and show how such relationship can be accounted for.We see that more reasonably represented relationships can be discovered in the lower layers and that validation accuracies are converging at these layers, which naturally leads to layer pruning.We also see that distilling the relationship knowledge from a lower layer helps models produce better distribution.Experimental results demonstrate that our method makes substantial improvement on quantifying ambiguity without gold distribution labels.As positive side-effects, our method is found to reduce the model size significantly and improve latency, both attractive aspects of NLU products. 1Premise It's summer time and two girls play with bubbles near a boat dock.Hypothesis It is warm outside.Label distribution Entailment: 0.
Interpreting Sentiment Composition with Latent Semantic Tree

As the key to sentiment analysis, sentiment composition considers the classification of a constituent via classifications of its contained sub-constituents and rules operated on them.Such compositionality has been widely studied previously in the form of hierarchical trees including untagged and sentiment ones, which are intrinsically suboptimal in our view.To address this, we propose semantic tree, a new tree form capable of interpreting the sentiment composition in a principled way.Semantic tree is a derivation of a context-free grammar (CFG) describing the specific composition rules on difference semantic roles, which is designed carefully following previous linguistic conclusions.However, semantic tree is a latent variable since there is no its annotation in regular datasets.Thus, in our method, it is marginalized out via inside algorithm and learned to optimize the classification performance.Quantitative and qualitative results demonstrate that our method not only achieves better or competitive results compared to baselines in the setting of regular and domain adaptation classification, and also generates plausible tree explanations 1 .
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation

Language models with the Transformers structure have shown great performance in natural language processing.However, there still poses problems when fine-tuning pre-trained language models on downstream tasks, such as over-fitting or representation collapse.In this work, we propose HyPe, a simple yet effective fine-tuning technique to alleviate such problems by perturbing hidden representations of Transformers layers.Unlike previous works that only add noise to inputs or parameters, we argue that the hidden representations of Transformers layers convey more diverse and meaningful language information.Therefore, making the Transformers layers more robust to hidden representation perturbations can further benefit the fine-tuning of PLMs en bloc.We conduct extensive experiments and analyses on GLUE and other natural language inference datasets.Results demonstrate that HyPe outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers.In addition, HyPe acquires negligible computational overheads, and is better than and compatible with previous state-of-theart fine-tuning techniques.Codes are released at https://github.com/Yuanhy1997/HyPe.
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning

Logical reasoning over text is an important ability that requires understanding the semantics of the text and reasoning through them to arrive at correct inferences.Prior works on pretraining language models to improve the logical reasoning ability require complex processing of training data (e.g., aligning symbolic knowledge to text), yielding task-specific solutions that are not easy to adapt to any general text corpus.In this work, we propose APOLLO, a simple adaptive pretraining approach to improve the logical reasoning skills of language models.We select a subset of Wikipedia for adaptive pretraining using a set of logical inference keywords as filter words.Further, we propose two self-supervised loss functions for training.First, we modify the masked language modeling loss to mask specific parts-of-speech words that likely require higher-order reasoning to predict them.Second, we propose a sentence-level classification loss that teaches the model to distinguish between entailment and contradiction types of sentences.The proposed pretraining paradigm is both simple and independent of task formats.We demonstrate the effectiveness of APOLLO by comparing it with prior baselines on two logical reasoning datasets.APOLLO performs comparably on ReClor and outperforms baselines on LogiQA.
Gradient Ascent Post-training Enhances Language Model Generalization

In this work, we empirically show that updating pretrained LMs (350M, 1.3B, 2.7B) with just a few steps of Gradient Ascent Post-training (GAP) on random, unlabeled text corpora enhances its zero-shot generalization capabilities across diverse NLP tasks.Specifically, we show that GAP can allow LMs to become comparable to 2-3x times larger LMs across 12 different NLP tasks.We also show that applying GAP on out-of-distribution corpora leads to the most reliable performance improvements.Our findings indicate that GAP can be a promising method for improving the generalization capability of LMs without any task-specific finetuning 1 .
Fantastic Expressions and Where to Find Them: Chinese Simile Generation with Multiple Constraints

Similes occur in the creative context of describing a concept (i.e., tenor) by making a literally false yet figuratively meaningful comparison to another (i.e., vehicle).Previous efforts form simile generation as a context-free generation task, focusing on simile-style transfer or writing a simile from a given prefix.However, generated texts under such settings might be undesirable, such as hardly meeting the simile definition (e.g., missing vehicle) or difficult to address certain preferences of content as humans wish (e.g., describe the color of apples through the simile).We believe that a simile could be more qualified and user-oriented if incorporated with pre-specified constraints.To this end, we introduce controllable simile generation (CSG), a new task that requires the model to generate a simile with multiple simile elements, e.g., context and vehicle.To facilitate this task, we present GraCe, including 61.3k simile-element annotated Chinese similes.Based on it, we propose a CSG model Similor to benchmark this task, including a vehicle retrieval module Scorer to obtain the explicable comparison for a given tenor in the vehicle-unknown situation.Both statistical and experimental analyses show that GraCe is of high quality beyond all other Chinese simile datasets, in terms of the number (8 vs. 3) of annotation elements, Is-Simile accuracy (98.9% vs. 78.7%),and increasing model-performance gains for both uncontrollable and controllable simile generation.Meanwhile, Similor can serve as a strong baseline for CSG, especially with Scorer, which beats model-based retrieval methods without any re-training.
Using Neural Machine Translation for Generating Diverse Challenging Exercises for Language Learner

We propose a novel approach to automatically generate distractors for cloze exercises for English language learners, using round-trip neural machine translation.A carrier sentence is translated from English into another (pivot) language and back, and distractors are produced by aligning the original sentence with its round-trip translation.We make use of 16 linguistically-diverse pivots and generate hundreds of translation hypotheses in each direction.We show that using hundreds of translations allows us to generate a rich set of challenging distractors.Moreover, we find that typologically unrelated language pivots contribute more diverse candidate distractors, compared to language pivots that are closely related.We further evaluate the use of machine translation systems of varying quality and find that better quality MT systems produce more challenging distractors.Finally, we conduct a study with language learners, demonstrating that the automatically generated distractors are of the same difficulty as the gold distractors produced by human experts. 1
SciLit: A Platform for Joint Scientific Literature Discovery, Summarization and Citation Generation

Scientific writing involves retrieving, summarizing, and citing relevant papers, which can be time-consuming processes.Although in many workflows these processes are serially linked, there are opportunities for natural language processing (NLP) to provide end-to-end assistive tools.We propose SCILIT, a pipeline that automatically recommends relevant papers, extracts highlights, and suggests a reference sentence as a citation of a paper, taking into consideration the user-provided context and keywords.SCILIT efficiently recommends papers from large databases of hundreds of millions of papers using a two-stage prefetching and re-ranking literature search system that flexibly deals with addition and removal of a paper database.We provide a convenient user interface that displays the recommended papers as extractive summaries and that offers abstractively-generated citing sentences which are aligned with the provided context and which mention the chosen keyword(s).Our assistive tool for literature discovery and scientific writing is available at https://scilit.vercel.app
Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization

The multilingual neural machine translation (NMT) model has a promising capability of zero-shot translation, where it could directly translate between language pairs unseen during training.For good transfer performance from supervised directions to zero-shot directions, the multilingual NMT model is expected to learn universal representations across different languages.This paper introduces a cross-lingual consistency regularization, Cross-ConST, to bridge the representation gap among different languages and boost zero-shot translation performance.The theoretical analysis shows that CrossConST implicitly maximizes the probability distribution for zero-shot translation, and the experimental results on both low-resource and high-resource benchmarks show that CrossConST consistently improves the translation performance.The experimental analysis also proves that CrossConST could close the sentence representation gap and better align the representation space.Given the universality and simplicity of CrossConST, we believe it can serve as a strong baseline for future multilingual NMT research.
On Search Strategies for Document-Level Neural Machine Translation

Compared to sentence-level systems, document-level neural machine translation (NMT) models produce a more consistent output across a document and are able to better resolve ambiguities within the input.There are many works on document-level NMT, mostly focusing on modifying the model architecture or training strategy to better accommodate the additional context-input.On the other hand, in most works, the question on how to perform search with the trained model is scarcely discussed, sometimes not mentioned at all.In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding.We start with the most popular document-level NMT approach and compare different decoding schemes, some from the literature and others proposed by us.In the comparison, we are using both, standard automatic metrics, as well as specific linguistic phenomena on three standard document-level translation benchmarks.We find that most commonly used decoding strategies perform similar to each other and that higher quality context information has the potential to further improve the translation.
Combining Tradition with Modernness: Exploring Event Representations in Vision-and-Language Models for Visual Goal-Step Inference

Procedural knowledge understanding underlies the ability to infer goal-step relations.The task of Visual Goal-Step Inference addresses this ability in the multimodal domain.It requires the identification of images that depict the steps necessary to accomplish a textually expressed goal.The best existing methods encode texts and images either with independent encoders, or with object-level multimodal encoders using blackbox transformers.This stands in contrast to early, linguistically inspired methods for event representations, which focus on capturing the most crucial information, namely actions and participants, to learn stereotypical event sequences and hence procedural knowledge.In this work, we study various methods and their effects on procedural knowledge understanding of injecting the early shallow event representations to nowadays multimodal deep learning-based models.We find that the early, linguistically inspired methods for representing event knowledge do contribute to understand procedures in combination with modern visionand-language models.This supports further exploration of more complex event structures in combination with large language models. 1