Reviewerly/exHarmony huggingface dataset:
Language acquisition: do children and language models follow similar learning stages?

During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master increasingly complex syntactic structures.However, the computational principles that lead to this learning trajectory remain largely unknown.To investigate this, we here compare the learning trajectories of deep language models to those of children.Specifically, we test whether, during its training, GPT-2 exhibits stages of language acquisition comparable to those observed in children aged between 18 months and 6 years.For this, we train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step, using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks.We then compare these evaluations with the behavior of 54 children during language production.Our analyses reveal three main findings.First, similarly to children, the language models tend to learn linguistic skills in a systematic order.Second, this learning scheme is parallel: the language tasks that are learned last improve from the very first training steps.Third, somebut not all -learning stages are shared between children and these language models.Overall, these results shed new light on the principles of language acquisition, and highlight important divergences in how humans and modern algorithms learn to process natural language.
Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models

Natural Language Inference (NLI) tasks involving temporal inference remain challenging for pre-trained language models (LMs).Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages.It is unclear whether current LMs realize the generalization capacity for temporal inference across languages.In this paper, we present JAMP, a Japanese NLI benchmark focused on temporal inference.Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis.To begin the data annotation process, we create diverse inference templates based on the formal semantics test suites.We then automatically generate diverse NLI examples by using the Japanese case frame dictionary and well-designed templates while controlling the distribution of inference patterns and gold labels.We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments (i.e., temporal inference patterns).Our findings demonstrate that LMs struggle with specific linguistic phenomena, such as habituality, indicating that there is potential for the development of more effective NLI models across languages.
Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree

Linguistic annotations, especially for controversial topics like hate speech detection, are frequently contested due to annotator backgrounds and positionalities.In such situations, preserving this disagreement through the machine learning pipeline can be important for downstream use cases.However, capturing disagreement can increase annotation time and expense.Fortunately, for many tasks, not all examples are equally controversial; we develop an active learning approach, Disagreement Aware Active Learning (DAAL) that concentrates annotations on examples where model entropy and annotator entropy are the most different.Because we cannot know the true entropy of annotations on unlabeled examples, we estimate a model that predicts annotator entropy trained using very few multiply-labeled examples.We find that traditional uncertainty-based active learning underperforms simple passive learning on tasks with high levels of disagreement, but that our active learning approach is able to successfully improve on passive learning, reducing the number of annotations required by at least 24% on average across several datasets.
Decouple knowledge from paramters for plug-and-play language modeling

Pre-trained language models (PLM) have made impressive results in various NLP tasks.It has been revealed that one of the key factors to their success is the parameters of these models implicitly learn all kinds of knowledge during pre-training.However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks.First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving.Second, it lacks interpretability and prevents humans from understanding which knowledge PLM requires for a certain problem.In this paper, we introduce PlugLM, a pre-training model with differentiable plug-in memory (DPM).The key intuition is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the DPM.To justify this design choice, we conduct evaluations in three settings including: (1) domain adaptation.PlugLM obtains 3.95 F1 improvements across four domains on average without any in-domain pre-training.(2) knowledge update.PlugLM could absorb new knowledge in a training-free way after pretraining is done.(3) in-task knowledge learning.PlugLM could be further improved by incorporating training samples into DPM with knowledge prompting 1 .
Generating Hashtags for Short-form Videos with Guided Signals

Short-form video hashtag recommendation (SVHR) aims to recommend hashtags to content creators from videos and corresponding descriptions.Most prior studies regard SVHR as a classification or ranking problem and select hashtags from a set of limited candidates.However, in reality, users can create new hashtags, and trending hashtags change rapidly over time on social media.Both of these properties cannot be easily modeled with classification approaches.To bridge this gap, we formulate SVHR as a generation task that better represents how hashtags are created naturally.Additionally, we propose the Guided Generative Model (GGM) where we augment the input features by retrieving relevant hashtags from a large-scale hashtag pool as extra guidance signals.Experimental results on two short-form video datasets show that our generative models outperform strong classification baselines, and the guidance signals further boost the performance by 8.11 and 2.17 absolute ROUGE-1 scores on average, respectively.We also perform extensive analyses including human evaluation, demonstrating that our generative model can create meaningful and relevant novel hashtags while achieving state-of-the-art performance on known hashtags 1 .
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning

The robustness of multimodal deep learning models to realistic changes in the input text is critical for their applicability to important tasks such as text-to-image retrieval and cross-modal entailment.To measure robustness, several existing approaches edit the text data, but do so without leveraging the cross-modal information present in multimodal data.Information from the visual modality, such as color, size, and shape, provide additional attributes that users can include in their inputs.Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., "girl on a chair" → "little girl on a wooden chair").Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic.We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and cross-modal entailment to perform poorly, resulting in relative drops of ∼ 15% in MRR and ∼ 20% in F 1 score, respectively.Crowd-sourced annotations demonstrate that cross-modal insertions lead to higher quality augmentations for multimodal data than augmentations using text-only data, and are equivalent in quality to original examples.We release the code to encourage robustness evaluations of deep vision-and-language models: https://github.com/claws-lab/multimodal-robustness
Event-independent temporal positioning: application to French clinical text

Extracting temporal relations usually entails identifying and classifying the relation between two mentions.However, the definition of temporal mentions strongly depends on the text type and the application domain.Clinical text in particular is complex.It may describe events that occurred at different times, contain redundant information and a variety of domain-specific temporal expressions.In this paper, we propose a novel eventindependent representation of temporal relations that is task-independent and, therefore, domain-independent.We are interested in identifying homogeneous text portions from a temporal standpoint and classifying the relation between each text portion and the document creation time.Temporal relation extraction is cast as a sequence labeling task and evaluated on oncology notes.We further evaluate our temporal representation by the temporal positioning of toxicity events of chemotherapy administrated to colon and lung cancer patients described in French clinical reports.An overall macro F-measure of 0.86 is obtained for temporal relation extraction by a neural token classification model trained on clinical texts written in French.Our results suggest that the toxicity event extraction task can be performed successfully by automatically identifying toxicity events and placing them within the patient timeline (F-measure .62).The proposed system has the potential to assist clinicians in the preparation of tumor board meetings.
Silver Syntax Pre-training for Cross-Domain Relation Extraction

Relation Extraction (RE) remains a challenging task, especially when considering realistic outof-domain evaluations.One of the main reasons for this is the limited training size of current RE datasets: obtaining high-quality (manually annotated) data is extremely expensive and cannot realistically be repeated for each new domain.An intermediate training step on data from related tasks has shown to be beneficial across many NLP tasks.However, this setup still requires supplementary annotated data, which is often not available.In this paper, we investigate intermediate pre-training specifically for RE.We exploit the affinity between syntactic structure and semantic RE, and identify the syntactic relations which are closely related to RE by being on the shortest dependency path between two entities.We then take advantage of the high accuracy of current syntactic parsers in order to automatically obtain large amounts of low-cost pre-training data.By pre-training our RE model on the relevant syntactic relations, we are able to outperform the baseline in five out of six cross-domain setups, without any additional annotated data.
Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization

The neural metrics recently received considerable attention from the research community in the automatic evaluation of machine translation.Unlike text-based metrics that have interpretable and consistent evaluation mechanisms for various data sources, the reliability of neural metrics in assessing out-of-distribution data remains a concern due to the disparity between training data and real-world data.This paper aims to address the inference bias of neural metrics through uncertainty minimization during test time, without requiring additional data.Our proposed method comprises three steps: uncertainty estimation, test-time adaptation, and inference.Specifically, the model employs the prediction uncertainty of the current data as a signal to update a small fraction of parameters during test time and subsequently refine the prediction through optimization.To validate our approach, we apply the proposed method to three representative models and conduct experiments on the WMT21 benchmarks.The results obtained from both in-domain and out-of-distribution evaluations consistently demonstrate improvements in correlation performance across different models.Furthermore, we provide evidence that the proposed method effectively reduces model uncertainty.
Automatic Readability Assessment for Closely Related Languages

In recent years, the main focus of research on automatic readability assessment (ARA) has shifted towards using expensive deep learningbased methods with the primary goal of increasing models' accuracy.This, however, is rarely applicable for low-resource languages where traditional handcrafted features are still widely used due to the lack of existing NLP tools to extract deeper linguistic representations.In this work, we take a step back from the technical component and focus on how linguistic aspects such as mutual intelligibility or degree of language relatedness can improve ARA in a low-resource setting.We collect short stories written in three languages in the Philippines -Tagalog, Bikol, and Cebuano -to train readability assessment models and explore the interaction of data and features in various crosslingual setups.Our results show that the inclusion of CROSSNGO, a novel specialized feature exploiting n-gram overlap applied to languages with high mutual intelligibility, significantly improves the performance of ARA models compared to the use of off-the-shelf large multilingual language models alone.Consequently, when both linguistic representations are combined, we achieve state-of-the-art results for Tagalog and Cebuano, and baseline scores for ARA in Bikol.We release our data and code at github.com/imperialite/ara-close-lang
Myths about Writing Systems in Speech &amp; Language Technology

Natural language processing is largely focused on written text processing.However, many computational linguists tacitly endorse myths about the nature of writing.We highlight two of these mythsthe conflation of language and writing, and the notion that Chinese, Japanese, and Korean writing is ideographic-and suggest how the community can dispel them.
RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation

Attribute-controlled translation (ACT) is a subtask of machine translation that involves controlling stylistic or linguistic attributes (like formality and gender) of translation outputs.While ACT has garnered attention in recent years due to its usefulness in real-world applications, progress in the task is currently limited by dataset availability, since most prior approaches rely on supervised methods.To address this limitation, we propose Retrieval and Attribute-Marking enhanced Prompting (RAMP), which leverages large multilingual language models to perform ACT in few-shot and zero-shot settings.RAMP improves generation accuracy over the standard prompting approach by (1) incorporating a semantic similarity retrieval component for selecting similar in-context examples, and (2) marking in-context examples with attribute annotations.Our comprehensive experiments show that RAMP is a viable approach in both zero-shot and few-shot settings.
Measuring the Effect of Influential Messages on Varying Personas

Predicting how a user responds to news events enables important applications such as allowing intelligent agents or content producers to estimate the effect on different communities and revise unreleased messages to prevent unexpected bad outcomes such as social conflict and moral injury.We present a new task, Response Forecasting on Personas for News Media, to estimate the response a persona (characterizing an individual or a group) might have upon seeing a news message.Compared to the previous efforts which only predict generic comments to news, the proposed task not only introduces personalization in the modeling but also predicts the sentiment polarity and intensity of each response.This enables more accurate and comprehensive inference on the mental state of the persona.Meanwhile, the generated sentiment dimensions make the evaluation and application more reliable.We create the first benchmark dataset, which consists of 13,357 responses to 3,847 news headlines from Twitter.We further evaluate the SOTA neural language models with our dataset.The empirical results suggest that the included persona attributes are helpful for the performance of all response dimensions.Our analysis shows that the best-performing models are capable of predicting responses that are consistent with the personas, and as a byproduct, the task formulation also enables many interesting applications in the analysis of social network groups and their opinions, such as the discovery of extreme opinion groups.
Exploring Large Language Models for Classical Philology

Recent advances in NLP have led to the creation of powerful language models for many languages including Ancient Greek and Latin.While prior work on Classical languages unanimously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages: we explore (i) encoder-only and encoder-decoder architectures using ROBERTA and T5 as strong model types, and create for each of them (ii) a monolingual Ancient Greek and a multilingual instance that includes Latin and English.We evaluate all models on morphological and syntactic tasks, including lemmatization, which demonstrates the added value of T5's decoding abilities.We further define two probing tasks to investigate the knowledge acquired by models pre-trained on Classical texts.Our experiments provide the first benchmarking analysis of existing models of Ancient Greek.Results show that our models provide significant improvements over the SoTA.The systematic analysis of model types can inform future research in designing language models for Classical languages, including the development of novel generative tasks.We make all our models available as community resources, along with a large curated pre-training corpus for Ancient Greek, to support the creation of a larger, comparable model zoo for Classical Philology.Our models and resources are available at https://github.com/Heidelberg-NLP/ancient-language-models.
Relational Sentence Embedding for Flexible Semantic Matching

We present Relational Sentence Embedding (RSE), a new paradigm to further discover the potential of sentence embeddings.Prior work mainly models the similarity between sentences based on their embedding distance.Because of the complex semantic meanings conveyed, sentence pairs can have various relation types, including but not limited to entailment, paraphrasing, and question-answer.It poses challenges to existing embedding methods to capture such relational information.We handle the problem by learning associated relational embeddings.Specifically, a relation-wise translation operation is applied to the source sentence to infer the corresponding target sentence with a pre-trained Siamese-based encoder.Later, the fine-grained relational similarity scores can be estimated with learned relational embeddings.We benchmark our method on 19 datasets covering a wide range of tasks, including semantic textual similarity, transfer, and domain-specific tasks.Experimental results show that our method is effective and flexible in modelling sentence relations and outperforms a series of state-of-the-art sentence embedding methods. 1
An end-to-end neural model based on cliques and scopes for frame extraction in long breast radiology reports

We consider the task of automatically extracting various overlapping frames, i.e, structured entities composed of multiple labels and mentions, from long clinical breast radiology documents.While many methods exist for related topics such as event extraction, slot filling, or discontinuous entity recognition, a challenge in our study resides in the fact that clinical reports typically contain overlapping frames that span multiple sentences or paragraphs.We propose a new method that addresses these difficulties and evaluate it on a new annotated corpus.Despite the small number of documents, we show that the hybridization between knowledge injection and a learning-based system allows us to quickly obtain proper results.We will also introduce the concept of scope relations and show that it both improves the performance of our system, and provides a visual explanation of the predictions.
Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors

In the wake of responsible AI, interpretability methods, which attempt to provide an explanation for the predictions of neural models have seen rapid progress.In this work, we are concerned with explanations that are applicable to natural language processing (NLP) models and tasks, and we focus specifically on the analysis of counterfactual, contrastive explanations.We note that while there have been several explainers proposed to produce counterfactual explanations, their behaviour can vary significantly and the lack of a universal ground truth for the counterfactual edits imposes an insuperable barrier on their evaluation.We propose a new back translation-inspired evaluation methodology that utilises earlier outputs of the explainer as ground truth proxies to investigate the consistency of explainers.We show that by iteratively feeding the counterfactual to the explainer we can obtain valuable insights into the behaviour of both the predictor and the explainer models, and infer patterns that would be otherwise obscured.Using this methodology, we conduct a thorough analysis and propose a novel metric to evaluate the consistency of counterfactual generation approaches with different characteristics across available performance indicators. 1
Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge

The ability of commonsense reasoning (CR) decides whether a neural machine translation (NMT) model can move beyond pattern recognition.Despite the rapid advancement of NMT and the use of pretraining to enhance NMT models, research on CR in NMT is still in its infancy, leaving much to be explored in terms of effectively training NMT models with high CR abilities and devising accurate automatic evaluation metrics.This paper presents a comprehensive study aimed at expanding the understanding of CR in NMT.For the training, we confirm the effectiveness of incorporating pretrained knowledge into NMT models and subsequently utilizing these models as robust testbeds for investigating CR in NMT.For the evaluation, we propose a novel entity-aware evaluation method that takes into account both the NMT candidate and important entities in the candidate, which is more aligned with human judgement.Based on the strong testbed and evaluation methods, we identify challenges in training NMT models with high CR abilities and suggest directions for further unlabeled data utilization and model design.We hope that our methods and findings will contribute to advancing the research of CR in NMT.