Reviewerly/exHarmony huggingface dataset:
A Memory Model for Question Answering from Streaming Data Supported by Rehearsal and Anticipation of Coreference Information

Existing question answering methods often assume that the input content (e.g., documents or videos) is always accessible to solve the task.Alternatively, memory networks were introduced to mimic the human process of incremental comprehension and compression of the information in a fixed-capacity memory.However, these models only learn how to maintain memory by backpropagating errors in the answers through the entire network.Instead, it has been suggested that humans have effective mechanisms to boost their memorization capacities, such as rehearsal and anticipation.Drawing inspiration from these, we propose a memory model that performs rehearsal and anticipation while processing inputs to memorize important information for solving question answering tasks from streaming data.The proposed mechanisms are applied self-supervised during training through masked modeling tasks focused on coreference information.We validate our model on a short-sequence (bAbI) dataset as well as large-sequence textual (Nar-rativeQA) and video (ActivityNet-QA) question answering datasets, where it achieves substantial improvements over previous memory network approaches.Furthermore, our ablation study confirms the proposed mechanisms' importance for memory models.
Conjunct Lengths in English, Dependency Length Minimization, and Dependency Structure of Coordination

This paper confirms that, in English binary coordinations, left conjuncts tend to be shorter than right conjuncts, regardless of the position of the governor of the coordination.We demonstrate that this tendency becomes stronger when length differences are greater, but only when the governor is on the left or absent, not when it is on the right.We explain this effect via Dependency Length Minimization and we show that this explanation provides support for symmetrical dependency structures of coordination, as opposed to structures headed by the first conjunct.
HW-TSC at IWSLT2023: Break the Quality Ceiling of Offline Track via Pre-Training and Domain Adaptation

This paper describes HW-TSC's submissions to the IWSLT 2023 Offline Speech Translation task, including speech translation of talks from English to German, English to Chinese and English to Japanese.We participated in all three tracks (Constrained training, Constrained with Large Language Models training, Unconstrained training), with using cascaded architectures models.We use data enhancement, pretraining models and other means to improve the quality of ASR, and use a variety of techniques including R-Drop, deep model, domain data selection, etc. to improve the quality of NMT.Compared with last year's best results, we have improved by 2.1 BLEU in the MuST-C English-German test set.
Robust Learning for Multi-party Addressee Recognition with Discrete Addressee Codebook

Addressee recognition aims to identify addressees in multi-party conversations. While state-of-the-art addressee recognition models have achieved promising performance, they still suffer from the issue of robustness when applied in real-world scenes. When exposed to a noisy environment, these models regard the noise as input and identify the addressee in a pre-given addressee closed set, while the addressees of the noise do not belong to this closed set, thus leading to the wrong identification of addressee. To this end, we propose a Robust Addressee Recognition (RAR) method, which discrete the addressees into a character codebook, making it able to represent open set addressees and robust in a noisy environment. Experimental results show that the introduction of the addressee character codebook helps to represent the open set addressees and highly improves the robustness of addressee recognition even if the input is noise.
A Cautious Generalization Goes a Long Way: Learning Morphophonological Rules

Explicit linguistic knowledge, encoded by resources such as rule-based morphological analyzers, continues to prove useful in downstream NLP tasks, especially for low-resource languages and dialects.Rules are an important asset in descriptive linguistic grammars.However, creating such resources is usually expensive and non-trivial, especially for spoken varieties with no written standard.In this work, we present a novel approach for automatically learning morphophonological rules of Arabic from a corpus.Motivated by classic cognitive models for rule learning, rules are generalized cautiously.Rules that are memorized for individual items are only allowed to generalize to unseen forms if they are sufficiently reliable in the training data.The learned rules are further examined to ensure that they capture true linguistic phenomena described by domain experts.We also investigate the learnability of rules in low-resource settings across different experimental setups and dialects
Knowledge Injection for Disease Names in Logical Inference between Japanese Clinical Texts

In the medical field, there are many clinical texts such as electronic medical records, and research on Japanese natural language processing using these texts has been conducted.One such research involves Recognizing Textual Entailment (RTE) in clinical texts using a semantic analysis and logical inference system, ccg2lambda.However, it is difficult for existing inference systems to correctly determine the entailment relations, if the input sentence contains medical domain specific paraphrases such as disease names.In this study, we propose a method to supplement the equivalence relations of disease names as axioms by identifying candidates for paraphrases that lack in theorem proving.Candidates of paraphrases are identified by using a model for the NER task for disease names and a disease name dictionary.We also construct an inference test set that requires knowledge injection of disease names and evaluate our inference system.Experiments showed that our inference system was able to correctly infer for 106 out of 149 inference test sets.
Do transformer models do phonology like a linguist?

Neural sequence-to-sequence models have been very successful at tasks in phonology and morphology that seemingly require a capacity for intricate linguistic generalisations.In this paper, we perform a detailed breakdown of the power of such models to capture various phonological generalisations and to benefit from exposure to one phonological rule to infer the behaviour of another similar rule.We present two types of experiments, one of which establishes the efficacy of the transformer model on 29 different processes.The second experiment type follows a priming and held-out case split where our model is exposed to two (or more) phenomena; one which is used as a primer to make the model aware of a linguistic category (e.g.voiceless stops) and a second one which contains a rule with a withheld case that the model is expected to infer (e.g.word-final devoicing with a missing training example such as b→p).Our results show that the transformer model can successfully model all 29 phonological phenomena considered, regardless of perceived process difficulty.We also show that the model can generalise linguistic categories and structures, such as vowels and syllables, through priming processes.
Accelerating Transformer Inference for Translation via Parallel Decoding

Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT).The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality.In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction.We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference.This formulation allows to speed up existing models without training or modifications while retaining translation quality.We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources.Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.
Predicting the Quality of Revisions in Argumentative Writing

The ability to revise in response to feedback is critical to students' writing success.In the case of argument writing in specific, identifying whether an argument revision (AR) is successful or not is a complex problem because AR quality is dependent on the overall content of an argument.For example, adding the same evidence sentence could strengthen or weaken existing claims in different argument contexts (ACs).To address this issue we developed Chain-of-Thought prompts to facilitate ChatGPT-generated ACs for AR quality predictions.The experiments on two corpora, our annotated elementary essays and existing college essays benchmark, demonstrate the superiority of the proposed ACs over baselines.
Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers

There exists a wide variety of efficiency methods for natural language processing (NLP) tasks, such as pruning, distillation, dynamic inference, quantization, etc. From a different perspective, we can consider an efficiency method as an operator applied on a model.Naturally, we may construct a pipeline of operators, i.e., to apply multiple efficiency methods on the model sequentially.In this paper, we study the plausibility of this idea, and more importantly, the commutativity and cumulativeness of efficiency operators.We make two interesting observations from our experiments:(1) The operators are commutative-the order of efficiency methods within the pipeline has little impact on the final results; (2) The operators are also cumulative-the final results of combining several efficiency methods can be estimated by combining the results of individual methods.These observations deepen our understanding of efficiency operators and provide useful guidelines for building them in real-world applications.
A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions

We investigate the representation of pretrained language models and humans, using the idea of word definition modeling-how well a word is represented by its definition, and vice versa.Our analysis shows that a word representation in pretrained language models does not successfully map its human-written definition and its usage in example sentences.We then present a simple method DefBERT that integrates pretrained models with word semantics in dictionaries.We show its benefits on newly-proposed tasks of definition ranking and definition sense disambiguation.Furthermore, we present the results on standard word similarity tasks and short text classification tasks where models are required to encode semantics with only a few words.The results demonstrate the effectiveness of integrating word definitions and pretrained language models. 1
Visual Coherence Loss for Coherent and Visually Grounded Story Generation

Local coherence is essential for text generation models.We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters.In this paper, we propose a loss function inspired by a linguistic theory of coherence for learning image sequence representations.We further propose combining features from an object detector and a face detector to construct stronger character features.To evaluate visual grounding that current reference-based metrics do not measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences.Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories.
Zero-Shot Prompting for Implicit Intent Prediction and Recommendation with Commonsense Reasoning

The current generation of intelligent assistants require explicit user requests to perform tasks or services, often leading to lengthy and complex conversations.In contrast, human assistants can infer multiple implicit intents from utterances via their commonsense knowledge, thereby simplifying interactions.To bridge this gap, this paper proposes a framework for multidomain dialogue systems.This framework automatically infers implicit intents from user utterances, and prompts a large pre-trained language model to suggest suitable task-oriented bots.By leveraging commonsense knowledge, our framework recommends associated bots in a zero-shot manner, enhancing interaction efficiency and effectiveness.This approach substantially reduces interaction complexity, seamlessly integrates various domains and tasks, and represents a significant step towards creating more human-like intelligent assistants that can reason about implicit intents, offering a superior user experience. 1
KnowComp at SemEval-2023 Task 7: Fine-tuning Pre-trained Language Models for Clinical Trial Entailment Identification

In this paper, we present our system for the textual entailment identification task as a subtask of the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data.The entailment identification task aims to determine whether a medical statement affirms a valid entailment given a clinical trial premise or forms a contradiction with it.Since the task is inherently a text classification task, we propose a system that performs binary classification given a statement and its associated clinical trial.Our proposed system leverages a human-defined prompt to aggregate the information contained in the statement, section name, and clinical trials.Pre-trained language models are then finetuned on the prompted input sentences to learn to discriminate the inferential relation between the statement and clinical trial.To validate our system, we conduct extensive experiments with a wide variety of pre-trained language models.Our best system is built on DeBERTa-v3-large, which achieves an F1 score of 0.764 and secures the fifth rank in the official leaderboard.Further analysis indicates that leveraging our designed prompt is effective, and our model suffers from a low recall.Our code and pre-trained models are available at https://github.com/
An Empirical Analysis of Leveraging Knowledge for Low-Resource Task-Oriented Semantic Parsing

Task-oriented semantic parsing has drawn a lot of interest from the NLP community, and especially the voice assistant use-cases as it enables representing the meaning of user requests with arbitrarily nested semantics, including multiple intents and compound entities.SOTA models are large seq2seq transformers and require hundreds of thousands of annotated examples to be trained.However annotating such data to bootstrap new domains or languages is expensive and error-prone, especially for requests made of nested semantics.In addition large models easily break the tight latency constraints imposed in a user-facing production environment.As part of this work we explore leveraging external knowledge as a replacement for additional annotated data in order to improve model accuracy in low-resource and low-compute settings.We demonstrate that using knowledgeenhanced encoders inside seq2seq models does not result in performance gains by itself, but multitask learning to uncover entities in addition to the parse generation is a simple yet effective way of improving performance across the domains and data regimes.We show this is especially true in the low-compute low-data setting and for entity-rich domains, with relative gains up to 74.48% in some cases on the TOPv2 dataset.
Balancing Lexical and Semantic Quality in Abstractive Summarization

An important problem of the sequence-tosequence neural models widely used in abstractive summarization is exposure bias.To alleviate this problem, re-ranking systems have been applied in recent years.Despite some performance improvements, this approach remains underexplored.Previous works have mostly specified the rank through the ROUGE score and aligned candidate summaries, but there can be quite a large gap between the lexical overlap metric and semantic similarity.In this paper, we propose a novel training method in which a re-ranker balances the lexical and semantic quality.We further newly define false positives in ranking and present a strategy to reduce their influence.Experiments on the CNN/DailyMail and XSum datasets show that our method can estimate the meaning of summaries without seriously degrading the lexical aspect.More specifically, it achieves an 89.67 BERTScore on the CNN/DailyMail dataset, reaching new state-of-the-art performance.
Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution

Zero-shot cross-lingual transfer is when a multilingual model is trained to perform a task in one language and then is applied to another language.Although the zero-shot cross-lingual transfer approach has achieved success in various classification tasks (Wu and Dredze, 2019), its performance on natural language generation tasks falls short in quality (Rönnqvist et al., 2019;Vu et al., 2022) and sometimes outputs an incorrect language (Xue et al., 2021).In our study, we show that the fine-tuning process learns language invariant representations, which is beneficial for classification tasks but harmful for generation tasks.Motivated by this, we propose a simple method to regularize the model from learning language invariant representations and a method to select model checkpoints without a development set in the target language, both resulting in better generation quality.Experiments on three semantically diverse generation tasks show that our method reduces the accidental translation problem by 68% and improves the ROUGE-L score (Lin, 2004) by 1.5 on average.
AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing

In this paper, we examine the current state-ofthe-art in AMR parsing, which relies on ensemble strategies by merging multiple graph predictions.Our analysis reveals that the present models often violate AMR structural constraints.To address this issue, we develop a validation method, and show how ensemble models can exploit SMATCH metric weaknesses to obtain higher scores, but sometimes result in corrupted graphs.Additionally, we highlight the demanding need to compute the SMATCH score among all possible predictions.To overcome these challenges, we propose two novel ensemble strategies based on Transformer models, improving robustness to structural constraints, while also reducing the computational time.Our methods provide new insights for enhancing AMR parsers and metrics.Our code is available at github.com/babelscape/AMRs-Assemble.
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity

Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings, and typically display outlier dimensions.This seems to be true for both monolingual and multilingual models, although much less work has been done on the multilingual context.Why these outliers occur and how they affect the representations is still an active area of research.We investigate outlier dimensions and their relationship to anisotropy in multiple pre-trained multilingual language models.We focus on cross-lingual semantic similarity tasks, as these are natural tasks for evaluating multilingual representations.Specifically, we examine sentence representations.Sentence transformers which are fine-tuned on parallel resources (that are not always available) perform better on this task, and we show that their representations are more isotropic.However, we aim to improve multilingual representations in general.We investigate how much of the performance difference can be made up by only transforming the embedding space without fine-tuning, and visualise the resulting spaces.We test different operations: Removing individual outlier dimensions, cluster-based isotropy enhancement, and ZCA whitening.We publish our code for reproducibility.
A Set Prediction Network For Extractive Summarization

Extractive summarization focuses on extracting salient sentences from the source document and incorporating them in the summary without changing their wording or structure.The naive approach for extractive summarization is sentence classification, which makes independent binary decisions for each sentence, resulting in the model cannot detect the dependencies between sentences in the summary.Recent approaches introduce an autoregressive decoder to detect redundancy relationship between sentences by step-by-step sentence selection, but bring train-inference gap.To address these issues, we formulate extractive summarization as a salient sentence set recognition task.To solve the sentence set recognition task, we propose a set prediction network (SetSum), which sets up a fixed set of learnable queries to extract the entire sentence set of the summary, while capturing the dependencies between them.Different from previous methods with an auto-regressive decoder, we employ a non-autoregressive decoder to predict the sentences within the summary in parallel during both the training and inference process, which eliminates the traininference gap.Experimental results on both single-document and multi-document extracted summary datasets show that our approach outperforms previous state-of-the-art models.
Rogue Scores

Correct, comparable, and reproducible model evaluation is essential for progress in machine learning. Over twenty years, thousands of language and vision models have been evaluated with a popular metric called ROUGE. Does this widespread benchmark metric meet these three evaluation criteria? This systematic review of over two thousand publications using ROUGE finds: (A) Critical evaluation decisions and parameters are routinely omitted, making most reported scores irreproducible. (B) Differences in evaluation protocol are common, affect scores, and impact the comparability of results reported in many papers. (C) Thousands of papers use nonstandard evaluation packages with software defects that produce provably incorrect scores. Estimating the overall impact of these findings is difficult: because software citations are rare, it is nearly impossible to distinguish between correct ROUGE scores and incorrect “rogue scores.”