Reviewerly/exHarmony huggingface dataset:
Discourse Information for Document-Level Temporal Dependency Parsing

In this study, we examine the benefits of incorporating discourse information into documentlevel temporal dependency parsing.Specifically, we evaluate the effectiveness of integrating both high-level discourse profiling information, which describes the discourse function of sentences, and surface-level sentence position information into temporal dependency graph (TDG) parsing.Unexpectedly, our results suggest that simple sentence position information, particularly when encoded using our novel sentence-position embedding method, performs the best, perhaps because it does not rely on noisy model-generated feature inputs.Our proposed system surpasses the current state-of-the-art TDG parsing systems in performance.Furthermore, we aim to broaden the discussion on the relationship between temporal dependency parsing and discourse analysis, given the substantial similarities shared between the two tasks.We argue that discourse analysis results should not be merely regarded as an additional input feature for temporal dependency parsing.Instead, adopting advanced discourse analysis techniques and research insights can lead to more effective and comprehensive approaches to temporal information extraction tasks.
Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing

Interactive semantic parsing based on natural language (NL) feedback, where users provide feedback to correct the parser mistakes, has emerged as a more practical scenario than the traditional one-shot semantic parsing.However, prior work has heavily relied on humanannotated feedback data to train the interactive semantic parser, which is prohibitively expensive and not scalable.In this work, we propose a new task of simulating NL feedback for interactive semantic parsing.We accompany the task with a novel feedback evaluator.The evaluator is specifically designed to assess the quality of the simulated feedback, based on which we decide the best feedback simulator from our proposed variants.On a text-to-SQL dataset, we show that our feedback simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser.In low-data settings, our feedback simulator can help achieve comparable error correction performance as trained using the costly, full set of human annotations.
MISGENDERED: Limits of Large Language Models in Understanding Pronouns

Content Warning: This paper contains examples of misgendering and erasure that could be offensive and potentially triggering.Gender bias in language technologies has been widely studied, but research has mostly been restricted to a binary paradigm of gender.It is essential also to consider non-binary gender identities, as excluding them can cause further harm to an already marginalized group.In this paper, we comprehensively evaluate popular language models for their ability to correctly use English gender-neutral pronouns (e.g., singular they, them) and neo-pronouns (e.g., ze, xe, thon) that are used by individuals whose gender identity is not represented by binary pronouns.We introduce MISGENDERED, a framework for evaluating large language models' ability to correctly use preferred pronouns, consisting of (i) instances declaring an individual's pronoun, followed by a sentence with a missing pronoun, and (ii) an experimental setup for evaluating masked and auto-regressive language models using a unified method.When prompted outof-the-box, language models perform poorly at correctly predicting neo-pronouns (averaging 7.6% accuracy) and gender-neutral pronouns (averaging 31.0%accuracy).This inability to generalize results from a lack of representation of non-binary pronouns in training data and memorized associations.Few-shot adaptation with explicit examples in the prompt improves the performance but plateaus at only 45.4% for neo-pronouns.We release the full dataset, code, and demo at https://tamannahossainkay. github.io/misgendered/.
Tagged End-to-End Simultaneous Speech Translation Training Using Simultaneous Interpretation Data

Simultaneous speech translation (SimulST) translates partial speech inputs incrementally.Although the monotonic correspondence between input and output is preferable for smaller latency, it is not the case for distant language pairs such as English and Japanese.A prospective approach to this problem is to mimic simultaneous interpretation (SI) using SI data to train a SimulST model.However, the size of such SI data is limited, so the SI data should be used together with ordinary bilingual data whose translations are given in offline.In this paper, we propose an effective way to train a SimulST model using mixed data of SI and offline.The proposed method trains a single model using the mixed data with style tags that tell the model to generate SI-or offline-style outputs.Experiment results show improvements of BLEURT in different latency ranges, and our analyses revealed the proposed model generates SI-style outputs more than the baseline.
MLModeler5 at SemEval-2023 Task 3: Detecting the Category and the Framing Techniques in Online News in a Multi-lingual Setup

Throughout the ages, we have seen various leaders, dictators, politicians, CEOs, companies, sports agencies, organizations like the UN harness the power of news articles to shape and influence the minds of the people of the world.Various persuasion and framing techniques are employed while writing articles to achieve this aim.It has become necessary now, more so than ever, to identify these techniques and strategies to help understand these articles, reduce the spread of fake news, stop propaganda stories and ensure that people of the world are well informed.To help achieve these tasks, we have implemented RoBERTa and AL-BERT models for Subtask 1/2 Task 3 SemEval-2023.Both these models are BERT-based architectures.The results of these models are validated on the dataset provided by the organizers of this task.
MTCue: Learning Zero-Shot Control of Extra-Textual Attributes by Leveraging Unstructured Context in Neural Machine Translation

Efficient utilisation of both intra- and extra-textual context remains one of the critical gaps between machine and human translation. Existing research has primarily focused on providing individual, well-defined types of context in translation, such as the surrounding text or discrete external variables like the speaker’s gender. This work introduces MTCue, a novel neural machine translation (NMT) framework that interprets all context (including discrete variables) as text. MTCue learns an abstract representation of context, enabling transferability across different data settings and leveraging similar attributes in low-resource scenarios. With a focus on a dialogue domain with access to document and metadata context, we extensively evaluate MTCue in four language pairs in both translation directions. Our framework demonstrates significant improvements in translation quality over a parameter-matched non-contextual baseline, as measured by BLEU (+0.88) and Comet (+1.58). Moreover, MTCue significantly outperforms a “tagging” baseline at translating English text. Analysis reveals that the context encoder of MTCue learns a representation space that organises context based on specific attributes, such as formality, enabling effective zero-shot control. Pre-training on context embeddings also improves MTCue’s few-shot performance compared to the “tagging” baseline. Finally, an ablation study conducted on model components and contextual variables further supports the robustness of MTCue for context-based NMT.
RFiD: Towards Rational Fusion-in-Decoder for Open-Domain Question Answering

Open-Domain Question Answering (ODQA) systems necessitate a reader model capable of generating answers by simultaneously referring to multiple passages.Although representative models like Fusion-in-Decoder (FiD) have been proposed to address this challenge, these systems can inadvertently rely on spurious features instead of genuine causal relationships between the question and the passages to generate answers.To counter this problem, we introduce the Rational Fusion-in-Decoder (RFiD) model.Our model leverages the encoders of FiD to differentiate between causal relationships and spurious features, subsequently guiding the decoder to generate answers informed by this discernment.Experimental results on two ODQA datasets, Natural Questions (NQ) and TriviaQA (TQ), demonstrate that our model surpasses previous methods, achieving improvements of up to 1.5 and 0.7 in Exact Match scores on NQ, and exhibits an enhanced ability to identify causal relationships.
Comparing Neural Question Generation Architectures for Reading Comprehension

In recent decades, there has been a significant push to leverage technology to aid both teachers and students in the classroom.Language processing advancements have been harnessed to provide better tutoring services, automated feedback to teachers, improved peer-to-peer feedback mechanisms, and measures of student comprehension for reading.Automated question generation systems have the potential to significantly reduce teachers' workload in the latter.In this paper, we compare three different neural architectures for question generation across two types of reading material: narratives and textbooks.For each architecture, we explore the benefits of including question attributes in the input representation.Our models show that a T5 architecture has the best overall performance, with a RougeL score of 0.536 on a narrative corpus and 0.316 on a textbook corpus.We break down the results by attribute and discover that the attribute can improve the quality of some types of generated questions, including Action and Character, but this is not true for all models.
Emergent Modularity in Pre-trained Transformers

This work examines the presence of modularity in pre-trained Transformers, a feature commonly found in human brains and thought to be vital for general intelligence.In analogy to human brains, we consider two main characteristics of modularity: (1) functional specialization of neurons: we evaluate whether each neuron is mainly specialized in a certain function, and find that the answer is yes.(2) function-based neuron grouping: we explore finding a structure that groups neurons into modules by function, and each module works for its corresponding function.Given the enormous amount of possible structures, we focus on Mixture-of-Experts as a promising candidate, which partitions neurons into experts and usually activates different experts for different inputs.Experimental results show that there are functional experts, where clustered are the neurons specialized in a certain function.Moreover, perturbing the activations of functional experts significantly affects the corresponding function.Finally, we study how modularity emerges during pre-training, and find that the modular structure is stabilized at the early stage, which is faster than neuron stabilization.It suggests that Transformers first construct the modular structure and then learn fine-grained neuron functions.Our code and data are available at https://github.com/THUNLP/modularity-analysis.
Towards Parameter-Efficient Integration of Pre-Trained Language Models In Temporal Video Grounding

This paper explores the task of Temporal Video Grounding (TVG) where, given an untrimmed video and a natural language sentence query, the goal is to recognize and determine temporal boundaries of action instances in the video described by the query.Recent works tackled this task by improving query inputs with large pre-trained language models (PLM) at the cost of more expensive training.However, the effects of this integration are unclear, as these works also propose improvements in the visual inputs.Therefore, this paper studies the effects of PLMs in TVG and assesses the applicability of parameter-efficient training with NLP adapters.We couple popular PLMs with a selection of existing approaches and test different adapters to reduce the impact of the additional parameters.Our results on three challenging datasets show that, without changing the visual inputs, TVG models greatly benefited from the PLM integration and fine-tuning, stressing the importance of sentence query representation in this task.Furthermore, NLP adapters were an effective alternative to full fine-tuning, even though they were not tailored to our task, allowing PLM integration in larger TVG models and delivering results comparable to SOTA models.Finally, our results shed light on which adapters work best in different scenarios.
Back-Transliteration of English Loanwords in Japanese

We propose methods for transliterating English loanwords in Japanese from their Japanese written form (katakana/romaji) to their original English written form. Our data is a Japanese-English loanwords dictionary that we have created ourselves. We employ two approaches: direct transliteration, which directly converts words from katakana to English, and indirect transliteration, which utilizes the English pronunciation as a means to convert katakana words into their corresponding English sound representations, which are subsequently converted into English words. Additionally, we compare the effectiveness of using katakana versus romaji as input characters. We develop 6 models of 2 types for our experiments: one with an English lexicon-filter, and the other without. For each type, we built 3 models, including a pair n-gram based on WFSTs and two sequence-to-sequence models leveraging LSTM and transformer. Our best performing model was the pair n-gram model with a lexicon-filter, directly transliterating from katakana to English.
Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution

Although recent neural models for coreference resolution have led to substantial improvements on benchmark datasets, transferring these models to new target domains containing out-of-vocabulary spans and requiring differing annotation schemes remains challenging.Typical approaches involve continued training on annotated target-domain data, but obtaining annotations is costly and time-consuming.We show that annotating mentions alone is nearly twice as fast as annotating full coreference chains.Accordingly, we propose a method for efficiently adapting coreference models, which includes a high-precision mention detection objective and requires annotating only mentions in the target domain.Extensive evaluation across three English coreference datasets: CoNLL-2012 (news/conversation), i2b2/VA (medical notes), and previously unstudied child welfare notes, reveals that our approach facilitates annotation-efficient transfer and results in a 7-14% improvement in average F1 without increasing annotator time 1 .
Multilingual Sequence-to-Sequence Models for Hebrew NLP

Recent work attributes progress in NLP to large language models (LMs) with increased model size and large quantities of pretraining data.Despite this, current state-of-the-art LMs for Hebrew are both under-parameterized and under-trained compared to LMs in other languages.Additionally, previous work on pretrained Hebrew LMs focused on encoderonly models.While the encoder-only architecture is beneficial for classification tasks, it does not cater well for sub-word prediction tasks, such as Named Entity Recognition, when considering the morphologically rich nature of Hebrew.In this paper we argue that sequence-to-sequence generative architectures are more suitable for large LMs in morphologically rich languages (MRLs) such as Hebrew.We demonstrate this by casting tasks in the Hebrew NLP pipeline as text-to-text tasks, for which we can leverage powerful multilingual, pretrained sequence-to-sequence models as mT5, eliminating the need for a separate, specialized, morpheme-based, decoder.Using this approach, our experiments show substantial improvements over previously published results on all existing Hebrew NLP benchmarks.These results suggest that multilingual sequence-to-sequence models present a promising building block for NLP for MRLs.
Generating Dialog Responses with Specified Grammatical Items for Second Language Learning

This paper proposes a new second language learning task of generating a response including specified grammatical items.We consider two approaches: 1) fine-tuning a pre-trained language model (DialoGPT) by reinforcement learning and 2) providing a few-shot prompt to a large language model (GPT-3).For reinforcement learning, we examine combinations of three reward functions that consider grammatical items, diversity, and fluency.Our experiments confirm that both approaches can generate responses including the specified grammatical items and that it is crucial to consider fluency rather than diversity as the reward function.
Multi-hop Evidence Retrieval for Cross-document Relation Extraction

Relation Extraction (RE) has been extended to cross-document scenarios because many relations are not simply described in a single document.This inevitably brings the challenge of efficient open-space evidence retrieval to support the inference of cross-document relations, along with the challenge of multi-hop reasoning on top of entities and evidence scattered in an open set of documents.To combat these challenges, we propose MR.COD (Multi-hop evidence retrieval for Cross-document relation extraction), which is a multi-hop evidence retrieval method based on evidence path mining and ranking.We explore multiple variants of retrievers to show evidence retrieval is essential in cross-document RE.We also propose a contextual dense retriever for this setting.Experiments on CodRED show that evidence retrieval with MR.COD effectively acquires crossdocument evidence and boosts end-to-end RE performance in both closed and open settings. 1
Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction

Task-oriented dialogues often require agents to enact complex, multi-step procedures in order to meet user requests.While large language models have found success automating these dialogues in constrained environments, their widespread deployment is limited by the substantial quantities of task-specific data required for training.The following paper presents a data-efficient solution to constructing dialogue systems, leveraging explicit instructions derived from agent guidelines, such as company policies or customer service manuals.Our proposed Knowledge-Augmented Dialogue System (KADS) combines a large language model with a knowledge retrieval module that pulls documents outlining relevant procedures from a predefined set of policies, given a user-agent interaction.To train this system, we introduce a semi-supervised pre-training scheme that employs dialogue-document matching and actionoriented masked language modeling with partial parameter freezing.We evaluate the effectiveness of our approach on prominent taskoriented dialogue datasets, Action-Based Conversations Dataset and Schema-Guided Dialogue, for two dialogue tasks: action state tracking and workflow discovery.Our results demonstrate that procedural knowledge augmentation improves accuracy predicting inand out-of-distribution actions while preserving high performance in settings with low or sparse data.
DiscoFlan: Instruction Fine-tuning and Refined Text Generation for Discourse Relation Label Classification

This paper introduces DiscoFlan, our system for the DISRPT 2023 shared task on discourse relation classification.We leverage recent advances in NLP finetuning and use Flan-T5 as a multilingual discourse relation classifier.Our model uses multilingual instructional prompts to finetune on datasets from different languages and generate relation labels as classification outputs.The model's hyperparameters are tuned to enable efficient label generation by finetuning on low-resource datasets.Moreover, we introduce a post-processing step to tackle the problem of label mismatches caused by the generative nature of a seq2seq model by using the label distribution.In contrast to the previous state-of-the-art model, our approach eliminates the need for hand-crafted features in computing the discourse relation classes.Overall, DiscoFlan showcases how instruction finetuning can perform multilingual discourse relation classification for the DISRPT 2023 discourse relation classification shared task.
Sampling Better Negatives for Distantly Supervised Named Entity Recognition

Distantly supervised named entity recognition (DS-NER) has been proposed to exploit the automatically labeled training data instead of human annotations.The distantly annotated datasets are often noisy and contain a considerable number of false negatives.The recent approach uses a weighted sampling approach to select a subset of negative samples for training.However, it requires a good classifier to assign weights to the negative samples.In this paper, we propose a simple and straightforward approach for selecting the top negative samples that have high similarities with all the positive samples for training.Our method achieves consistent performance improvements on four distantly supervised NER datasets.Our analysis also shows that it is critical to differentiate the true negatives from the false negatives. 1
EEL: Efficiently Encoding Lattices for Reranking

Standard decoding approaches for conditional text generation tasks typically search for an output hypothesis with high model probability, but this may not yield the best hypothesis according to human judgments of quality.Reranking to optimize for downstream metrics can better optimize for quality, but many metrics of interest are computed with pre-trained language models, which are slow to apply to large numbers of hypotheses.We explore an approach for reranking hypotheses by using Transformers to efficiently encode lattices of generated outputs, a method we call EEL.With a single Transformer pass over the entire lattice, we can approximately compute a contextualized representation of each token as if it were only part of a single hypothesis in isolation.We combine this approach with a new class of token-factored rerankers (TFRs) that allow for efficient extraction of high reranker-scoring hypotheses from the lattice.Empirically, our approach incurs minimal degradation error compared to the exponentially slower approach of encoding each hypothesis individually.When applying EEL with TFRs across three text generation tasks, our results show both substantial speedup compared to naive reranking and often better performance on downstream metrics than comparable approaches.1
teamPN at SemEval-2023 Task 1: Visual Word Sense Disambiguation Using Zero-Shot MultiModal Approach

Visual Word Sense Disambiguation shared task at SemEval-2023 aims to identify an image corresponding to the intended meaning of a given ambiguous word (with related context) from a set of candidate images.The lack of textual description for the candidate image and the corresponding word's ambiguity makes it a challenging problem.This paper describes teamPN's multi-modal and modular approach to solving this in English track of the task.We efficiently used recent multi-modal pre-trained models backed by real-time multi-modal knowledge graphs to augment textual knowledge for the images and select the best matching image accordingly.We outperformed the baseline model by 5 points and proposed a unique approach that can further work as a framework for other modular and knowledge-backed solutions.