Reviewerly/exHarmony huggingface dataset:
SPARTA at CASE 2021 Task 1: Evaluating Different Techniques to Improve Event Extraction

We participated in the Shared Task 1 at CASE 2021, Subtask 4 on protest event extraction from news articles (Hürriyetoglu et al., 2022) and examined different techniques aimed at improving the performance of the winning system from the last competition round (Hürriyetoglu et al., 2021).We evaluated in-domain pretraining, task-specific pre-fine-tuning, alternative loss function, translation of the English training dataset into other target languages (i.e., Portuguese, Spanish, and Hindi) for the token classification task, and a simple data augmentation technique by random sentence reordering.This paper summarizes the results, showing that random sentence reordering leads to a consistent improvement of the model performance.
How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts.

Neural Machine Translation systems built on top of Transformer-based architectures are routinely improving the state-of-the-art in translation quality according to word-overlap metrics.However, a growing number of studies also highlight the inherent gender bias that these models incorporate during training, which reflects poorly in their translations.In this work, we investigate whether these models can be instructed to fix their bias during inference using targeted, guided instructions as contexts.By translating relevant contextual sentences during inference along with the input, we observe large improvements in reducing the gender bias in translations, across three popular test suites (WinoMT, BUG, SimpleGen).We further propose a novel metric to assess several large pretrained models (OPUS-MT, M2M-100) on their sensitivity towards using contexts during translation to correct their biases.Our approach requires no fine-tuning and thus can be used easily in production systems to de-bias translations from stereotypical gender-occupation bias 1 .We hope our method, along with our metric, can be used to build better, bias-free translation systems.
CORT: A New Baseline for Comparative Opinion Classification by Dual Prompts

Comparative opinion is a common linguistic phenomenon.The opinion is expressed by comparing multiple targets on a shared aspect, e.g., "camera A is better than camera B in picture quality".Among the various subtasks in opinion mining, comparative opinion classification is relatively less studied.Current solutions use rules or classifiers to identify opinions, i.e., better, worse, or same, through feature engineering.Because the features are directly derived from the input sentence, these solutions are sensitive to the order of the targets mentioned in the sentence.For example, "camera A is better than camera B" means the same as "camera B is worse than camera A"; but the features of these two sentences are completely different.In this paper, we approach comparative opinion classification through prompt learning, taking the advantage of embedded knowledge in pre-trained language model.We design a twin framework with dual prompts, named CORT.This extremely simple model delivers state-ofthe-art and robust performance on all benchmark datasets for comparative opinion classification.We believe CORT well serves as a new baseline for comparative opinion classification.
Synergy with Translation Artifacts for Training and Inference in Multilingual Tasks

Translation has played a crucial role in improving the performance on multilingual tasks: (1) to generate the target language data from the source language data for training and (2) to generate the source language data from the target language data for inference.However, prior works have not considered the use of both translations simultaneously.This paper shows that combining them can synergize the results on various multilingual sentence classification tasks.We empirically find that translation artifacts stylized by translators are the main factor of the performance gain.Based on this analysis, we adopt two training methods, SupCon and MixUp, considering translation artifacts.Furthermore, we propose a crosslingual fine-tuning algorithm called MUSC, which uses SupCon and MixUp jointly and improves the performance.Our code is available at https://github.com/jongwooko/MUSC.
Measuring Context-Word Biases in Lexical Semantic Datasets

State-of-the-art pretrained contextualized models (PCM) eg.BERT use tasks such as WiC and WSD to evaluate their wordin-context representations.This inherently assumes that performance in these tasks reflect how well a model represents the coupled word and context semantics.We question this assumption by presenting the first quantitative analysis on the context-word interaction being tested in major contextual lexical semantic tasks.To achieve this, we run probing baselines on masked input, and propose measures to calculate and visualize the degree of context or word biases in existing datasets.The analysis was performed on both models and humans.Our findings demonstrate that models are usually not being tested for word-in-context semantics in the same way as humans are in these tasks, which helps us better understand the model-human gap.Specifically, to PCMs, most existing datasets fall into the extreme ends (the retrieval-based tasks exhibit strong target word bias while WiC-style tasks and WSD show strong context bias); In comparison, humans are less biased and achieve much better performance when both word and context are available than with masked input.We recommend our framework for understanding and controlling these biases for model interpretation and future task design.
MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective

Teaching neural models to generate narrative coherent texts is a critical problem.Recent pretrained language models have achieved promising results, but there is still a gap between human written texts and machine-generated outputs.In this work, we propose a novel multitask training strategy for coherent text generation grounded on the cognitive theory of writing, which empowers the model to learn essential subskills needed for writing including planning and reviewing besides end-to-end generation.We extensively evaluate our model on three open-ended generation tasks including story generation, news article writing and argument generation.Experiments show that our model achieves better results on both few-shot and fully-supervised settings than strong baselines, and human evaluations confirm that our model can generate more coherent outputs.
Subword Segmental Language Modelling for Nguni Languages

Subwords have become the standard units of text in NLP, enabling efficient open-vocabulary models.With algorithms like byte-pair encoding (BPE), subword segmentation is viewed as a preprocessing step applied to the corpus before training.This can lead to sub-optimal segmentations for low-resource languages with complex morphologies.We propose a subword segmental language model (SSLM) that learns how to segment words while being trained for autoregressive language modelling.By unifying subword segmentation and language modelling, our model learns subwords that optimise LM performance.We train our model on the 4 Nguni languages of South Africa.These are low-resource agglutinative languages, so subword information is critical.As an LM, SSLM outperforms existing approaches such as BPE-based models on average across the 4 languages.Furthermore, it outperforms standard subword segmenters on unsupervised morphological segmentation.We also train our model as a word-level sequence model, resulting in an unsupervised morphological segmenter that outperforms existing methods by a large margin for all 4 languages.Our results show that learning subword segmentation is an effective alternative to existing subword segmenters, enabling the model to discover morpheme-like subwords that improve its LM capabilities.
Improving Low-Resource Languages in Pre-Trained Multilingual Language Models

Pre-trained multilingual language models are the foundation of many NLP approaches, including cross-lingual transfer solutions.However, languages with small available monolingual corpora are often not well-supported by these models leading to poor performance.We propose an unsupervised approach to improve the cross-lingual representations of lowresource languages by bootstrapping word translation pairs from monolingual corpora and using them to improve language alignment in pretrained language models.We perform experiments on nine languages, using contextual word retrieval and zero-shot named entity recognition to measure both intrinsic cross-lingual word representation quality and downstream task performance, showing improvements on both tasks.Our results show that it is possible to improve pre-trained multilingual language models by relying only on non-parallel resources.
Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality

A recent line of work in NLP focuses on the (dis)ability of models to generalise compositionally for artificial languages.However, when considering natural language tasks, the data involved is not strictly, or locally, compositional.Quantifying the compositionality of data is a challenging task, which has been investigated primarily for short utterances.We use recursive neural models (Tree-LSTMs) with bottlenecks that limit the transfer of information between nodes.We illustrate that comparing data's representations in models with and without the bottleneck can be used to produce a compositionality metric.The procedure is applied to the evaluation of arithmetic expressions using synthetic data, and sentiment classification using natural language data.We demonstrate that compression through a bottleneck impacts non-compositional examples disproportionately and then use the bottleneck compositionality metric (BCM) to distinguish compositional from non-compositional samples, yielding a compositionality ranking over a dataset.
Eeny, meeny, miny, moe. How to choose data for morphological inflection.

Data scarcity is a widespread problem in numerous natural language processing (NLP) tasks for low-resource languages.Within morphology, the labour-intensive work of tagging/glossing data is a serious bottleneck for both NLP and language documentation.Active learning (AL) aims to reduce the cost of data annotation by selecting data that is most informative for improving the model.In this paper, we explore four sampling strategies for the task of morphological inflection using a Transformer model: a pair of oracle experiments where data is chosen based on whether the model already can or cannot inflect the test forms correctly, as well as strategies based on high/low model confidence, entropy, as well as random selection.We investigate the robustness of each strategy across 30 typologically diverse languages.We also perform a more in-depth case study of Natügu.Our results show a clear benefit to selecting data based on model confidence and entropy.Unsurprisingly, the oracle experiment, where only incorrectly handled forms are chosen for further training, which is presented as a proxy for linguist/language consultant feedback, shows the most improvement.This is followed closely by choosing low-confidence and high-entropy predictions.We also show that despite the conventional wisdom of larger data sets yielding better accuracy, introducing more instances of high-confidence or low-entropy forms, or forms that the model can already inflect correctly, can reduce model performance.
Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations

Extracting entities, events, event arguments, and relations (i.e., task instances) from text represents four main challenging tasks in information extraction (IE), which have been solved jointly (JointIE) to boost the overall performance for IE.As such, previous work often leverages two types of dependencies between the tasks, i.e., cross-instance and cross-type dependencies representing relatedness between task instances and correlations between information types of the tasks.However, the crosstask dependencies in prior work are not optimal as they are only designed manually according to some task heuristics.To address this issue, we propose a novel model for JointIE that aims to learn cross-task dependencies from data.In particular, we treat each task instance as a node in a dependency graph where edges between the instances are inferred through information from different layers of a pretrained language model (e.g., BERT).Furthermore, we utilize the Chow-Liu algorithm to learn a dependency tree between information types for JointIE by seeking to approximate the joint distribution of the types from data.Finally, the Chow-Liu dependency tree is used to generate cross-type patterns, serving as anchor knowledge to guide the learning of representations and dependencies between instances for JointIE.Experimental results show that our proposed model significantly outperforms strong JointIE baselines over four datasets with different languages.
The Curious Case of Control

Children acquiring English make systematic errors on subject control sentences even after they have reached near-adult competence (Chomsky, 1969), possibly due to heuristics based on semantic roles (Maratsos, 1974).Given the advanced fluency of large generative language models, we ask whether model outputs are consistent with these heuristics, and to what degree different models are consistent with each other.We find that models can be categorized by behavior into three separate groups, with broad differences between the groups.The outputs of models in the largest group are consistent with positional heuristics that succeed on subject control but fail on object control.This result is surprising, given that object control is orders of magnitude more frequent in the text data used to train such models.We examine to what degree the models are sensitive to prompting with agent-patient information, finding that raising the salience of agent and patient relations results in significant changes in the outputs of most models.Based on this observation, we leverage an existing dataset of semantic protorole annotations (White et al., 2020) to explore the connections between control and labeling event participants with properties typically associated with agents and patients. 1You will be given a context and a question.Answer the question with either " " or " ".
A Report on the FigLang 2022 Shared Task on Understanding Figurative Language

We present the results of the Shared Task on Understanding Figurative Language that we conducted as a part of the 3rd Workshop on Figurative Language Processing (FigLang 2022) at EMNLP 2022.The shared task is based on the FLUTE dataset (Chakrabarty et al., 2022), which consists of NLI pairs containing figurative language along with free text explanations for each NLI instance.The task challenged participants to build models that are able to not only predict the right label for a figurative NLI instance, but also generate a convincing freetext explanation.The participants were able to significantly improve upon provided baselines in both automatic and human evaluation settings.We further summarize the submitted systems and discuss the evaluation results.
Rethinking Style Transformer with Energy-based Interpretation: Adversarial Unsupervised Style Transfer using a Pretrained Model

Style control, content preservation, and fluency determine the quality of text style transfer models.To train on a nonparallel corpus, several existing approaches aim to deceive the style discriminator with an adversarial loss.However, adversarial training significantly degrades fluency compared to the other two metrics.In this work, we explain this phenomenon using energy-based interpretation, and leverage a pretrained language model to improve fluency.Specifically, we propose a novel approach which applies the pretrained language model to the text style transfer framework by restructuring the discriminator and the model itself, allowing the generator and the discriminator to also take advantage of the power of the pretrained model.We evaluated our model on three public benchmarks GYAFC, Amazon, and Yelp and achieved state-of-the-art performance on the overall metrics.
Improving Zero-Shot Event Extraction via Sentence Simplification

The success of sites such as ACLED and Our World in Data have demonstrated the massive utility of extracting events in structured formats from large volumes of textual data in the form of news, social media, blogs and discussion forums.Event extraction can provide a window into ongoing geopolitical crises and yield actionable intelligence.In this work, we cast socio-political conflict event extraction as a machine reading comprehension (MRC) task.In this approach, extraction of socio-political actors and targets from a sentence is framed as an extractive questionanswering problem conditioned on an event type.There are several advantages of using MRC for this task including the ability to leverage large pretrained multilingual language models and their ability to perform zero-shot extraction.Moreover, we find that the problem of longrange dependencies, i.e., large lexical distance between trigger and argument words and the difficulty of processing syntactically complex sentences plague MRC-based approaches.To address this, we present a general approach to improve the performance of MRC-based event extraction by performing unsupervised sentence simplification guided by the MRC model itself.We evaluate our approach on the ICEWS geopolitical event extraction dataset, with specific attention to 'Actor' and 'Target' argument roles.We show how such context simplification can improve the performance of MRC-based event extraction by more than 5% for actor extraction and more than 10% for target extraction.
KPDROP: Improving Absent Keyphrase Generation

Keyphrase generation is the task of generating phrases (keyphrases) that summarize the main topics of a given document.Keyphrases can be either present or absent from the given document.While the extraction of present keyphrases has received much attention in the past, only recently a stronger focus has been placed on the generation of absent keyphrases.However, generating absent keyphrases is challenging; even the best methods show only a modest degree of success.In this paper, we propose a model-agnostic approach called keyphrase dropout (or KPDROP) to improve absent keyphrase generation.In this approach, we randomly drop present keyphrases from the document and turn them into artificial absent keyphrases during training.We test our approach extensively and show that it consistently improves the absent performance of strong baselines in both supervised and resourceconstrained semi-supervised settings 1 .
Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs

Can we teach natural language understanding models to track their beliefs through intermediate points in text?We propose a representation learning framework called breakpoint modeling that allows for learning of this type.Given any text encoder and data marked with intermediate states (breakpoints) along with corresponding textual queries viewed as true/false propositions (i.e., the candidate beliefs of a model, consisting of information changing through time) our approach trains models in an efficient and end-to-end fashion to build intermediate representations that facilitate teaching and direct querying of beliefs at arbitrary points alongside solving other end tasks.To show the benefit of our approach, we experiment with a diverse set of NLU tasks including relational reasoning on CLUTRR and narrative understanding on bAbI.Using novel belief prediction tasks for both tasks, we show the benefit of our main breakpoint transformer, based on T5, over conventional representation learning approaches in terms of processing efficiency, prediction accuracy and prediction consistency, all with minimal to no effect on corresponding QA endtasks.To show the feasibility of incorporating our belief tracker into more complex reasoning pipelines, we also obtain SOTA performance on the three-tiered reasoning challenge for the TRIP benchmark (around 23-32% absolute improvement on Tasks 2-3). 1
It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New SemAntoNeg Benchmark

We investigate to what extent a hundred publicly available, popular neural language models capture meaning systematically.Sentence embeddings obtained from pretrained or finetuned language models can be used to perform particular tasks, such as paraphrase detection, semantic textual similarity assessment or natural language inference.Common to all of these tasks is that paraphrastic sentences, that is, sentences that carry (nearly) the same meaning, should have (nearly) the same embeddings regardless of surface form.We demonstrate that performance varies greatly across different language models when a specific type of meaning-preserving transformation is applied: two sentences should be identified as paraphrastic if one of them contains a negated antonym in relation to the other one, such as I am not guilty versus I am innocent.We introduce and release SemAntoNeg, a new test suite containing 3152 entries for probing paraphrasticity in sentences incorporating negation and antonyms.Among other things, we show that language models fine-tuned for natural language inference outperform other types of models, especially the ones fine-tuned to produce general-purpose sentence embeddings, on the test suite.Furthermore, we show that most models designed explicitly for paraphrasing are rather mediocre in our task.
Combinatory Grammar Tells Underlying Relevance among Entities

Relation extraction (RE) is an important task in natural language processing which aims to annotate the relation between two given entities, which requires a deep understanding of the running text.To import model performance, existing approaches leverage syntactic information to facilitate the relation extraction process, where they mainly focus on dependencies among words while paying limited attention to other types of syntactic structure.Considering that combinatory categorial grammar (CCG) is a lexicalized grammatical formalism that carries the syntactic and semantic knowledge for text understanding, we propose an alternative solution for RE that takes advantage of CCG to detect the relation between entities.In doing so, we perform a multi-task learning process to learn from RE and autoannotated CCG supertags, where an attention mechanism is performed over all input words to distinguish the important ones for RE with the attention weights guided by the supertag decoding process.We evaluate our model on two widely used English benchmark datasets (i.e., ACE2005EN and SemEval 2010 Task 8 datasets) for RE, where the effectiveness of our approach is demonstrated by the experimental results with our approach achieving state-ofthe-art performance on both datasets.