Reviewerly/exHarmony huggingface dataset:
Improving Long Context Document-Level Machine Translation

Document-level context for neural machine translation (NMT) is crucial to improve the translation consistency and cohesion, the translation of ambiguous inputs, as well as several other linguistic phenomena.Many works have been published on the topic of document-level NMT, but most restrict the system to only local context, typically including just the one or two preceding sentences as additional information.This might be enough to resolve some ambiguous inputs, but it is probably not sufficient to capture some document-level information like the topic or style of a conversation.When increasing the context size beyond just the local context, there are two challenges: (i) the memory usage increases exponentially (ii) the translation performance starts to degrade.We argue that the widely-used attention mechanism is responsible for both issues.Therefore, we propose a constrained attention variant that focuses the attention on the most relevant parts of the sequence, while simultaneously reducing the memory consumption.For evaluation, we utilize targeted test sets in combination with novel evaluation techniques to analyze the translations in regards to specific discourserelated phenomena.We find that our approach is a good compromise between sentence-level NMT vs attending to the full context, especially in low resource scenarios.
UMUTeam at SemEval-2023 Task 3: Multilingual transformer-based model for detecting the Genre, the Framing, and the Persuasion Techniques in Online News

In this manuscript, we describe the participation of the UMUTeam in SemEval-2023 Task 3, a shared task on detecting different aspects of news articles and other web documents, such as document category, framing dimensions, and persuasion technique in a multilingual setup.The task has been organized into three related subtasks, and we have been involved in the first two.Our approach is based on a fine-tuned multilingual transformer-based model that uses the dataset of all languages at once and a sentence transformer model to extract the most relevant chunk of a text for subtasks 1 and 2. The input data was truncated to 200 tokens with 50 overlaps using the sentence-transformer model to obtain the subset of text most related to the articles' titles.Our system has performed good results in subtask 1 in most languages, and in some cases, such as French and German, we have archived first place in the official leader board.As for task 2, our system has also performed very well in all languages, ranking in all the top 10.
Annotating Decomposition in Time: Three Approaches for Again

This paper reports on a three-part series of original methods geared towards producing semantic annotations for the decompositional marker again.The three methods are (i) exhaustive expert annotation based on a comprehensive set of guidelines, (ii) extension of expert annotation by predicting presuppositions with a Multinomial Naïve Bayes classifier in the context of a meta-analysis to optimize feature selection and (iii) quality-controlled crowdsourcing with ensuing evaluation and KMeans clustering of annotation vectors.
Fixed Input Parameterization for Efficient Prompting

Recent works have shown that attaching prompts to the input is effective at conditioning Language Models (LM) to perform specific tasks.However, prompts are always included in the input text during inference, even when they are fixed, thus incurring substantial computational and memory overhead.Also, there is currently no straightforward method of utilizing prompts that are longer than the maximum input length of the LMs without incurring additional costs during inference.We formally define Fixed Input Parameterization (FIP) problem that focuses on injecting the fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input.We show that in scenarios with long fixed prompts, FIP can be up to 280 times more efficient in terms of total FLOPs than previous approaches.We further explore methodologies for FIP and show promising results in persona-dependent conversation, semantic parsing, and zero-shot learning with task instructions.Through these explorations, we show that FIP can be a promising direction for conditioning language models, in scenarios with long and fixed prompts 1 .
Towards Stable Natural Language Understanding via Information Entropy Guided Debiasing

Although achieving promising performance, current Natural Language Understanding models tend to utilize dataset biases instead of learning the intended task, which always leads to performance degradation on out-of-distribution (OOD) samples.To increase the performance stability, previous debiasing methods empirically capture bias features from data to prevent the model from corresponding biases.However, our analyses show that the empirical debiasing methods may fail to capture part of the dataset biases and mistake semantic information of input text as biases, which limits the effectiveness of debiasing.To address these issues, we propose a debiasing framework IEGDB that comprehensively detects the dataset biases to induce a set of biased features, and purify the biased features with the guidance of information entropy.Experimental results show that IEGDB can consistently improve the stability of performance on OOD datasets for a set of widely adopted NLU models.
When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP

Multi-task learning (MTL) aims at achieving a better model by leveraging data and knowledge from multiple tasks.However, MTL does not always work -sometimes negative transfer occurs between tasks, especially when aggregating loosely related skills, leaving it an open question when MTL works.Previous studies show that MTL performance can be improved by algorithmic tricks.However, what tasks and skills should be included is less well explored.In this work, we conduct a case study in Financial NLP where multiple datasets exist for skills relevant to the domain, such as numeric reasoning and sentiment analysis.Due to the task difficulty and data scarcity in the Financial NLP domain, we explore when aggregating such diverse skills from multiple datasets with MTL can work.Our findings suggest that the key to MTL success lies in skill diversity, relatedness between tasks, and choice of aggregation size and shared capacity.Specifically, MTL works well when tasks are diverse but related, and when the size of the task aggregation and the shared capacity of the model are balanced to avoid overwhelming certain tasks. 1
Language Detoxification with Attribute-Discriminative Latent Space

Transformer-based Language Models (LMs) have achieved impressive results on natural language understanding tasks, but they can also generate toxic text such as insults, threats, and profanity, limiting their real-world applications. To overcome this issue, a few text generation approaches aim to detoxify toxic texts using additional LMs or perturbations. However, previous methods require excessive memory, computations, and time which are serious bottlenecks in their real-world application. To address such limitations, we propose an effective yet efficient method for language detoxification using an attribute-discriminative latent space. Specifically, we project the latent space of an original Transformer LM onto a discriminative latent space that well-separates texts by their attributes using a projection block and an attribute discriminator. This allows the LM to control the text generation to be non-toxic with minimal memory and computation overhead. We validate our model, Attribute-Discriminative Language Model (ADLM) on detoxified language and dialogue generation tasks, on which our method significantly outperforms baselines both in performance and efficiency.
How does the task complexity of masked pretraining objectives affect downstream performance?

Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts.Although simpler and computationally efficient pretraining objectives, e.g., predicting the first character of a masked token, have recently shown comparable results to MLM, no objectives with a masking scheme actually outperform it in downstream tasks.Motivated by the assumption that their lack of complexity plays a vital role in the degradation, we validate whether more complex masked objectives can achieve better results and investigate how much complexity they should have to perform comparably to MLM.Our results using GLUE, SQuAD, and Universal Dependencies benchmarks demonstrate that more complicated objectives tend to show better downstream results with at least half of the MLM complexity needed to perform comparably to MLM.Finally, we discuss how we should pretrain a model using a masked objective from the task complexity perspective.
A Language-First Approach for Procedure Planning

Procedure planning, or the ability to predict a series of steps that can achieve a given goal conditioned on the current observation, is critical for building intelligent embodied agents that can assist users in everyday tasks.Encouraged by the recent success of language models (LMs) for zero-shot (Huang et al., 2022a;Ahn et al., 2022) and few-shot planning (Micheli and Fleuret, 2021), we hypothesize that LMs may be equipped with stronger priors for planning compared to their visual counterparts.To this end, we propose a language-first procedure planning framework with modularized design: we first align the current and goal observations with corresponding steps and then use a pre-trained LM to predict the intermediate steps.Under this framework, we find that using an image captioning model for alignment can already match state-of-the-art performance and by designing a double retrieval model conditioned over current and goal observations jointly, we can achieve large improvements (19.2% -98.9% relatively higher success rate than state-of-the-art) on both COIN (Tang et al., 2019) and CrossTask (Zhukov et al., 2019) benchmarks.Our work verifies the planning ability of LMs and demonstrates how LMs can serve as a powerful "reasoning engine" even when the input is provided in another modality. 1
Towards the First Named Entity Recognition of Inuktitut for an Improved Machine Translation

Named Entity Recognition is a crucial step to ensure good quality performance of several Natural Language Processing applications and tools, including machine translation and information retrieval.Moreover, it is considered as a fundamental module of many Natural Language Understanding tasks such as questionanswering systems.This paper presents a first study on NER for an under-represented Indigenous Inuit language of Canada, Inuktitut, which lacks linguistic resources and large labeled data.Our proposed NER model for Inuktitut is built by transferring linguistic characteristics from English to Inuktitut, based on either rules or bilingual word embeddings.We provide an empirical study based on a comparison with the state of the art models and as well as intrinsic and extrinsic evaluations.In terms of Recall, Precision and F-score, the obtained results show the effectiveness of the proposed NER methods.Furthermore, it improved the performance of Inuktitut-English Neural Machine Translation.
CL-UZH at SemEval-2023 Task 10: Sexism Detection through Incremental Fine-Tuning and Multi-Task Learning with Label Descriptions

The widespread popularity of social media has led to an increase in hateful, abusive, and sexist language, motivating methods for the automatic detection of such phenomena.The goal of the SemEval shared task Towards Explainable Detection of Online Sexism (EDOS 2023) is to detect sexism in English social media posts (subtask A), and to categorize such posts into four coarse-grained sexism categories (subtask B), and eleven fine-grained subcategories (subtask C).In this paper, we present our submitted systems for all three subtasks, based on a multi-task model that has been fine-tuned on a range of related tasks and datasets before being fine-tuned on the specific EDOS subtasks.We implement multi-task learning by formulating each task as binary pairwise text classification, where the dataset and label descriptions are given along with the input text.The results show clear improvements over a finetuned DeBERTa-V3 serving as a baseline leading to F 1 -scores of 85.9% in subtask A (rank 13/84), 64.8% in subtask B (rank 19/69), and 44.9% in subtask C (26/63). 1 OFFENSIVE CONTENT WARNING: This report contains some examples of hateful content.This is strictly for the purposes of enabling this research, and we have sought to minimize the number of examples where possible.Please be aware that this content could be offensive and cause you distress.dataset label type label value size DGHSD hate speech yes: 46.
Authorship Attribution of Late 19th Century Novels using GAN-BERT

Authorship attribution aims to identify the author of an anonymous text.The task becomes even more worthwhile when it comes to literary works.For example, pen names were commonly used by female authors in the 19th century resulting in some literary works being incorrectly attributed or claimed.With this motivation, we collated a dataset of late 19thcentury novels in English.Due to the imbalance in the dataset and the unavailability of enough data per author, we employed the GAN-BERT model along with data sampling strategies to fine-tune a transformer-based model for authorship attribution.Differently from the earlier studies on the GAN-BERT model, we conducted transfer learning on comparatively smaller author subsets to train more focused author-specific models yielding performance over 0.88 accuracy and F1 scores.Furthermore, we observed that increasing the sample size has a negative impact on the model's performance.Our research mainly contributes to the ongoing authorship attribution research using GAN-BERT architecture, especially in attributing disputed novelists in the late 19th century.
Recall, Expand, and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing

Ultra-fine entity typing (UFET) predicts extremely free-formed types (e.g., president, politician) of a given entity mention (e.g., Joe Biden) in context.State-of-the-art (SOTA) methods use the cross-encoder (CE) based architecture.CE concatenates a mention (and its context) with each type and feeds the pair into a pretrained language model (PLM) to score their relevance.It brings deeper interaction between the mention and the type to reach better performance but has to perform N (the type set size) forward passes to infer all the types of a single mention.CE is therefore very slow in inference when the type set is large (e.g., N = 10k for UFET).To this end, we propose to perform entity typing in a recall-expand-filter manner.The recall and expansion stages prune the large type set and generate K (typically much smaller than N ) most relevant type candidates for each mention.At the filter stage, we use a novel model called MCCE to concurrently encode and score all these K candidates in only one forward pass to obtain the final type prediction.We investigate different model options for each stage and conduct extensive experiments to compare each option, experiments show that our method reaches SOTA performance on UFET and is thousands of times faster than the CE-based architecture.We also found our method is very effective in fine-grained (130 types) and coarse-grained (9 types) entity typing.Our code is available at
Dense-ATOMIC: Towards Densely-connected ATOMIC with High Knowledge Coverage and Massive Multi-hop Paths

ATOMIC is a large-scale commonsense knowledge graph (CSKG) containing everyday ifthen knowledge triplets, i.e., {head event, relation, tail event}.The one-hop annotation manner made ATOMIC a set of independent bipartite graphs, which ignored the numerous links between events in different bipartite graphs and consequently caused shortages in knowledge coverage and multi-hop paths.In this work, we aim to construct Dense-ATOMIC with high knowledge coverage and massive multi-hop paths.The events in ATOMIC are normalized to a consistent pattern at first.We then propose a CSKG completion method called Rel-CSKGC to predict the relation given the head event and the tail event of a triplet, and train a CSKG completion model based on existing triplets in ATOMIC.We finally utilize the model to complete the missing links in ATOMIC and accordingly construct Dense-ATOMIC.Both automatic and human evaluation on an annotated subgraph of ATOMIC demonstrate the advantage of Rel-CSKGC over strong baselines.We further conduct extensive evaluations on Dense-ATOMIC in terms of statistics, human evaluation, and simple downstream tasks, all proving Dense-ATOMIC's advantages in Knowledge Coverage and Multi-hop Paths.Both the source code of Rel-CSKGC and Dense-ATOMIC are publicly available on https://github.com/NUSTM/Dense-ATOMIC.
MidMed: Towards Mixed-Type Dialogues for Medical Consultation

Most medical dialogue systems assume that patients have clear goals (medicine querying, surgical operation querying, etc.) before medical consultation.However, in many real scenarios, due to the lack of medical knowledge, it is usually difficult for patients to determine clear goals with all necessary slots.In this paper, we identify this challenge as how to construct medical consultation dialogue systems to help patients clarify their goals.To mitigate this challenge, we propose a novel task and create a human-tohuman mixed-type medical consultation dialogue corpus, termed MidMed 1 , covering five dialogue types: task-oriented dialogue for diagnosis, recommendation, knowledge-grounded dialogue, QA, and chitchat.MidMed covers four departments (otorhinolaryngology, ophthalmology, skin, and digestive system), with 8,175 dialogues.Furthermore, we build baselines on MidMed and propose an instructionguiding medical dialogue generation framework, termed InsMed, to address this task.Experimental results show the effectiveness of InsMed.
Easy-to-Hard Learning for Information Extraction

Information extraction (IE) systems aim to automatically extract structured information, such as named entities, relations between entities, and events, from unstructured texts.While most existing work addresses a particular IE task, universally modeling various IE tasks with one model has achieved great success recently.Despite their success, they employ a one-stage learning strategy, i.e., directly learning to extract the target structure given the input text, which contradicts the human learning process.In this paper, we propose a unified easy-tohard learning framework consisting of three stages, i.e., the easy stage, the hard stage, and the main stage, for IE by mimicking the human learning process.By breaking down the learning process into multiple stages, our framework facilitates the model to acquire general IE task knowledge and improve its generalization ability.Extensive experiments across four IE tasks demonstrate the effectiveness of our framework.We achieve new state-of-the-art results on 13 out of 17 datasets.
Towards Building a Robust Toxicity Predictor

Recent NLP literature pays little attention to the robustness of toxicity language predictors, while these systems are most likely to be used in adversarial contexts. This paper presents a novel adversarial attack, \texttt{ToxicTrap}, introducing small word-level perturbations to fool SOTA text classifiers to predict toxic text samples as benign. \texttt{ToxicTrap} exploits greedy based search strategies to enable fast and effective generation of toxic adversarial examples. Two novel goal function designs allow \texttt{ToxicTrap} to identify weaknesses in both multiclass and multilabel toxic language detectors. Our empirical results show that SOTA toxicity text classifiers are indeed vulnerable to the proposed attacks, attaining over 98\% attack success rates in multilabel cases. We also show how a vanilla adversarial training and its improved version can help increase robustness of a toxicity detector even against unseen attacks.
Rethinking Translation Memory Augmented Neural Machine Translation

This paper rethinks translation memory augmented neural machine translation (TMaugmented NMT) from two perspectives, i.e., a probabilistic view of retrieval and the variancebias decomposition principle.The finding demonstrates that TM-augmented NMT is good at the ability of fitting data (i.e., lower bias) but is more sensitive to the fluctuations in the training data (i.e., higher variance), which provides an explanation to a recently reported contradictory phenomenon on the same translation task: TM-augmented NMT substantially advances vanilla NMT under the high-resource scenario whereas it fails under the low-resource scenario.Then we propose a simple yet effective TM-augmented NMT model to promote the variance and address the contradictory phenomenon.Extensive experiments show that the proposed TMaugmented NMT achieves consistent gains over both conventional NMT and existing TM-augmented NMT under two variancepreferable (low-resource and plug-and-play) scenarios as well as the high-resource scenario.
Gradient-Boosted Decision Tree for Listwise Context Model in Multimodal Review Helpfulness Prediction

Multimodal Review Helpfulness Prediction (MRHP) aims to rank product reviews based on predicted helpfulness scores and has been widely applied in e-commerce via presenting customers with useful reviews.Previous studies commonly employ fully-connected neural networks (FCNNs) as the final score predictor and pairwise loss as the training objective.However, FCNNs have been shown to perform inefficient splitting for review features, making the model difficult to clearly differentiate helpful from unhelpful reviews.Furthermore, pairwise objective, which works on review pairs, may not completely capture the MRHP goal to produce the ranking for the entire review list, and possibly induces low generalization during testing.To address these issues, we propose a listwise attention network that clearly captures the MRHP ranking context and a listwise optimization objective that enhances model generalization.We further propose gradient-boosted decision tree as the score predictor to efficaciously partition product reviews' representations.Extensive experiments demonstrate that our method achieves state-of-the-art results and polished generalization performance on two large-scale MRHP benchmark datasets.
Rudolf Christoph Eucken at SemEval-2023 Task 4: An Ensemble Approach for Identifying Human Values from Arguments

The subtle human values we acquire through life experiences govern our thoughts and gets reflected in our speech.It plays an integral part in capturing the essence of our individuality and making it imperative to identify such values in computational systems that mimic human actions.Computational argumentation is a field that deals with the argumentation capabilities of humans and can benefit from identifying such values.Motivated by that, we present an ensemble approach for detecting human values from argument text.Our ensemble comprises three models: (i) An entailment-based model for determining the human values based on their descriptions, (ii) A Roberta-based classifier that predicts the set of human values from an argument.(iii) A Roberta-based classifier to predict a reduced set of human values from an argument.We experiment with different ways of combining the models and report our results.Furthermore, our best combination achieves an overall F1 score of 0.48 on the main test set.