Reviewerly/exHarmony huggingface dataset:
Foiling Training-Time Attacks on Neural Machine Translation Systems

Neural machine translation (NMT) systems are vulnerable to backdoor attacks, whereby an attacker injects poisoned samples into training such that a trained model produces malicious translations.Nevertheless, there is little research on defending against such backdoor attacks in NMT.In this paper, we first show that backdoor attacks that have been successful in text classification are also effective against machine translation tasks.We then present a novel defence method that exploits a key property of most backdoor attacks: namely the asymmetry between the source and target language sentences, which is used to facilitate malicious text insertions, substitutions and suchlike.Our technique uses word alignment coupled with language model scoring to detect outlier tokens, and thus can find and filter out training instances which may contain backdoors.Experimental results demonstrate that our technique can significantly reduce the success of various attacks by up to 89.0%, while not affecting predictive accuracy.
Style Transfer as Data Augmentation: A Case Study on Named Entity Recognition

In this work, we take the named entity recognition task in the English language as a case study and explore style transfer as a data augmentation method to increase the size and diversity of training data in low-resource scenarios.We propose a new method to effectively transform the text from a high-resource domain to a low-resource domain by changing its style-related attributes to generate synthetic data for training.Moreover, we design a constrained decoding algorithm along with a set of key ingredients for data selection to guarantee the generation of valid and coherent data.Experiments and analysis on five different domain pairs under different data regimes demonstrate that our approach can significantly improve results compared to current state-of-the-art data augmentation methods.Our approach is a practical solution to data scarcity, and we expect it to be applicable to other NLP tasks. 1
Human Age Estimation Through Audio Utilising MFCC and RNN

Age is one of human main attributes. Age is important factor to improve communication experience. Age estimation has been used in several applications to improve user experience. Therefore, an approach is needed to estimate the user age, one of which is through audio. In this study, Mel Frequency Cepstrum Coefficients (MFCC) and Recurrent Neural Network (RNN) will be used to estimate age through audio. MFCC is used to get features from audio data, while RNN is used to estimate age. Dataset used here was taken from corpus of user speech data on the Common Voice website. This study shows that MFCC and RNN methods are able to estimate human age through audio with highest accuracy obtained in SimpleRNN is 0.5647, and 0.7087 in LSTM.
Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluation

The automatic detection of hate speech online is an active research area in NLP. Most of the studies to date are based on social media datasets that contribute to the creation of hate speech detection models trained on them. However, data creation processes contain their own biases, and models inherently learn from these dataset-specific biases. In this paper, we perform a large-scale cross-dataset comparison where we fine-tune language models on different hate speech detection datasets. This analysis shows how some datasets are more generalizable than others when used as training data. Crucially, our experiments show how combining hate speech detection datasets can contribute to the development of robust hate speech detection models. This robustness holds even when controlling by data size and compared with the best individual datasets.
Conversation Derailment Forecasting with Graph Convolutional Networks

Online conversations are particularly susceptible to derailment, which can manifest itself in the form of toxic communication patterns like disrespectful comments or verbal abuse.Forecasting conversation derailment predicts signs of derailment in advance enabling proactive moderation of conversations.Current state-ofthe-art approaches to address this problem rely on sequence models that treat dialogues as text streams.We propose a novel model based on a graph convolutional neural network that considers dialogue user dynamics and the influence of public perception on conversation utterances.Through empirical evaluation, we show that our model effectively captures conversation dynamics and outperforms the state-of-the-art models on the CGA and CMV benchmark datasets by 1.5% and 1.7%, respectively.
NAIST Simultaneous Speech-to-speech Translation System for IWSLT 2023

This paper describes NAIST's submission to the IWSLT 2023 Simultaneous Speech Translation task:English-to-{German, Japanese, Chinese} speech-to-text translation and English-to-Japanese speech-to-speech translation.Our speech-to-text system uses an end-to-end multilingual speech translation model based on large-scale pre-trained speech and text models.We add Inter-connections into the model to incorporate the outputs from intermediate layers of the pre-trained speech model and augment prefix-to-prefix text data using Bilingual Prefix Alignment to enhance the simultaneity of the offline speech translation model.Our speech-to-speech system employs an incremental text-to-speech module that consists of a Japanese pronunciation estimation model, an acoustic model, and a neural vocoder.
Is Continuous Prompt a Combination of Discrete Prompts? Towards a Novel View for Interpreting Continuous Prompts

The broad adoption of continuous prompts has brought state-of-the-art results on a diverse array of downstream natural language processing (NLP) tasks.Nonetheless, little attention has been paid to the interpretability and transferability of continuous prompts.Faced with the challenges, we investigate the feasibility of interpreting continuous prompts as the weighting of discrete prompts by jointly optimizing prompt fidelity and downstream fidelity.Our experiments show that: (1) one can always find a combination of discrete prompts as the replacement of continuous prompts that performs well on downstream tasks; (2) our interpretable framework faithfully reflects the reasoning process of source prompts; (3) our interpretations provide effective readability and plausibility, which is helpful to understand the decisionmaking of continuous prompts and discover potential shortcuts.Moreover, through the bridge constructed between continuous prompts and discrete prompts using our interpretations, it is promising to implement the cross-model transfer of continuous prompts without extra training signals.We hope this work will lead to a novel perspective on the interpretations of continuous prompts.
CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation

We propose a method to control the attributes of Language Models (LMs) for the text generation task using Causal Average Treatment Effect (ATE) scores and counterfactual augmentation.We explore this method, in the context of LM detoxification, and propose the Causally Fair Language (CFL) architecture for detoxifying pre-trained LMs in a plug-and-play manner.Our architecture is based on a Structural Causal Model (SCM) that is mathematically transparent and computationally efficient as compared with many existing detoxification techniques.We also propose several new metrics that aim to better understand the behaviour of LMs in the context of toxic text generation.Further, we achieve state of the art performance for toxic degeneration, which are computed using REALTOXICITYPROMPTS (RTP) benchmark.Our experiments show that CFL achieves such a detoxification without much impact on the model perplexity.We also show that CFL mitigates the unintended bias problem through experiments on the BOLD dataset.
Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency

Word error rate (WER) and character error rate (CER) are standard metrics in Speech Recognition (ASR), but one problem has always been alternative spellings: If one's system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word.Japanese is notorious for "lacking orthography": most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation.In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR.We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana.In a manual evaluation, raters rated 95.4% of the proposed spelling variants as plausible.ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%-3.1% absolute reduction in CER depending on the task.
HHU at SemEval-2023 Task 3: An Adapter-based Approach for News Genre Classification

This paper describes our approach for Subtask 1 of Task 3 at SemEval-2023.In this subtask, task participants were asked to classify multilingual news articles for one of three classes: Reporting, Opinion Piece or Satire.By training an AdapterFusion layer composing the taskadapters from different languages, we successfully combine the language-exclusive knowledge and show that this improves the results in nearly all cases, including in zero-shot scenarios.
Everything you need to know about Multilingual LLMs: Towards fair, performant and reliable models for languages of the world

Sunayana Sitaram, Monojit Choudhury, Barun Patra, Vishrav Chaudhary, Kabir Ahuja, Kalika Bali. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts). 2023.
REV: Information-Theoretic Evaluation of Free-Text Rationales

Generating free-text rationales is a promising step towards explainable NLP, yet evaluating such rationales remains a challenge.Existing metrics have mostly focused on measuring the association between the rationale and a given label.We argue that an ideal metric should focus on the new information uniquely provided in the rationale that is otherwise not provided in the input or the label.We investigate this research problem from an information-theoretic perspective using conditional V-information (Hewitt et al., 2021).More concretely, we propose a metric called REV (Rationale Evaluation with conditional V-information), to quantify the amount of new, label-relevant information in a rationale beyond the information already available in the input or the label.Experiments across four benchmarks with reasoning tasks, including chain-of-thought, demonstrate the effectiveness of REV in evaluating rationale-label pairs, compared to existing metrics.We further demonstrate REV is consistent with human judgments on rationale evaluations and provides more sensitive measurements of new information in free-text rationales.When used alongside traditional performance metrics, REV provides deeper insights into models' reasoning and prediction processes.1 * Work done during an internship at AI2. 1
The RST Continuity Corpus

We present the RST Continuity Corpus (RST-CC), a corpus of discourse relations annotated for continuity dimensions.Continuity or discontinuity (maintaining or shifting deictic centres across discourse segments) is an important property of discourse relations, but the two are correlated in greatly varying ways.To analyse this correlation, the relations in the RST-CC are annotated using operationalised versions of Givón's (1993) continuity dimensions.We also report on the inter-annotator agreement, and discuss recurrent annotation issues.First results show substantial variation of continuity dimensions within and across relation types.
Revisiting Non-Autoregressive Translation at Scale

In real-world systems, scaling has been critical for improving the translation quality in autoregressive translation (AT), which however has not been well studied for non-autoregressive translation (NAT).In this work, we bridge the gap by systematically studying the impact of scaling on NAT behaviors.Extensive experiments on six WMT benchmarks over two advanced NAT models show that scaling can alleviate the commonly-cited weaknesses of NAT models, resulting in better translation performance.To reduce the side-effect of scaling on decoding speed, we empirically investigate the impact of NAT encoder and decoder on the translation performance.Experimental results on the large-scale WMT20 En-De show that the asymmetric architecture (e.g.bigger encoder and smaller decoder) can achieve comparable performance with the scaling model, while maintaining the superiority of decoding speed with standard NAT models.To this end, we establish a new benchmark by validating scaled NAT models on the scaled dataset, which can be regarded as a strong baseline for future works.We release code and system outputs at https://github.com/DeepLearnXMU/Scaling4NAT.
ConvRGX: Recognition, Generation, and Extraction for Self-trained Conversational Question Answering

Collecting and constructing human-annotated corpora for training conversational questionanswering (CQA) models has recently been shown to be inefficient and costly.To solve this problem, previous works have proposed training QA models with automatically generated QA data.In this work, we extend earlier studies on QA synthesis, and propose an efficient QA data generation algorithm under conversational settings.Our model recognizes potential dialogue topics, generates corresponding questions, and extracts answers from grounding passages.To improve the quality of generated QAs and downstream self-training of CQA models, we propose dropout and agreement-based QA selection methods.We conduct experiments on both data augmentation and domain adaptation settings.Experiments on the QuAC and Doc2Dial tasks show that the proposed method can significantly improve the quality of generated QA data, and also improves the accuracy of self-trained CQA models based on the constructed training corpora.
Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction

Document-level relation extraction (DocRE)aims to infer complex semantic relations among entities in a document.Distant supervision (DS) is able to generate massive auto-labeled data, which can improve DocRE performance.Recent works leverage pseudo labels generated by the pre-denoising model to reduce noise in DS data.However, unreliable pseudo labels bring new noise, e.g., adding false pseudo labels and losing correct DS labels.Therefore, how to select effective pseudo labels to denoise DS data is still a challenge in document-level distant relation extraction.To tackle this issue, we introduce uncertainty estimation technology to determine whether pseudo labels can be trusted.In this work, we propose a Documentlevel distant Relation Extraction framework with Uncertainty Guided label denoising, UG-DRE.Specifically, we propose a novel instancelevel uncertainty estimation method, which measures the reliability of the pseudo labels with overlapping relations.By further considering the long-tail problem, we design dynamic uncertainty thresholds for different types of relations to filter high-uncertainty pseudo labels.We conduct experiments on two public datasets.Our framework outperforms strong baselines by 1.91 F 1 and 2.28 Ign F 1 on the RE-DocRED dataset.
Dramatic Conversation Disentanglement

We present a new dataset for studying conversation disentanglement in movies and TV series.While previous work has focused on conversation disentanglement in IRC chatroom dialogues, movies and TV shows provide a space for studying complex pragmatic patterns of floor and topic change in face-to-face multiparty interactions.In this work, we draw on theoretical research in sociolinguistics, sociology, and film studies to operationalize a conversational thread (including the notion of a floor change) in dramatic texts, and use that definition to annotate a dataset of 10,033 dialogue turns (comprising 2,209 threads) from 831 movies.We compare the performance of several disentanglement models on this dramatic dataset, and apply the best-performing model to disentangle 808 movies.We see that, contrary to expectation, average thread lengths do not decrease significantly over the past 40 years, and characters portrayed by actors who are women, while underrepresented, initiate more new conversational threads relative to their speaking time. GEORGIE. Morning.GEORGIE SR.How's the ankle?GEORGIE.I will be all right.Think I will be able to start against Nacodoches?GEORGIE SR.I can't play favorites Georgie, depends on how hard you work. MISSY. Mom
Rethinking Annotation: Can Language Learners Contribute?

Researchers have traditionally recruited native speakers to provide annotations for widely used benchmark datasets.However, there are languages for which recruiting native speakers can be difficult, and it would help to find learners of those languages to annotate the data.In this paper, we investigate whether language learners can contribute annotations to benchmark datasets.In a carefully controlled annotation experiment, we recruit 36 language learners, provide two types of additional resources (dictionaries and machine-translated sentences), and perform mini-tests to measure their language proficiency.We target three languages, English, Korean, and Indonesian, and the four NLP tasks of sentiment analysis, natural language inference, named entity recognition, and machine reading comprehension.We find that language learners, especially those with intermediate or advanced levels of language proficiency, are able to provide fairly accurate labels with the help of additional resources.Moreover, we show that data annotation improves learners' language proficiency in terms of vocabulary and grammar.One implication of our findings is that broadening the annotation task to include language learners can open up the opportunity to build benchmark datasets for languages for which it is difficult to recruit native speakers.
The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers

Applying language models to natural language processing tasks typically relies on the representations in the final model layer, as intermediate hidden layer representations are presumed to be less informative.In this work, we argue that due to the gradual improvement across model layers, additional information can be gleaned from the contrast between higher and lower layers during inference.Specifically, in choosing between the probable next token predictions of a generative model, the predictions of lower layers can be used to highlight which candidates are best avoided.We propose a novel approach that utilizes the contrast between layers to improve text generation outputs, and show that it mitigates degenerative behaviors of the model in open-ended generation, significantly improving the quality of generated texts.Furthermore, our results indicate that contrasting between model layers at inference time can yield substantial benefits to certain aspects of general language model capabilities, more effectively extracting knowledge during inference from a given set of model parameters.
FC-KBQA: A Fine-to-Coarse Composition Framework for Knowledge Base Question Answering

The generalization problem on KBQA has drawn considerable attention.Existing research suffers from the generalization issue brought by the entanglement in the coarse-grained modeling of the logical expression, or inexecutability issues due to the fine-grained modeling of disconnected classes and relations in real KBs.We propose a Fine-to-Coarse Composition framework for KBQA (FC-KBQA) to both ensure the generalization ability and executability of the logical expression.The main idea of FC-KBQA is to extract relevant finegrained knowledge components from KB and reformulate them into middle-grained knowledge pairs for generating the final logical expressions.FC-KBQA derives new state-of-theart performance on GrailQA and WebQSP, and runs 4 times faster than the baseline.Our code is now available at
Extracting Shopping Interest-Related Product Types from the Web

Recommending a diversity of product types (PTs) is important for a good shopping experience when customers are looking for products around their high-level shopping interests (SIs) such as hiking.However, the SI-PT connection is typically absent in e-commerce product catalogs and expensive to construct manually due to the volume of potential SIs, which prevents us from establishing a recommender with easily accessible knowledge systems.To establish such connections, we propose to extract PTs from the Web pages containing handcrafted PT recommendations for SIs.The extraction task is formulated as binary HTML node classification given the general observation that an HTML node in our target Web pages can present one and only one PT phrase.Accordingly, we introduce TRENC, which stands for Tree-Transformer Encoders for Node Classification.It improves the internode dependency modeling with modified attention mechanisms that preserve the longterm sibling and ancestor-descendant relations.TRENC also injects SI into node features for better semantic representation.Trained on pages regarding limited SIs, TRENC is ready to be applied to other unobserved interests.Experiments on our manually constructed dataset, WEBPT, show that TRENC outperforms the best baseline model by 2.37 F 1 points in the zero-shot setup.The performance indicates the feasibility of constructing SI-PT relations and using them to power downstream applications such as search and recommendation.
Evaluating Multilingual Speech Translation under Realistic Conditions with Resegmentation and Terminology

We present the ACL 60/60 evaluation sets for multilingual translation of ACL 2022 technical presentations into 10 target languages. This dataset enables further research into multilingual speech translation under realistic recording conditions with unsegmented audio and domain-specific terminology, applying NLP tools to text and speech in the technical domain, and evaluating and improving model robustness to diverse speaker demographics.
Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations

Human-annotated labels and explanations are critical for training explainable NLP models.However, unlike human-annotated labels whose quality is easier to calibrate (e.g., with a majority vote), human-crafted free-form explanations can be quite subjective.Before blindly using them as ground truth to train ML models, a vital question needs to be asked: How do we evaluate a human-annotated explanation's quality?In this paper, we build on the view that the quality of a human-annotated explanation can be measured based on its helpfulness (or impairment) to the ML models' performance for the desired NLP tasks for which the annotations were collected.In comparison to the commonly used Simulatability score, we define a new metric that can take into consideration of the helpfulness of an explanation for model performance at both fine-tuning and inference.With the help of a unified dataset format, we evaluated the proposed metric on five datasets (e.g., e-SNLI) against two model architectures (T5 and BART), and the results show that our proposed metric can objectively evaluate the quality of human-annotated explanations, while Simulatability falls short.