Reviewerly/exHarmony huggingface dataset:
Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language Recognition

Existing supervised sign language recognition systems rely on an abundance of well-annotated data.Instead, an unsupervised speech-to-sign language recognition (SSR-U) system learns to translate between spoken and sign languages by observing only non-parallel speech and signlanguage corpora.We propose speech2sign-U, a neural network-based approach capable of both character-level and word-level SSR-U.Our approach significantly outperforms baselines directly adapted from unsupervised speech recognition (ASR-U) models by as much as 50% recall@10 on several challenging American sign language corpora with various levels of sample sizes, vocabulary sizes, and audio and visual variability.The code is available at cactuswiththoughts/UnsupSpeech2Sign.git.
Cross-lingual Continual Learning

The longstanding goal of multi-lingual learning has been to develop a universal cross-lingual model that can withstand the changes in multilingual data distributions.There has been a large amount of work to adapt such multilingual models to unseen target languages.However, the majority of work in this direction focuses on the standard one-hop transfer learning pipeline from source to target languages, whereas in realistic scenarios, new languages can be incorporated at any time in a sequential manner.In this paper, we present a principled Cross-lingual Continual Learning (CCL) evaluation paradigm, where we analyze different categories of approaches used to continually adapt to emerging data from different languages.We provide insights into what makes multilingual sequential learning particularly challenging.To surmount such challenges, we benchmark a representative set of cross-lingual continual learning algorithms and analyze their knowledge preservation, accumulation, and generalization capabilities compared to baselines on carefully curated datastreams.The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata, which go beyond conventional transfer learning.
SRI-B’s Systems for IWSLT 2023 Dialectal and Low-resource Track: Marathi-Hindi Speech Translation

This paper describes the speech translation systems SRI-B developed for the IWSLT 2023 Evaluation Campaign Dialectal and Lowresource track: Marathi-Hindi Speech Translation.We propose systems for both the constrained (systems are trained only on the datasets provided by the organizers) and the unconstrained conditions (systems can be trained with any resource).For both the conditions, we build end-to-end speech translation networks comprising of a conformer encoder and a transformer decoder.Under both the conditions, we leverage Marathi Automatic Speech Recognition (ASR) data to pre-train the encoder and subsequently train the entire model on the speech translation data.Our results demonstrate that pre-training the encoder with ASR data is a key step in significantly improving the speech translation performance.We also show that conformer encoders are inherently superior to its transformer counterparts for speech translation tasks.Our primary submissions achieved a BLEU% score of 31.2 on the constrained condition and 32.4 on the unconstrained condition.We secured the top position in the constrained condition and second position in the unconstrained condition.
hhuEDOS at SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS) Binary Sexism Detection (Subtask A)

In this paper, we describe our contribution to the SemEval-2023 Task 10 (Subtask A), a shared task on detecting and predicting sexist language.The dataset consists of labeled sexist and non-sexist data targeted towards women acquired from both Reddit and Gab.We present and compare several approaches we experimented with and our final submitted model.Additional error analysis is given to recognize challenges we dealt with in our process.A total of 84 teams participated.Our model ranks 55th overall in Subtask A of the shared task.
StFX NLP at SemEval-2023 Task 1: Multimodal Encoding-based Methods for Visual Word Sense Disambiguation

SemEval-2023's Task 1, Visual Word Sense Disambiguation, a task about text semantics and visual semantics, is about selecting the best-matched image to represent a target word in a limited context.We explored several methods, including image captioning methods and CLIP-based methods, and submitted our predictions in the competition for this task.This paper will focus on the methods we used and their performance, and provide an analysis and discussion of their performance.
BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases

Energy-based models (EBMs) have gained popularity for controlled text generation due to their high applicability to a wide range of constraints.However, sampling from EBMs is non-trivial, as it often requires a large number of iterations to converge to plausible text, which slows down the decoding process and makes it less practical for real-world applications.In this work, we propose BOLT, which relies on tunable biases to directly adjust the language model's output logits.Unlike prior work, BOLT maintains the generator's autoregressive nature to assert a strong control on token-wise conditional dependencies and overall fluency, and thus converges faster.When compared with state-of-the-arts on controlled generation tasks using both soft constraints (e.g., sentiment control) and hard constraints (e.g., keyword-guided topic control), BOLT demonstrates significantly improved efficiency and fluency.On sentiment control, BOLT is 7x faster than competitive baselines, and more fluent in 74.4% of the evaluation samples according to human judges.
Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features

A challenge towards developing NLP systems for the world's languages is understanding how they generalize to typological differences relevant for real-world applications.To this end, we propose M2C, a morphologicallyaware framework for behavioral testing of NLP models.We use M2C to generate tests that probe models' behavior in light of specific linguistic features in 12 typologically diverse languages.We evaluate state-of-the-art language models on the generated tests.While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish.Our findings motivate the development of models that address these blind spots. 11 We make all code publicly available at https://github.com/google-research/multi-morph-checklist.2 For instance, while tone is present in around 80% of African languages (Adebara and Abdul-Mageed, 2022), few Indo-European languages can be considered tonal.0.0 25.0 50.0 75.
Controlled Text Generation with Hidden Representation Transformations

We propose CHRT (Control Hidden Representation Transformation) -a controlled language generation framework that steers large language models to generate text pertaining to certain attributes (such as toxicity).CHRT gains attribute control by modifying the hidden representation of the base model through learned transformations.We employ a contrastive-learning framework to learn these transformations that can be combined to gain multi-attribute control.The effectiveness of CHRT is experimentally shown by comparing it with seven baselines over three attributes.CHRT outperforms all the baselines in the task of detoxification, positive sentiment steering, and text simplification while minimizing the loss in linguistic qualities.Further, our approach has the lowest inference latency of only 0.01 seconds more than the base model, making it the most suitable for high-performance production environments.We open-source our code and release two novel datasets to further propel controlled language generation research.
Semantic-Aware Dynamic Retrospective-Prospective Reasoning for Event-Level Video Question Answering

Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to obtain the visual information needed to provide optimal answers.However, despite significant progress in model performance, few studies have focused on using the explicit semantic connections between the question and visual information especially at the event level.There is need for using such semantic connections to facilitate complex reasoning across video frames.Therefore, we propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering.Specifically, we explicitly use the Semantic Role Labeling (SRL) structure of the question in the dynamic reasoning process where we decide to move to the next frame based on which part of the SRL structure (agent, verb, patient, etc.) of the question is being focused on.We conduct experiments on a benchmark EVQA dataset -TrafficQA.Results show that our proposed approach achieves superior performance compared to previous state-of-the-art models.
MultiQG-TI: Towards Question Generation from Multi-modal Sources

We study the new problem of automatic question generation (QG) from multi-modal sources containing images and texts, significantly expanding the scope of most of the existing work that focuses exclusively on QG from only textual sources.We propose a simple solution for our new problem, called MultiQG-TI, which enables a text-only question generator to process visual input in addition to textual input.Specifically, we leverage an image-to-text model and an optical character recognition model to obtain the textual description of the image and extract any texts in the image, respectively, and then feed them together with the input texts to the question generator.We only fine-tune the question generator while keeping the other components fixed.On the challenging ScienceQA dataset, we demonstrate that MultiQG-TI significantly outperforms ChatGPT with few-shot prompting, despite having hundred-times less trainable parameters.Additional analyses empirically confirm the necessity of both visual and textual signals for QG and show the impact of various modeling choices.Code is available at https://rb.gy/020tw * Work done while at Rice University.A solution is made up of two or more substances that are completely mixed.In a solution, solute particles are mixed into a solvent........
BIG-C: a Multimodal Multi-Purpose Dataset for Bemba

We present BIG-C (Bemba Image Grounded Conversations), a large multimodal dataset for Bemba.While Bemba is the most populous language of Zambia, it exhibits a dearth of resources which render the development of language technologies or language processing research almost impossible.The dataset is comprised of multi-turn dialogues between Bemba speakers based on images, transcribed and translated into English.There are more than 92,000 utterances/sentences, amounting to more than 180 hours of audio data with corresponding transcriptions and English translations.We also provide baselines on speech recognition (ASR), machine translation (MT) and speech translation (ST) tasks, and sketch out other potential future multimodal uses of our dataset.We hope that by making the dataset available to the research community, 1 this work will foster research and encourage collaboration across the language, speech, and vision communities especially for languages outside the "traditionally" used high-resourced ones.
SaFER: A Robust and Efficient Framework for Fine-tuning BERT-based Classifier with Noisy Labels

Learning on noisy datasets is a challenging problem when pre-trained language models are applied to real-world text classification tasks.In numerous industrial applications, acquiring task-specific datasets with 100% accurate labels is difficult, thus many datasets are accompanied by label noise at different levels.Previous work has shown that existing noisehandling methods could not improve the peak performance of BERT on noisy datasets, and might even deteriorate it.In this paper, we propose SaFER 1 , a robust and efficient finetuning framework for BERT-based text classifiers, combating label noises without access to any clean data for training or validation.Utilizing a label-agnostic early-stopping strategy and self-supervised learning, our proposed framework achieves superior performance in terms of both accuracy and speed on multiple text classification benchmarks.The trained model is finally fully deployed in several industrial biomedical literature mining tasks and demonstrates high effectiveness and efficiency.
Robustness of Learning from Task Instructions

Traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples.This paradigm seriously hinders the development of task generalization since preparing a taskspecific example set is costly.To build a system that can quickly and easily generalize to new tasks, textual instructions have been adopted as an emerging trend of supervision recently.These instructions give the model the definition of the task and allow the model to output the appropriate answer based on the instructions and inputs.However, task instructions are often expressed in different forms, which can be interpreted from two threads (Lou et al., 2023): firstly, some instructions are short sentences and are pretrained language model (PLM) oriented, such as prompts, while other instructions are paragraphs and are human-oriented, such as those annotation guidelines in Amazon MTurk; secondly, different end-users very likely explain the same task with instructions of different textual expressions.A robust system for task generalization should be able to handle any new tasks regardless of the variability of instructions.However, the system robustness in dealing with a wide range of instructions is still unexplored.This work investigates the system robustness when the instructions of new tasks are (i) manipulated, (ii) paraphrased, or (iii) from different levels of conciseness.To our knowledge, this is the first work that systematically studies how robust a PLM is when it is supervised by instructions with different factors of variability. 1
Distilling Efficient Language-Specific Models for Cross-Lingual Transfer

Massively multilingual Transformers (MMTs), such as mBERT and XLM-R, are widely used for cross-lingual transfer learning.While these are pretrained to represent hundreds of languages, end users of NLP systems are often interested only in individual languages.For such purposes, the MMTs' language coverage makes them unnecessarily expensive to deploy in terms of model size, inference time, energy, and hardware cost.We thus propose to extract compressed, language-specific models from MMTs which retain the capacity of the original MMTs for cross-lingual transfer.This is achieved by distilling the MMT bilingually, i.e., using data from only the source and target language of interest.Specifically, we use a two-phase distillation approach, termed BIS-TILLATION: (i) the first phase distils a general bilingual model from the MMT, while (ii) the second, task-specific phase sparsely fine-tunes the bilingual 'student' model using a task-tuned variant of the original MMT as its 'teacher'.We evaluate this distillation technique in zeroshot cross-lingual transfer across a number of standard cross-lingual benchmarks.The key results indicate that the distilled models exhibit minimal degradation in target language performance relative to the base MMT despite being significantly smaller and faster.Furthermore, we find that they outperform multilingually distilled models such as DistilmBERT and MiniLMv2 while having a very modest training budget in comparison, even on a perlanguage basis.
ACSMKRHR at SemEval-2023 Task 10: Explainable Online Sexism Detection(EDOS)

People are expressing their opinions online for a lot of years now.Although these opinions and comments provide people an opportunity of expressing their views, there is a lot of hate speech that can be found online.More specifically, sexist comments are very popular affecting and creating a negative impact on a lot of women and girls online.This paper describes the approaches of the SemEval-2023 Task 10 competition for Explainable Online Sexism Detection (EDOS).The task has been divided into 3 subtasks, introducing different classes of sexist comments.We have approached these tasks using the bert-cased and uncased models which are trained on the annotated dataset that has been provided in the competition.Task A provided the best F1 score of 80% on the test set, and tasks B and C provided 58% and 40% respectively.
Matesub: The Translated Subtitling Tool at the IWSLT2023 Subtitling Task

This paper briefly describes Matesub, the subtitling tool Translated used to participate in the Subtitling shared task at IWSLT 2023.Matesub is a professional web-based tool that combines state-of-the-art AI with a WYSIWYG editor.The automatic generation of subtitles in Matesub is based on a cascade architecture, composed of ASR, text segmenter and MT neural models, which allows covering any pair from about 60 languages and their variants.
Towards Zero-Shot Persona Dialogue Generation with In-Context Learning

Much work has been done to improve persona consistency by finetuning a pretrained dialogue model on high-quality human-annoated persona datasets.However, these methods still face the challenges of high cost and poor scalability.To this end, we propose a simpleyet-effective approach to significantly improve zero-shot persona consistency via in-context learning.Specifically, we first pre-train a persona-augmented dialogue generation model and then utilize in-context prompting mechanism to realize zero-shot persona customization.Experimental results demonstrate that our method can dramatically improve persona consistency without compromising coherence and informativeness in zero-shot settings.
Scale-Invariant Infinite Hierarchical Topic Model

Hierarchical topic models have been employed to organize a large number of diverse topics from corpora into a latent tree structure. However, existing models yield fragmented topics with overlapping themes whose expected probability becomes exponentially smaller along the depth of the tree.To solve this intrinsic problem, we propose a scale-invariant infinite hierarchical topic model (ihLDA). The ihLDA adaptively adjusts the topic creation to make the expected topic probability decay considerably slower than that in existing models. Thus, it facilitates the estimation of deeper topic structures encompassing diverse topics in a corpus. Furthermore, the ihLDA extends a widely used tree-structured prior (Adams et al., 2010) in a hierarchical Bayesian way, which enables drawing an infinite topic tree from the base tree while efficiently sampling the topic assignments for the words.Experiments demonstrate that the ihLDA has better topic uniqueness and hierarchical diversity thanexisting approaches, including state-of-the-art neural models.
An Invariant Learning Characterization of Controlled Text Generation

Controlled generation refers to the problem of creating text that contains stylistic or semantic attributes of interest.Many approaches reduce this problem to training a predictor of the desired attribute.For example, researchers hoping to deploy a large language model to produce non-toxic content may use a toxicity classifier to filter generated text.In practice, the generated text to classify, which is determined by user prompts, may come from a wide range of distributions.In this paper, we show that the performance of controlled generation may be poor if the distributions of text in response to user prompts differ from the distribution the predictor was trained on.To address this problem, we cast controlled generation under distribution shift as an invariant learning problem: the most effective predictor should be invariant across multiple text environments.We then discuss a natural solution that arises from this characterization and propose heuristics for selecting natural environments.We study this characterization and the proposed method empirically using both synthetic and real data.Experiments demonstrate both the challenge of distribution shift in controlled generation and the potential of invariance methods in this setting.
Aristoxenus at SemEval-2023 Task 4: A Domain-Adapted Ensemble Approach to the Identification of Human Values behind Arguments

This paper presents our system for the SemEval-2023 Task 4, which aims to identify human values behind arguments by classifying whether or not an argument draws on a specific category. Our approach leverages a second-phase pre-training method to adapt a RoBERTa Language Model (LM) and tackles the problem using a One-Versus-All strategy. Final predictions are determined by a majority voting module that combines the outputs of an ensemble of three sets of per-label models. We conducted experiments to evaluate the impact of different pre-trained LMs on the task, comparing their performance in both pre-trained and task-adapted settings. Our findings show that fine-tuning the RoBERTa LM on the task-specific dataset improves its performance, outperforming the best-performing baseline BERT approach. Overall, our approach achieved a macro-F1 score of 0.47 on the official test set, demonstrating its potential in identifying human values behind arguments.
Learning to Substitute Spans towards Improving Compositional Generalization

Despite the rising prevalence of neural sequence models, recent empirical evidences suggest their deficiency in compositional generalization. One of the current de-facto solutions to this problem is compositional data augmentation, aiming to incur additional compositional inductive bias. Nonetheless, the improvement offered by existing handcrafted augmentation strategies is limited when successful systematic generalization of neural sequence models requires multi-grained compositional bias (i.e., not limited to either lexical or structural biases only) or differentiation of training sequences in an imbalanced difficulty distribution. To address the two challenges, we first propose a novel compositional augmentation strategy dubbed Span Substitution (SpanSub) that enables multi-grained composition of substantial substructures in the whole training set. Over and above that, we introduce the Learning to Substitute Span (L2S2) framework which empowers the learning of span substitution probabilities in SpanSub in an end-to-end manner by maximizing the loss of neural sequence models, so as to outweigh those challenging compositions with elusive concepts and novel surroundings. Our empirical results on three standard compositional generalization benchmarks, including SCAN, COGS and GeoQuery (with an improvement of at most 66.5%, 10.3%, 1.2%, respectively), demonstrate the superiority of SpanSub, L2S2 and their combination.
“Geen makkie”: Interpretable Classification and Simplification of Dutch Text Complexity

An inclusive society needs to facilitate access to information for all of its members, including citizens with low literacy and with non-native language skills.We present an approach to assess Dutch text complexity on the sentence level and conduct an interpretability analysis to explore the link between neural models and linguistic complexity features. 1Building on these findings, we develop the first contextual lexical simplification model for Dutch and publish a pilot dataset for evaluation.We go beyond previous work which primarily targeted lexical substitution and propose strategies for adjusting the model's linguistic register to generate simpler candidates.Our results indicate that continual pre-training and multi-task learning with conceptually related tasks are promising directions for ensuring the simplicity of the generated substitutions.Our code repository and the simplification dataset are available on GitHub.