Reviewerly/exHarmony huggingface dataset:
DUB: Discrete Unit Back-translation for Speech Translation

How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST.Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation(DUB) to answer two questions (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://anonymous.4open.science/r/DUB/.
Building Accurate Low Latency ASR for Streaming Voice Search in E-commerce

Automatic Speech Recognition (ASR) is essential for any voice-based application. The streaming capability of ASR becomes necessary to provide immediate feedback to the user in applications like Voice Search. LSTM/RNN and CTC based ASR systems are very simple to train and deploy for low latency streaming applications but have lower accuracy when compared to the state-of-the-art models. In this work, we build accurate LSTM, attention and CTC based streaming ASR models for large-scale Hinglish (blend of Hindi and English) Voice Search. We evaluate how various modifications in vanilla LSTM training improve the system’s accuracy while preserving the streaming capabilities. We also discuss a simple integration of end-of-speech (EOS) detection with CTC models, which helps reduce the overall search latency. Our model achieves a word error rate (WER) of 3.69% without EOS and 4.78% with EOS, with ~1300 ms (~46.64%) reduction in latency.
On the Role of Parallel Data in Cross-lingual Transfer Learning

While prior work has established that the use of parallel data is conducive for cross-lingual learning, it is unclear if the improvements come from the data itself, or if it is the modeling of parallel interactions that matters. Exploring this, we examine the usage of unsupervised machine translation to generate synthetic parallel data, and compare it to supervised machine translation and gold parallel data. We find that even model generated parallel data can be useful for downstream tasks, in both a general setting (continued pretraining) as well as the task-specific setting (translate-train), although our best results are still obtained using real parallel data. Our findings suggest that existing multilingual models do not exploit the full potential of monolingual data, and prompt the community to reconsider the traditional categorization of cross-lingual learning approaches.
Attention as a Guide for Simultaneous Speech Translation

In simultaneous speech translation (SimulST), effective policies that determine when to write partial translations are crucial to reach high output quality with low latency.Towards this objective, we propose EDATT (Encoder-Decoder Attention), an adaptive policy that exploits the attention patterns between audio source and target textual translation to guide an offlinetrained ST model during simultaneous inference.EDATT exploits the attention scores modeling the audio-translation relation to decide whether to emit a partial hypothesis or wait for more audio input.This is done under the assumption that, if attention is focused towards the most recently received speech segments, the information they provide can be insufficient to generate the hypothesis (indicating that the system has to wait for additional audio input).Results on en→{de, es} show that EDATT yields better results compared to the SimulST state of the art, with gains respectively up to 7 and 4 BLEU points for the two languages, and with a reduction in computational-aware latency up to 1.4s and 0.7s compared to existing SimulST policies applied to offline-trained models.
On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning

Incorporating contrastive learning objectives in sentence representation learning (SRL) has yielded significant improvements on many sentence-level NLP tasks.However, it is not well understood why contrastive learning works for learning sentence-level semantics.In this paper, we aim to help guide future designs of sentence representation learning methods by taking a closer look at contrastive SRL through the lens of isotropy, contextualization and learning dynamics.We interpret its successes through the geometry of the representation shifts and show that contrastive learning brings isotropy, and drives high intra-sentence similarity: when in the same sentence, tokens converge to similar positions in the semantic space.We also find that what we formalize as "spurious contextualization" is mitigated for semantically meaningful tokens, while augmented for functional ones.We find that the embedding space is directed towards the origin during training, with more areas now better defined.We ablate these findings by observing the learning dynamics with different training temperatures, batch sizes and pooling methods.
I2C Huelva at SemEval-2023 Task 4: A Resampling and Transformers Approach to Identify Human Values behind Arguments

This paper presents the approaches proposedfor I2C Group to address the SemEval-2023Task 4: Identification of Human Values behindArguments (ValueEval)”, whose goal is to classify 20 different categories of human valuesgiven a textual argument. The dataset of thistask consists of one argument per line, including its unique argument ID, conclusion, stanceof the premise towards the conclusion and thepremise text. To indicate whether the argumentdraws or not on that category a binary indication (1 or 0) is included. Participants can submit approaches that detect one, multiple, or allof these values in arguments. The task providesan opportunity for researchers to explore theuse of automated techniques to identify humanvalues in text and has potential applications invarious domains such as social science, politics,and marketing. To deal with the imbalancedclass distribution given, our approach undersamples the data. Additionally, the three components of the argument (conclusion, stanceand premise) are used for training. The systemoutperformed the BERT baseline according toofficial evaluation metrics, achieving a f1 scoreof 0.46.
A Probabilistic Framework for Discovering New Intents

Discovering new intents is of great significance for establishing the Task-Oriented Dialogue System.Most prevailing approaches either cannot transfer prior knowledge inherent in known intents or fall into the dilemma of forgetting prior knowledge in the follow-up.Furthermore, such approaches fail to thoroughly explore the inherent structure of unlabeled data, thereby failing to capture the fundamental characteristics that define an intent in general sense.In this paper, starting from the intuition that discovering intents should be beneficial for identifying known intents, we propose a probabilistic framework for discovering intents where intent assignments are treated as latent variables.We adopt the Expectation Maximization framework for optimization.Specifically, In the Estep, we conduct intent discovery and explore the intrinsic structure of unlabeled data by the posterior of intent assignments.In the M-step, we alleviate the forgetting of prior knowledge transferred from known intents by optimizing the discrimination of labeled data.Extensive experiments conducted on three challenging real-world datasets demonstrate the generality and effectiveness of the proposed framework and implementation.Codes is publicly available.
Evaluating Zero-Shot Event Structures: Recommendations for Automatic Content Extraction (ACE) Annotations

Zero-shot event extraction (EE) methods infer richly structured event records from text, based only on a minimal user specification and no training examples, which enables flexibility in exploring and developing applications.Most event extraction research uses the Automatic Content Extraction (ACE) annotated dataset to evaluate supervised EE methods, but can it be used to evaluate zero-shot and other low-supervision EE?We describe ACE's event structures and identify significant ambiguities and issues in current evaluation practice, including (1) coreferent argument mentions, (2) conflicting argument head conventions, and (3) ignorance of modality and event class details.By sometimes mishandling these subtleties, current work may dramatically understate the actual performance of zero-shot and other lowsupervision EE, considering up to 32% of correctly identified arguments and 25% of correctly ignored event mentions as false negatives.For each issue, we propose recommendations for future evaluations so the research community can better utilize ACE as an event evaluation resource.
Translation-Enhanced Multilingual Text-to-Image Generation

Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated imagecaption data in other languages; in the long run, this might widen inequitable access to TTI technology.In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems.We provide two key contributions.1) Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: TRANSLATE TRAIN, TRANS-LATE TEST, and ZERO-SHOT TRANSFER.2) We propose Ensemble Adapter (ENSAD), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance.Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of translation-enhanced mTTI systems and also validate the benefits of the proposed EN-SAD which derives consistent gains across all datasets.Further investigations on model variants, ablation studies, and qualitative analyses provide additional insights on the inner workings of the proposed mTTI approaches.
FashionKLIP: Enhancing E-Commerce Image-Text Retrieval with Fashion Multi-Modal Conceptual Knowledge Graph

Image-text retrieval is a core task in the multimodal domain, which arises a lot of attention from both research and industry communities.Recently, the booming of visionlanguage pre-trained (VLP) models has greatly enhanced the performance of cross-modal retrieval.However, the fine-grained interactions between objects from different modalities are far from well-established.This issue becomes more severe in the e-commerce domain, which lacks sufficient training data and fine-grained cross-modal knowledge.To alleviate the problem, this paper proposes a novel e-commerce knowledge-enhanced VLP model FashionKLIP.We first automatically establish a multi-modal conceptual knowledge graph from large-scale e-commerce image-text data, and then inject the prior knowledge into the VLP model to align across modalities at the conceptual level.The experiments conducted on a public benchmark dataset demonstrate that Fash-ionKLIP effectively enhances the performance of e-commerce image-text retrieval upon stateof-the-art VLP models by a large margin.The application of the method in real industrial scenarios also proves the feasibility and efficiency of FashionKLIP. 1
Improving Low Resource Speech Translation with Data Augmentation and Ensemble Strategies

This paper describes the speech translation system submitted as part of the IWSLT 2023 shared task on low resource speech translation.The low resource task aids in building models for language pairs where the training corpus is limited.In this paper, we focus on two language pairs, namely, Tamasheq-French (Tmh→Fra) and Marathi-Hindi (Mr→Hi) and implement a speech translation system that is unconstrained.We evaluate three strategies in our system: (a) Data augmentation where we perform different operations on audio as well as text samples, (b) an ensemble model that integrates a set of models trained using a combination of augmentation strategies, and (c) post-processing techniques where we explore the use of large language models (LLMs) to improve the quality of sentences that are generated.Experiments show how data augmentation can relatively improve the BLEU score by 5.2% over the baseline system for Tmh→Fra while an ensemble model further improves performance by 17% for Tmh→Fra and 23% for Mr→Hi task.
SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages

Text simplification research has mostly focused on sentence-level simplification, even though many desirable edits-such as adding relevant background information or reordering contentmay require document-level context.Prior work has also predominantly framed simplification as a single-step, input-to-output task, only implicitly modeling the fine-grained, span-level edits that elucidate the simplification process.To address both gaps, we introduce the SWIPE dataset, which reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) articles.In contrast to prior work, SWIPE leverages the entire revision history when pairing pages in order to better identify simplification edits.We work with Wikipedia editors to annotate 5,000 EW-SEW document pairs, labeling more than 40,000 edits with proposed 19 categories.To scale our efforts, we propose several models to automatically label edits, achieving an F-1 score of up to 70.6, indicating that this is a tractable but challenging NLU task.Finally, we categorize the edits produced by several simplification models and find that SWIPE-trained models generate more complex edits while reducing unwanted edits.
Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation

The decoder in simultaneous neural machine translation receives limited information from the source while having to balance the opposing requirements of latency versus translation quality.In this paper, we use an auxiliary targetside language model to augment the training of the decoder model.Under this notion of target adaptive training, generating rare or difficult tokens is rewarded which improves the translation quality while reducing latency.The predictions made by a language model in the decoder are combined with the traditional cross entropy loss which frees up the focus on the source side context.Our experimental results over multiple language pairs show that compared to previous state of the art methods in simultaneous translation, we can use an augmented target side context to improve BLEU scores significantly.We show improvements over the state of the art in the low latency range with lower average lagging values (faster output). 1 1 github.com/sfu-natlang/target_rescale_siMT
KG-FLIP: Knowledge-guided Fashion-domain Language-Image Pre-training for E-commerce

Various Vision-Language Pre-training (VLP) models (e.g., CLIP, BLIP) have sprung up and dramatically improved the benchmarks of public general-domain datasets (e.g., COCO, Flickr30k).Such models typically learn the cross-modal alignment from large-scale wellaligned image-text datasets.Adapting these models to downstream applications in specific domains, such as fashion, requires fine-grained in-domain image-text datasets.However, such datasets are usually less semantically aligned and smaller in scale, which requires more efficient pre-training strategies.In this paper, we propose a knowledge-guided fashion-domain language-image pre-training (KG-FLIP) framework that focuses on learning fine-grained representations in the e-commerce domain and utilizes external knowledge (i.e., product attribute schema) to improve the pre-training efficiency.Experimental results demonstrate that KG-FLIP outperforms previous state-of-the-art VLP models on Amazon data and the Fashion-Gen dataset by large margins.KG-FLIP has been successfully deployed in the Amazon catalog system to backfill missing attributes and improve the customer shopping experience.
Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning

Prompting has gained tremendous attention as an efficient method for the adaptation of largescale language models.However, prompts often act against human intuition and report unstable performances, which has motivated methods that automatically find effective prompts.One popular approach is gradient-based search, which iteratively updates a (randomly) initialized prompt towards the optimal one with the guide of gradients.We propose a novel regularization method, CoRe, for gradient-based prompt tuning techniques, which guides a prompt to produce a task context properly.CoRe realizes two regularization effects -context attuning and context filtering -that improve prediction performance in a zero-shot incontext learning setting where a model makes inferences only with the prompt tuned by CoRe, without any demonstration examples for incontext learning.Context attuning guides the context generated by the input and the tuned prompt toward embedding the appropriate context for the task.In our theoretical analysis, regularizing the context extends to improving zeroshot in-context learning performance.Context filtering steers the prompt to select only the task-related context so that context attuning solely focuses on creating and sending the right task context.We evaluate CoRe on natural language understanding datasets and two large language models, GPT2-XL and GPT-J.Our training scheme shows performance improvements up to 11.9% on GPT2-XL, and up to 6.3% on GPT-J in zero-shot settings.
Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection

Out-of-distribution (OOD) detection is a critical task for reliable predictions over text.Finetuning with pre-trained language models has been a de facto procedure to derive OOD detectors with respect to in-distribution (ID) data.Despite its common use, the understanding of the role of fine-tuning and its necessity for OOD detection is largely unexplored.In this paper, we raise the question: is fine-tuning necessary for OOD detection?We present a study investigating the efficacy of directly leveraging pre-trained language models for OOD detection, without any model fine-tuning on the ID data.We compare the approach with several competitive fine-tuning objectives, and offer new insights under various types of distributional shifts.Extensive evaluations on 8 diverse ID-OOD dataset pairs demonstrate nearperfect OOD detection performance (with 0% FPR95 in many cases), strongly outperforming its fine-tuned counterparts.We show that using distance-based detection methods, pretrained language models are near-perfect OOD detectors when the distribution shift involves a domain change.Furthermore, we study the effect of fine-tuning on OOD detection and identify how to balance ID accuracy with OOD detection performance.Our code is publically available 1 .
Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning

Parameter-efficient tuning methods (PETs) have achieved promising results in tuning large pre-trained language models (PLMs).By formalizing frozen PLMs and additional tunable parameters as systems and controls respectively, PETs can be theoretically grounded to optimal control and further viewed as optimizing the terminal cost and running cost in the optimal control literature.Despite the elegance of this theoretical grounding, in practice, existing PETs often ignore the running cost and only optimize the terminal cost, i.e., focus on optimizing the loss function of the output state, regardless of the running cost that depends on the intermediate states.Since it is nontrivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and use the regularization as the running cost of PETs.As the first work to propose regularized PETs that use stochastic bridges as the regularizers (running costs) for the intermediate states, we show the effectiveness and generality of this regularization across different tasks, PLMs and PETs.In view of the great potential and capacity, we believe more sophisticated regularizers can be designed for PETs and better performance can be achieved in the future.The code is released at https://github.com/thunlp/stochastic-bridge-pet/tree/main.
Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations

Abductive reasoning aims to find plausible explanations for an event.This style of reasoning is critical for commonsense tasks where there are often multiple plausible explanations.Existing approaches for abductive reasoning in natural language processing (NLP) often rely on manually generated annotations for supervision; however, such annotations can be subjective and biased.Instead of using direct supervision, this work proposes an approach for abductive commonsense reasoning that exploits the fact that only a subset of explanations is correct for a given context.The method uses posterior regularization to enforce a mutual exclusion constraint, encouraging the model to learn the distinction between fluent explanations and plausible ones.We evaluate our approach on a diverse set of abductive reasoning datasets; experimental results show that our approach outperforms or is comparable to directly applying pretrained language models in a zeroshot manner and other knowledge-augmented zero-shot methods.
Transformer-based Hebrew NLP models for Short Answer Scoring in Biology

Pre-trained large language models (PLMs) are adaptable to a wide range of downstream tasks by fine-tuning their rich contextual embeddings to the task, often without requiring much taskspecific data.In this paper, we explore the use of a recently developed Hebrew PLM -aleph-BERT -for automated short answer grading of high school biology items.We show that the alephBERT-based system outperforms a strong CNN-based baseline, and that it generalizes unexpectedly well in a zero-shot paradigm to items on an unseen topic that address the same underlying biological concepts, opening up the possibility of automatically assessing new items without item-specific fine-tuning.
GMU Systems for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks

This paper describes the GMU Systems for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks.We submitted systems for five low-resource tasks and the dialectal task.In this work, we explored self-supervised pretrained speech models and finetuned them on speech translation downstream tasks.We use the Wav2vec 2.0, XLSR-53, and Hubert as selfsupervised models.Unlike Hubert, Wav2vec 2.0 and XLSR-53 achieve the best results when we remove the top three layers.Our results show that Wav2vec 2.0 and Hubert perform similarly with their relative best configuration.In addition, we found that Wav2vec 2.0 pretrained on audio data of the same language as the source language of a speech translation model achieves better results.For the lowresource setting, the best results are achieved using either the Wav2vec 2.0 or Hubert models, while XLSR-53 achieves the best results for the dialectal transfer task.We find that XLSR-53 does not perform well for low-resource tasks.Using Wav2vec 2.0, we report close to 2 BLEU point improvements on the test set for the Tamasheq-French compared to the baseline system at the IWSLT 2022.