Reviewerly/exHarmony huggingface dataset:
Semantic Accuracy in Natural Language Generation: A Thesis Proposal

With the fast-growing popularity of current large pre-trained language models (LLMs), it is necessary to dedicate efforts to making them more reliable.In this thesis proposal, we aim to improve the reliability of natural language generation systems (NLG) by researching the semantic accuracy of their outputs.We look at this problem from the outside (evaluation) and from the inside (interpretability).We propose a novel method for evaluating semantic accuracy and discuss the importance of working towards a unified and objective benchmark for NLG metrics.We also review interpretability approaches which could help us pinpoint the sources of inaccuracies within the models and explore potential mitigation strategies.
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation

Mixture-of-Expert (MoE) models have obtained state-of-the-art performance in Neural Machine Translation (NMT) tasks.Existing works in MoE mostly consider a homogeneous design where the same number of experts of the same size are placed uniformly throughout the network.Furthermore, existing MoE works do not consider computational constraints (e.g., FLOPs, latency) to guide their design.To this end, we develop AutoMoE -a framework for designing heterogeneous MoE's under computational constraints.AutoMoE leverages Neural Architecture Search (NAS) to obtain efficient sparse MoE sub-transformers with 4× inference speedup (CPU) and FLOPs reduction over manually designed Transformers, with parity in BLEU score over dense Transformer and within 1 BLEU point of MoE SwitchTransformer, on aggregate over benchmark datasets for NMT.Heterogeneous search space with dense and sparsely activated Transformer modules (e.g., how many experts?where to place them?what should be their sizes?)allows for adaptive compute -where different amounts of computations are used for different tokens in the input.Adaptivity comes naturally from routing decisions which send tokens to experts of different sizes.AutoMoE code, data, and trained models are available at https://aka.ms/AutoMoE.
Two-step Text Summarization for Long-form Biographical Narrative Genre

Transforming narrative structure to implicit discourse relations in long-form text has recently seen a mindset shift toward assessing generation consistency.To this extent, summarization of lengthy biographical discourse is of practical benefit to readers, as it helps them decide whether immersing for days or weeks in a bulky book turns a rewarding experience.Machine-generated summaries can reduce the cognitive load and the time spent by authors to write the summary.Nevertheless, summarization faces significant challenges of factual inconsistencies with respect to the inputs.In this paper, we explored a two-step summary generation aimed to retain source-summary faithfulness.Our method uses a graph representation to rank sentence saliency in each of the novel chapters, leading to distributing summary segments in distinct regions of the chapter.Basing on the previously extracted sentences we produced an abstractive summary in a manner more computationally tractable for detecting inconsistent information.We conducted a series of quantitative analyses on a test set of four long biographical novels and showed to improve summarization quality in automatic evaluation over both single-tier settings and external baselines.
Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages

Vision-Language Pre-training (VLP) has advanced the performance of many visionlanguage tasks, such as image-text retrieval, visual entailment, and visual reasoning.The pre-training mostly utilizes lexical databases and image queries in English.Previous work has demonstrated that the pre-training in English does not transfer well to other languages in a zero-shot setting.However, multilingual pre-trained language models (MPLM) have excelled at a variety of single-modal language tasks.In this paper, we propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM.We utilize a cross-lingual contextualized token embeddings alignment approach to train text encoders for non-English languages.Our approach does not require image input and primarily uses machine translation, eliminating the need for target language data.Our evaluation across three distinct tasks (image-text retrieval, visual entailment, and natural language visual reasoning) demonstrates that this approach outperforms the state-of-the-art multilingual vision-language models without requiring large parallel corpora.Our code is available at https://github.com/Yasminekaroui/CliCoTea.
Concept2Box: Joint Geometric Embeddings for Learning Two-View Knowledge Graphs

Zijie Huang, Daheng Wang, Binxuan Huang, Chenwei Zhang, Jingbo Shang, Yan Liang, Zhengyang Wang, Xian Li, Christos Faloutsos, Yizhou Sun, Wei Wang. Findings of the Association for Computational Linguistics: ACL 2023. 2023.
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors

Obtaining labeled data to train a model for a task of interest is often expensive.Prior work shows training models on multitask data augmented with task descriptions (prompts) effectively transfers knowledge to new tasks.Towards efficiently building task-specific models, we assume access to a small number (32-1000) of unlabeled target-task examples and use those to retrieve the most similar labeled examples from a large pool of multitask data augmented with prompts.Compared to the current practice of finetuning models on uniformly sampled prompted multitask data (e.g., FLAN, T0), our approach of finetuning on cross-task nearest neighbors is significantly more data-efficient.Using only 2% of the data from the P3 pool without any labeled target-task data, our models outperform strong baselines trained on all available data by 3-30% on 12 out of 14 datasets representing held-out tasks including legal and scientific document QA.Similarly, models trained on cross-task nearest neighbors from SuperNat-uralInstructions, representing about 5% of the pool, obtain comparable performance to stateof-the-art models on 12 held-out tasks from that pool.Moreover, the models produced by our approach also provide a better initialization than single multitask finetuned models for fewshot finetuning on target-task data, as shown by a 2-23% relative improvement over fewshot finetuned T0-3B models on 8 datasets.We publicly release our code. 1
OPI PIB at SemEval-2023 Task 1: A CLIP-based Solution Paired with an Additional Word Context Extension

This article presents our solution for SemEval-2023 Task 1: Visual Word Sense Disambiguation.The aim of the task was to select the most suitable from a list of ten images for a given word, extended by a small textual context.Our solution comprises two parts.The first focuses on an attempt to further extend the textual context, based on word definitions contained in WordNet and in Open English WordNet.The second focuses on selecting the most suitable image using the CLIP model with previously developed word context and additional information obtained from the BEiT image classification model.Our solution allowed us to achieve a result of 70.84% on the official test dataset for the English language.
Adaptive Attention for Sparse-based Long-sequence Transformer

Recently, Transformers have been widely used in various fields and have achieved remarkable results. But it is still difficult for Transformer-based models to process longer sequences because self-attention in them scales quadratically with the sequence length. Although some models attempt to use sparse attention to reduce computational complexity, hand-crafted attention patterns are unable to select useful tokens adaptively according to the context. Thus, in this paper, we propose a novel efficient Transformer model with adaptive attention, A2-Former, for long sequence modeling. It can select useful tokens automatically in sparse attention by learnable position vectors, which consist of meta position and offset position vectors. Because the learnable offset position is not an integer vector, we utilize the interpolation technique to gather corresponding vectors from the input embedding matrix by discrete indexes. Experiments on Long Range Arena (LRA), a systematic and unified benchmark with different tasks, show that our model has achieved further improvement in performance compared with other sparse-based Transformers.
Cross-lingual AMR Aligner: Paying Attention to Cross-Attention

This paper introduces a novel aligner for Abstract Meaning Representation (AMR) graphs that can scale cross-lingually, and is thus capable of aligning units and spans in sentences of different languages.Our approach leverages modern Transformer-based parsers, which inherently encode alignment information in their cross-attention weights, allowing us to extract this information during parsing.This eliminates the need for English-specific rules or the Expectation Maximization (EM) algorithm that have been used in previous approaches.In addition, we propose a guided supervised method using alignment to further enhance the performance of our aligner.We achieve state-of-the-art results in the benchmarks for AMR alignment and demonstrate our aligner's ability to obtain them across multiple languages.Our code will be available at github.com/Babelscape/AMR-alignment.
VisText: A Benchmark for Semantically Rich Chart Captioning

Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities.However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns).In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts' construction, report key statistics, and identify perceptual and cognitive phenomena.In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph -a hierarchical representation of a chart's visual elements akin to a web page's Document Object Model (DOM).To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey.Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics.Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work.
Gradient-based Intra-attention Pruning on Pre-trained Language Models

Pre-trained language models achieve superior performance but are computationally expensive.Techniques such as pruning and knowledge distillation have been developed to reduce their sizes and latencies.In this work, we propose a structured pruning method GRAIN (Gradientbased Intra-attention pruning), which performs task-specific pruning with knowledge distillation and yields highly effective models.Different from common approaches that prune each attention head as a whole, GRAIN inspects and prunes intra-attention structures, which greatly expands the structure search space and enables more flexible models.We also propose a gradient separation strategy that reduces the interference of distillation on pruning for a better combination of the two approaches.Experiments on GLUE, SQuAD, and CoNLL 2003 show that GRAIN notably outperforms other methods, especially in the high sparsity regime, and achieves 6 ∼ 7× speedups while maintaining 93% ∼ 99% performance.Under extreme compression where only 3% transformer weights remain, the pruned model is still competitive compared to larger models.
NAISTeacher: A Prompt and Rerank Approach to Generating Teacher Utterances in Educational Dialogues

This paper presents our approach to the BEA 2023 shared task of generating teacher responses in educational dialogues, using the Teacher-Student Chatroom Corpus.Our system prompts GPT-3.5-turbo to generate initial suggestions, which are then subjected to reranking.We explore multiple strategies for candidate generation, including prompting for multiple candidates and employing iterative few-shot prompts with negative examples.We aggregate all candidate responses and rerank them based on DialogRPT scores.To handle consecutive turns in the dialogue data, we divide the task of generating teacher utterances into two components: teacher replies to the student and teacher continuations of previously sent messages.Through our proposed methodology, our system achieved the top score on both automated metrics and human evaluation, surpassing the reference human teachers on the latter.
Causes and Cures for Interference in Multilingual Translation

Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference.While there is a growing number of sophisticated methods that aim to eliminate interference, our understanding of interference as a phenomenon is still limited.This work identifies the main factors that contribute to interference in multilingual machine translation.Through systematic experimentation, we find that interference (or synergy) are primarily determined by model size, data size, and the proportion of each language pair within the total dataset.We observe that substantial interference occurs mainly when the model is very small with respect to the available training data, and that using standard transformer configurations with less than one billion parameters largely alleviates interference and promotes synergy.Moreover, we show that tuning the sampling temperature to control the proportion of each language pair in the data is key to balancing the amount of interference between low and high resource language pairs effectively, and can lead to superior performance overall.
WACO: Word-Aligned Contrastive Learning for Speech Translation

End-to-end Speech Translation (E2E ST) aims to directly translate source speech into target text.Existing ST methods perform poorly when only extremely small speech-text data are available for training.We observe that an ST model's performance closely correlates with its embedding similarity between speech and source transcript.In this paper, we propose Word-Aligned COntrastive learning (WACO), a simple and effective method for extremely low-resource speech-to-text translation.Our key idea is bridging word-level representations for both speech and text modalities via contrastive learning.We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark, and on a low-resource direction Maltese-English from IWSLT 2023.Our experiments demonstrate that WACO outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data.Code is available at
MANER: Mask Augmented Named Entity Recognition for Extreme Low-Resource Languages

This paper investigates the problem of Named Entity Recognition (NER) for extreme lowresource languages with only a few hundred tagged data samples.A critical enabler of most of the progress in NER is the readily available, large-scale training data for languages such as English and French.However, NER for lowresource languages remains relatively underexplored, leaving much room for improvement.We propose Mask Augmented Named Entity Recognition (MANER), a simple yet effective method that leverages the distributional hypothesis of pre-trained masked language models (MLMs) to improve NER performance for lowresource languages significantly.MANER repurposes the [mask] token in MLMs, which encodes valuable semantic contextual information, for NER prediction.Specifically, we prepend a [mask] token to every word in a sentence and predict the named entity for each word from its preceding [mask] token.We demonstrate that MANER is well-suited for NER in low-resource languages; our experiments show that for 100 languages with as few as 100 training examples, it improves on the state-of-the-art by up to 48% and by 12% on average on F1 score.We also perform detailed analyses and ablation studies to understand the scenarios that are best suited to MANER.
BUT Systems for IWSLT 2023 Marathi - Hindi Low Resource Speech Translation Task

This paper describes the systems submitted for Marathi to Hindi low-resource speech translation task.Our primary submission is based on an end-to-end direct speech translation system, whereas the contrastive one is a cascaded system.The backbone of both the systems is a Hindi-Marathi bilingual ASR system trained on 2790 hours of imperfect transcribed speech.The end-to-end speech translation system was directly initialized from the ASR, and then finetuned for direct speech translation with an auxiliary CTC loss for translation.The MT model for the cascaded system is initialized from a cross-lingual language model, which was then fine-tuned using 1.6 M parallel sentences.All our systems were trained from scratch on publicly available datasets.In the end, we use a language model to re-score the n-best hypotheses.Our primary submission achieved 30.5 and 39.6 BLEU whereas the contrastive system obtained 21.7 and 28.6 BLEU on official dev and test sets respectively.The paper also presents the analysis on several experiments that were conducted and outlines the strategies for improving speech translation in low-resource scenarios.
Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding

Leveraging representations from pre-trained transformer-based encoders achieves state-ofthe-art performance on numerous NLP tasks.Larger encoders can improve accuracy for spoken language understanding (SLU) but are challenging to use given the inference latency constraints of online systems (especially on CPU machines).We evaluate using a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU compared to using smaller 17M parameter BERT encoders with language-, domainand task-decoupled finetuning.Running inference with a larger shared encoder on GPU is latency neutral and reduces infrastructure cost compared to running inference for decoupled smaller encoders on CPU machines.The larger shared encoder reduces semantic error rates by 4.62% for test sets representing user requests to voice-controlled devices and 5.79% on the tail of the test sets on average across four languages.
Lauri Ingman at SemEval-2023 Task 4: A Chain Classifier for Identifying Human Values behind Arguments

Identifying expressions of human values in textual data is a crucial albeit complicated challenge, not least because ethics are highly variable, often implicit, and transcend circumstance.Opinions, arguments, and the like are generally founded upon more than one guiding principle, which are not necessarily independent.As such, little is known about how to classify and predict moral undertones in natural language sequences.Here, we describe and present a solution to ValueEval, our shared contribution to SemEval 2023 Task 4. Our research design focuses on investigating chain classifier architectures with pretrained contextualized embeddings to detect 20 different human values in written arguments.We show that our best model substantially surpasses the classification performance of the baseline method established in prior work.We discuss limitations to our approach and outline promising directions for future work.
Efficient Out-of-Domain Detection for Sequence to Sequence Models

Sequence-to-sequence (seq2seq) models based on the Transformer architecture have become a ubiquitous tool applicable not only to classical text generation tasks such as machine translation and summarization but also to any other task where an answer can be represented in a form of a finite text fragment (e.g., question answering).However, when deploying a model in practice, we need not only high performance but also an ability to determine cases where the model is not applicable.Uncertainty estimation (UE) techniques provide a tool for identifying out-of-domain (OOD) input where the model is susceptible to errors.State-of-the-art UE methods for seq2seq models rely on computationally heavyweight and impractical deep ensembles.In this work, we perform an empirical investigation of various novel UE methods for large pre-trained seq2seq models T5 and BART on three tasks: machine translation, text summarization, and question answering.We apply computationally lightweight density-based UE methods to seq2seq models and show that they often outperform heavyweight deep ensembles on the task of OOD detection 1 .
On the Strength of Sequence Labeling and Generative Models for Aspect Sentiment Triplet Extraction

Generative models have achieved great success in aspect sentiment triplet extraction tasks.However, existing methods ignore the mutual informative clues between aspect and opinion terms and may generate false paired triplets.Furthermore, the inherent limitations of generative models, i.e., the token-by-token decoding and the simple structured prompt, prevent models from handling complex structures especially multi-word terms and multi-triplet sentences.To address these issues, we propose a sequence labeling enhanced generative model.Firstly, we encode the dependency between aspect and opinion into two bidirectional templates to avoid false paired triplets.Secondly, we introduce a marker-oriented sequence labeling module to improve generative models' ability of tackling complex structures.Specifically, this module enables the generative model to capture the boundary information of aspect/opinion spans and provides hints to decode multiple triplets with the shared marker.Experimental results on four datasets prove that our model yields a new state-of-art performance.Our code and data are available at https://github.com/NLPWM-WHU/SLGM.
Minimalist Entity Disambiguation for Mid-Resource Languages

For many languages and applications, even though enough data is available for training Named Entity Disambiguation (NED) systems, few off-the-shelf models are available for use in practice.This is due to both the large size of state-of-the-art models, and to the computational requirements for recreating them from scratch.However, we observe that in practice, acceptable models can be trained and run with far fewer resources.In this work, we introduce MiniNED, a framework for creating small NED models from medium-sized datasets.The resulting models can be tuned for applicationspecific objectives and trade-offs, depending on practitioners' requirements concerning model size, frequency bias, and out-of-domain generalization.We evaluate the framework in nine languages, and achieve reasonable performance using models that are a fraction of the size of recent work.