Reviewerly/exHarmony huggingface dataset:
NAVER LABS Europe’s Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track

This paper presents NAVER LABS Europe’s systems for Tamasheq-French and Quechua-Spanish speech translation in the IWSLT 2023 Low-Resource track. Our work attempts to maximize translation quality in low-resource settings using multilingual parameter-efficient solutions that leverage strong pre-trained models. Our primary submission for Tamasheq outperforms the previous state of the art by 7.5 BLEU points on the IWSLT 2022 test set, and achieves 23.6 BLEU on this year’s test set, outperforming the second best participant by 7.7 points. For Quechua, we also rank first and achieve 17.7 BLEU, despite having only two hours of translation data. Finally, we show that our proposed multilingual architecture is also competitive for high-resource languages, outperforming the best unconstrained submission to the IWSLT 2021 Multilingual track, despite using much less training data and compute.
Multi-Domain Dialogue State Tracking with Disentangled Domain-Slot Attention

As the core of task-oriented dialogue systems, dialogue state tracking (DST) is designed to track the dialogue state through the conversation between users and systems.Multi-domain DST has been an important challenge in which the dialogue states across multiple domains need to consider.In recent mainstream approaches, each domain and slot are aggregated and regarded as a single query feeding into attention with the dialogue history to obtain domain-slot specific representations.In this work, we propose disentangled domain-slot attention for multi-domain dialogue state tracking.The proposed approach disentangles the domain-slot specific information extraction in a flexible and context-dependent manner by separating the query about domains and slots in the attention component.Through a series of experiments on MultiWOZ 2.0 and MultiWOZ 2.4 datasets, we demonstrate that our proposed approach outperforms the standard multi-head attention with aggregated domain-slot query.
Focal Training and Tagger Decouple for Grammatical Error Correction

In this paper, we investigate how to improve tagging-based Grammatical Error Correction models.We address two issues of current tagging-based approaches, label imbalance issue, and tagging entanglement issue.Then we propose to down-weight the loss of correctly classified labels using Focal Loss and decouple the error detection layer from the label tagging layer through an extra self-attention-based matching module.Experiments on three recent Chinese Grammatical Error Correction datasets show that our proposed methods are effective.We further analyze choices of hyper-parameters for Focal Loss and inference tweaking.
Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing

Despite increasing interest in the automatic detection of media frames in NLP, the problem is typically simplified as single-label classification and adopts a topic-like view on frames, evading modelling the broader document-level narrative.In this work, we revisit a widely used conceptualization of framing from the communication sciences which explicitly captures elements of narratives, including conflict and its resolution, and integrate it with the narrative framing of key entities in the story as heroes, victims or villains.We adapt an effective annotation paradigm that breaks a complex annotation task into a series of simpler binary questions, and present an annotated data set of English news articles, and a case study on the framing of climate change in articles from news outlets across the political spectrum.Finally, we explore automatic multi-label prediction of our frames with supervised and semisupervised approaches, and present a novel retrieval-based method which is both effective and transparent in its predictions.We conclude with a discussion of opportunities and challenges for future work on document-level models of narrative framing. 1
Brooke-English at SemEval-2023 Task 5: Clickbait Spoiling

The task of clickbait spoiling is: generating a short text that satisfies the curiosity induced by a clickbait post. Clickbait links to a web page and advertises its contents by arousing curiosity instead of providing an informative summary. Previous studies on clickbait spoiling has shown the approach that classifing the type of spoilers is needed, then generating the appropriate spoilers is more effective on the Webis Clickbait Spoiling Corpus 2022 dataset. Our contribution focused on study of the three classes (phrase, passage and multi) and finding appropriate models to generate spoilers foreach class. Results were analysed in each type of spoilers, revealed some reasons of having diversed results in different spoiler types. “passage” type spoiler was identified as the most difficult and the most valuable type of spoiler.
NLUBot101 at SemEval-2023 Task 3: An Augmented Multilingual NLI Approach Towards Online News Persuasion Techniques Detection

We describe our submission to SemEval 2023 Task 3, specifically the subtask on persuasion technique detection.In this work, our team tackled a novel task of classifying persuasion techniques in online news articles at a paragraph level.The low-resource multilingual datasets, along with the imbalanced label distribution, make this task challenging.Our team presented a cross-lingual data augmentation approach and leveraged a recently proposed multilingual natural language inference model to address these challenges.Our solution achieves the highest macro-F1 score for the English task, and top 5 micro-F1 scores on both the English and Russian leaderboards.We have made the source code of our models and experiments publically available at 1 .
Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization

Existing factual consistency evaluation approaches for text summarization provide binary predictions and limited insights into the weakness of summarization systems.Therefore, we propose the task of fine-grained inconsistency detection, the goal of which is to predict the fine-grained types of factual errors in a summary.Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FINEGRAINFACT, which explicitly represents the facts in the documents and summaries with semantic frames extracted by semantic role labeling, and highlights the related semantic frames to predict inconsistency.The highlighted semantic frames help verify predicted error types and correct inconsistent summaries.Experiment results demonstrate that our model outperforms strong baselines and provides evidence to support or refute the summary. 1 Source textMarcy Smith was woken up by her son David to find their house in Glovertown, Newfoundland and Labrador, completely engulfed in flames ... Mrs Smith said if it wasn't for her son, she and her daughter probably wouldn't have survived.David was on FaceTime to his father at the time, so was the only one awake and saw the flames out of the corner of his eye ... Error typeExample summary Extrinsic noun phrase error: Errors that add new object(s), subject(s), or prepositional object(s) that cannot be inferred from the source article.David was using FaceTime with Maggie Smith and saw the flames.Intrinsic noun phrase error: Errors that misrepresent object(s), subject(s), or prepositional object(s) from the source article.David was using FaceTime with Marcy Smith and saw the flames.Extrinsic predicate error: Errors that add new main verb(s) or adverb(s) that cannot be inferred from the source article.David was eating and saw the flames.Intrinsic predicate error: Errors that misrepresent main verb(s) or adverb(s) from the source article.David was engulfed and saw the flames.
OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking

Open-vocabulary state tracking is a more practical version of state tracking that aims to track state changes of entities throughout a process without restricting the state space and entity space.OpenPI (Tandon et al., 2020) is to date the only dataset annotated for open-vocabulary state tracking.However, we identify issues with the dataset quality and evaluation metric.For the dataset, we categorize 3 types of problems on the procedure level, step level and state change level respectively, and build a clean dataset OpenPI-C using multiple rounds of human judgment.For the evaluation metric, we propose a cluster-based metric to fix the original metric's preference for repetition.Model-wise, we enhance the seq2seq generation baseline by reinstating two key properties for state tracking: temporal dependency and entity awareness.The state of the world after an action is inherently dependent on the previous state.We model this dependency through a dynamic memory bank and allow the model to attend to the memory slots during decoding.On the other hand, the state of the world is naturally a union of the states of involved entities.Since the entities are unknown in the open-vocabulary setting, we propose a two-stage model that refines the state change prediction conditioned on entities predicted from the first stage.Empirical results show the effectiveness of our proposed model especially on the cluster-based metric.
CSECU-DSG at SemEval-2023 Task 10: Exploiting Transformers with Stacked LSTM for the Explainable Detection of Online Sexism

Sexism is a harmful phenomenon that provokes gender inequalities and social imbalances.The expanding application of sexist content on social media platforms creates an unwelcoming and discomforting environment for many users.The implication of sexism is a multi-faceted subject as it can be integrated with other categories of discrimination.Binary classification tools are frequently employed to identify sexist content, but most of them provide extensive, generic categories with no further insights.SemEval-2023 introduced the Explainable Detection of Online Sexism (EDOS) task that emphasizes detecting and explaining the category of sexist content.The content of this paper details our involvement in this task where we present a neural network architecture employing document embeddings from a finetuned transformer-based model into stacked long short-term memory (LSTM) and a fully connected linear (FCL) layer .Our proposed methodology obtained an F1 score of 0.8218 (ranked 51st) in Task A. It achieved an F1 score of 0.5986 (ranked 40th) and 0.4419 (ranked 28th) in Tasks B and C, respectively.
SLT at SemEval-2023 Task 1: Enhancing Visual Word Sense Disambiguation through Image Text Retrieval using BLIP

Based on recent progress in image-text retrieval techniques, this paper presents a fine-tuned model for the Visual Word Sense Disambiguation (VWSD) task.The proposed system finetunes a pre-trained model using ITC and ITM losses and employs a candidate selection approach for faster inference.The system was trained on the VWSD task dataset and evaluated on a separate test set using Mean Reciprocal Rank (MRR) metric.Additionally, the system was tested on the provided test set which contained Persian and Italian languages, and the results were evaluated on each language separately.Our proposed system demonstrates the potential of fine-tuning pre-trained models for complex language tasks and provides insights for further research in the field of image text retrieval.
Soft Language Clustering for Multilingual Model Pre-training

Multilingual pre-trained language models have demonstrated impressive (zero-shot) crosslingual transfer abilities, however, their performance is hindered when the target language has distant typology from source languages or when pre-training data is limited in size.In this paper, we propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally.Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.On the tasks of XTREME including text classification, sequence labeling, question answering, and sentence retrieval, both baseand large-size language models pre-trained with our proposed method exhibit consistent performance improvement.Furthermore, it provides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer 1 .
A New Dataset and Empirical Study for Sentence Simplification in Chinese

Sentence Simplification is a valuable technique that can benefit language learners and children a lot.However, current research focuses more on English sentence simplification.The development of Chinese sentence simplification is relatively slow due to the lack of data.To alleviate this limitation, this paper introduces CSS, a new dataset for assessing sentence simplification in Chinese.We collect manual simplifications from human annotators and perform data analysis to show the difference between English and Chinese sentence simplifications.Furthermore, we test several unsupervised and zero/few-shot learning methods on CSS and analyze the automatic evaluation and human evaluation results.In the end, we explore whether Large Language Models can serve as high-quality Chinese sentence simplification systems by evaluating them on CSS.
Multimedia Generative Script Learning for Task Planning

Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities.An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps.Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 5,652 tasks and 79,089 multimedia steps.This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps.We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective.We define metrics to evaluate both generation and inductive quality.Experiment results demonstrate that our approach significantly outperforms strong baselines 1 . Goal: make a bubbling brook braceletStep 1: gather the necessary items. Next Step?Step History:Step 5: cut the thread.Ground Truth: string the beads and clasp onto the string in the following order: 1 medium, 1 dark, 1 end of clasp, 1 dark, and 1 medium.BART: measure the length of the bead.
D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Bias

Despite recent advancements, NLP models continue to be vulnerable to bias.This bias often originates from the uneven distribution of real-world data and can propagate through the annotation process.Escalated integration of these models in our lives calls for methods to mitigate bias without overbearing annotation costs.While active learning (AL) has shown promise in training models with a small amount of annotated data, AL's reliance on the model's behavior for selective sampling can lead to an accumulation of unwanted bias rather than bias mitigation.However, infusing clustering with AL can overcome the bias issue of both AL and traditional annotation methods while exploiting AL's annotation efficiency.In this paper, we propose a novel adaptive clustering-based active learning algorithm, D-CALM, that dynamically adjusts clustering and annotation efforts in response to an estimated classifier errorrate.Experiments on eight datasets for a diverse set of text classification tasks, including emotion, hatespeech, dialog act, and book type detection, demonstrate that our proposed algorithm significantly outperforms baseline AL approaches with both pretrained transformers and traditional Support Vector Machines.D-CALM showcases robustness against different measures of information gain and, as evident from our analysis of label and error distribution, can significantly reduce unwanted model bias.
MedTem2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries

Discharge summaries are comprehensive medical records that encompass vital information about a patient's hospital stay.A crucial aspect of discharge summaries is the temporal information of treatments administered throughout the patient's illness.With an extensive volume of clinical documents, manually extracting and compiling a patient's medication list can be laborious, time-consuming, and susceptible to errors.The objective of this paper is to build upon the recent development on clinical NLP by temporally classifying treatments in clinical texts, specifically determining whether a treatment was administered between the time of admission and discharge from the hospital.State-of-the-art NLP methods including prompt-based learning on Generative Pretrained Transformers (GPTs) models and finetuning on pre-trained language models (PLMs) such as BERT were used to classify temporal relations between treatments and hospitalisation periods in discharge summaries.Fine-tuning with the BERT model achieved an F1 score of 92.45% and a balanced accuracy of 77.56%, while prompt learning using the T5 model and mixed templates resulted in an F1 score of 90.89% and a balanced accuracy of 72.07%.
TADA : Task Agnostic Dialect Adapters for English

Large Language Models, the dominant starting point for Natural Language Processing (NLP) applications, fail at a higher rate for speakers of English dialects other than Standard American English (SAE).Prior work addresses this using task-specific data or synthetic data augmentation, both of which require intervention for each dialect and task pair.This poses a scalability issue that prevents the broad adoption of robust dialectal English NLP.We introduce a simple yet effective method for task-agnostic dialect adaptation by aligning non-SAE dialects using adapters and composing them with taskspecific adapters from SAE. Task-Agnostic Dialect Adapters (TADA) improve dialectal robustness on 4 dialectal variants of the GLUE benchmark without task-specific supervision. 1
D2KLab at SemEval-2023 Task 2: Leveraging T-NER to Develop a Fine-Tuned Multilingual Model for Complex Named Entity Recognition

This paper presents D2KLab's system used for the shared task of "Multilingual Complex Named Entity Recognition (MultiCoNER II)", as part of SemEval 2023 Task 2. The system relies on a fine-tuned transformer based language model for extracting named entities.We present the architecture of the system, and we discuss our results and observations.Our implementation is open sourced at https://github.com/D2KLab/multiconer.
Model Analysis &amp; Evaluation for Ambiguous Question Answering

Ambiguous questions are a challenge for Question Answering models, as they require answers that cover multiple interpretations of the original query.To this end, these models are required to generate long-form answers that often combine conflicting pieces of information.Although recent advances in the field have shown strong capabilities in generating fluent responses, certain research questions remain unanswered.Does model/data scaling improve the answers' quality?Do automated metrics align with human judgment?To what extent do these models ground their answers in evidence?In this study, we aim to thoroughly investigate these aspects, and provide valuable insights into the limitations of the current approaches.To aid in reproducibility and further extension of our work, we open-source our code here.
MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

There have been several meta-evaluation studies on the correlation between human ratings and offline machine translation (MT) evaluation metrics such as BLEU, chrF2, BERTSCORE and COMET.These metrics have been used to evaluate simultaneous speech translation (SST) but their correlations with human ratings of SST, which has been recently collected as Continuous Ratings (CR), are unclear.In this paper, we leverage the evaluations of candidate systems submitted to the English-German SST task at IWSLT 2022 and conduct an extensive correlation analysis of CR and the aforementioned metrics.Our study reveals that the offline metrics are well correlated with CR and can be reliably used for evaluating machine translation in simultaneous mode, with some limitations on the test set size.We conclude that given the current quality levels of SST, these metrics can be used as proxies for CR, alleviating the need for large scale human evaluation.Additionally, we observe that correlations of the metrics with translation as a reference is significantly higher than with simultaneous interpreting, and thus we recommend the former for reliable evaluation.
Pragmatic Inference with a CLIP Listener for Contrastive Captioning

We propose a simple yet effective and robust method for contrastive captioning: generating discriminative captions that distinguish target images from very similar alternative distractor images.Our approach is built on a pragmatic inference procedure that formulates captioning as a reference game between a speaker, which produces possible captions describing the target, and a listener, which selects the target given the caption.Unlike previous methods that derive both speaker and listener distributions from a single captioning model, we leverage an offthe-shelf CLIP model to parameterize the listener.Compared with captioner-only pragmatic models, our method benefits from rich visionlanguage alignment representations from CLIP when reasoning over distractors.Like previous methods for discriminative captioning, our method uses a hyperparameter to control the tradeoff between the informativity (how likely captions are to allow a human listener to discriminate the target image) and the fluency of the captions.However, we find that our method is substantially more robust to the value of this hyperparameter than past methods, which allows us to automatically optimize the captions for informativity -outperforming past methods for discriminative captioning by 11% to 15% accuracy in human evaluations. 1
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages

Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model.In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers.Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling).In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary.We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the wordlevel tasks.Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training. 1