Reviewerly/exHarmony huggingface dataset:
xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages

We introduce a new proxy score for evaluating bitext mining based on similarity in a multilingual embedding space: xsim++.In comparison to xsim, this improved proxy leverages rulebased approaches to extend English sentences in any evaluation set with synthetic, hard-todistinguish examples which more closely mirror the scenarios we encounter during largescale mining.We validate this proxy by running a significant number of bitext mining experiments for a set of low-resource languages, and subsequently train NMT systems on the mined data.In comparison to xsim, we show that xsim++ is better correlated with the downstream BLEU scores of translation systems trained on mined bitexts, providing a reliable proxy of bitext mining performance without needing to run expensive bitext mining pipelines.xsim++ also reports performance for different error types, offering more fine-grained feedback for model development.
Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

Pragmatic reference enables efficient interpersonal communication.Prior work uses simple reference games to test models of pragmatic reasoning, often with unidentified speakers and listeners.In practice, however, speakers' sociocultural background shapes their pragmatic assumptions.For example, readers of this paper assume NLP refers to "Natural Language Processing," and not "Neuro-linguistic Programming."This work introduces the CULTURAL CODES dataset, which operationalizes sociocultural pragmatic inference in a simple word reference game.CULTURAL CODES is based on the multi-turn collaborative two-player game, Codenames Duet.Our dataset consists of 794 games with 7,703 turns, distributed across 153 unique players.Alongside gameplay, we collect information about players' personalities, values, and demographics.Utilizing theories of communication and pragmatics, we predict each player's actions via joint modeling of their sociocultural priors and the game context.Our experiments show that accounting for background characteristics significantly improves model performance for tasks related to both clue giving and guessing, indicating that sociocultural priors play a vital role in gameplay decisions.
Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic

This paper describes the system we designed for our participation in SemEval-2023 Task 12 Track 6 about Algerian dialect sentiment analysis.We propose a transformer language model approach combined with a lexicon mixing terms and emojis which is used in a postprocessing filtering stage.The Algerian sentiment lexicons were extracted manually from tweets.We report on our experiments on the Algerian dialect, where we compare the performance of MARBERT to the one of Arabic-BERT and CAMeLBERT on the training and development datasets of Task 12.We also analyze the contribution of our post-processing lexical filtering for sentiment analysis.Our system obtained an F1 score equal to 70%, ranking 9 th among 30 participants.
When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario

Large pre-trained language models (PLMs) have garnered significant attention for their versatility and potential for solving a wide spectrum of natural language processing (NLP) tasks.However, the cost of running these PLMs may be prohibitive.Furthermore, PLMs may not be open-sourced due to commercial considerations and potential risks of misuse, such as GPT-3.The parameters and gradients of PLMs are unavailable in this scenario.To solve the issue, black-box tuning has been proposed, which utilizes derivative-free optimization (DFO), instead of gradient descent, for training task-specific continuous prompts.However, these gradient-free methods still exhibit a significant gap compared to gradientbased methods.In this paper, we introduce gradient descent into black-box tuning scenario through knowledge distillation.Furthermore, we propose a novel method GDFO, which integrates gradient descent and derivative-free optimization to optimize task-specific continuous prompts in a harmonized manner.Experimental results show that GDFO can achieve significant performance gains over previous state-ofthe-art methods.
What, When, and How to Ground: Designing User Persona-Aware Conversational Agents for Engaging Dialogue

This paper presents a method for building a personalized open-domain dialogue system to address the WWH (WHAT, WHEN, and HOW) problem for natural response generation in a commercial setting, where personalized dialogue responses are heavily interleaved with casual response turns.The proposed approach involves weighted dataset blending, negative persona information augmentation methods, and the design of personalized conversation datasets to address the challenges of WWH in personalized, open-domain dialogue systems.Our work effectively balances dialogue fluency and tendency to ground, while also introducing a response-type label to improve the controllability and explainability of the grounded responses.The combination of these methods leads to more fluent conversations, as evidenced by subjective human evaluations as well as objective evaluations.
Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home

Enriching the quality of early childhood education with interactive math learning at home systems, empowered by recent advances in conversational AI technologies, is slowly becoming a reality.With this motivation, we implement a multimodal dialogue system to support play-based learning experiences at home, guiding kids to master basic math concepts.This work explores Spoken Language Understanding (SLU) pipeline within a task-oriented dialogue system developed for Kid Space, with cascading Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) components evaluated on our home deployment data with kids going through gamified math learning activities.We validate the advantages of a multi-task architecture for NLU and experiment with a diverse set of pretrained language representations for Intent Recognition and Entity Extraction tasks in the math learning domain.To recognize kids' speech in realistic home environments, we investigate several ASR systems, including the commercial Google Cloud and the latest open-source Whisper solutions with varying model sizes.We evaluate the SLU pipeline by testing our bestperforming NLU models on noisy ASR output to inspect the challenges of understanding children for math learning in authentic homes.
Language Anisotropic Cross-Lingual Model Editing

Multilingual pre-trained language models can learn task-specific abilities or memorize facts across multiple languages but inevitably make undesired predictions with specific inputs.Under similar observation, model editing aims to post-hoc calibrate a model targeted to specific inputs with keeping the model's raw behavior.However, existing work only studies the monolingual scenario, which lacks the cross-lingual transferability to perform editing simultaneously across languages.In this work, we focus on cross-lingual model editing.Firstly, we define the cross-lingual model editing task and corresponding metrics, where an edit in one language propagates to the others.Next, we propose a framework to naturally adapt monolingual model editing approaches to the crosslingual scenario using parallel corpus.Further, we propose language anisotropic editing to improve cross-lingual editing by amplifying different subsets of parameters for each language.On the newly defined cross-lingual model editing task, we empirically demonstrate the failure of monolingual baselines in propagating the edit to multiple languages and the effectiveness of the proposed language anisotropic model editing.Our code is publicly available at https://github.com/franklear/LiME.
Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Model

Sentence embedding is one of the most fundamental tasks in Natural Language Processing and plays an important role in various tasks.The recent breakthrough in sentence embedding is achieved by pre-trained language models (PLMs).Despite its success, an embedded vector (Sen2Vec) representing a point estimate does not naturally express uncertainty in a taskagnostic way.This paper thereby proposes an efficient framework on probabilistic sentence embedding (Sen2Pro) from PLMs, and it represents a sentence as a probability density distribution in an embedding space to reflect both model uncertainty and data uncertainty (i.e., many-to-one nature) in the sentence representation.The proposed framework performs in a plug-and-play way without retraining PLMs anymore, and it is easy to implement and generally applied on top of any PLM.The superiority of Sen2Pro over Sen2Vec has been theoretically verified and practically illustrated on different NLP tasks.
Unbalanced Optimal Transport for Unbalanced Word Alignment

Monolingual word alignment is crucial to model semantic interactions between sentences.In particular, null alignment, a phenomenon in which words have no corresponding counterparts, is pervasive and critical in handling semantically divergent sentences.Identification of null alignment is useful on its own to reason about the semantic similarity of sentences by indicating there exists information inequality.To achieve unbalanced word alignment that values both alignment and null alignment, this study shows that the family of optimal transport (OT), i.e., balanced, partial, and unbalanced OT, are natural and powerful approaches even without tailor-made techniques.Our extensive experiments covering unsupervised and supervised settings indicate that our generic OT-based alignment methods are competitive against the state-of-the-arts specially designed for word alignment, remarkably on challenging datasets with high null alignment frequencies.
On Prefix-tuning for Lightweight Out-of-distribution Detection

Out-of-distribution (OOD) detection, a fundamental task vexing real-world applications, has attracted growing attention in the NLP community.Recently fine-tuning based methods have made promising progress.However, it could be costly to store fine-tuned models for each scenario.In this paper, we depart from the classic fine-tuning based OOD detection toward a parameter-efficient alternative, and propose an unsupervised prefix-tuning based OOD detection framework termed PTO.Additionally, to take advantage of optional training data labels and targeted OOD data, two practical extensions of PTO are further proposed.Overall, PTO and its extensions offer several key advantages of being lightweight, easy-to-reproduce, and theoretically justified.Experimental results show that our methods perform comparably to, even better than, existing fine-tuning based OOD detection approaches under a wide range of metrics, detection settings, and OOD types.
MDC at SemEval-2023 Task 7: Fine-tuning Transformers for Textual Entailment Prediction and Evidence Retrieval in Clinical Trials

We present our entry to the Multi-evidence Natural Language Inference for Clinical Trial Data task at SemEval 2023.We submitted entries for both the evidence retrieval and textual entailment sub-tasks.For the evidence retrieval task, we fine-tuned the PubMedBERT transformer model to extract relevant evidence from clinical trial data given a hypothesis concerning either a single clinical trial or pair of clinical trials.Our best performing model achieved an F1 score of 0.804.For the textual entailment task, in which systems had to predict whether a hypothesis about either a single clinical trial or pair of clinical trials is true or false, we fine-tuned the BioLinkBERT transformer model.We passed our evidence retrieval model's output into our textual entailment model and submitted its output for the evaluation.Our best performing model achieved an F1 score of 0.695.
XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding

Recently, Contrastive Visual-Language Pretraining (CLIP) has demonstrated remarkable capability in various Visual Language Understanding (VLU) tasks.Yet, most CLIP-based methods require tasks-specific designs and sufficient training data.In this paper, we introduce a simple yet efficient paradigm for lowresource VLU named XtremeCLIP, which involves very few trainable parameters to improve the generalization ability of the trained models.In our XtremeCLIP framework, we reformulate a series of VLU tasks as a unified open-book affinity-matching problem.Furthermore, to handle the insufficient supervised signals in small datasets, we adopt contrastive learning to utilize the implicit sorting information of ground-truth labels to provide more supervised cues.Extensive experiments over multiple datasets on visual entailment, visual question answering, and image classification show that XtremeCLIP consistently outperforms existing baselines in low-resource settings.
EventOA: An Event Ontology Alignment Benchmark Based on FrameNet and Wikidata

Event ontology provides a shared and formal specification about what happens in the real world and can benefit many natural language understanding tasks.However, the independent development of event ontologies often results in heterogeneous representations that raise the need for establishing alignments between semantically related events.There exists a series of works about ontology alignment (OA), but they only focus on the entity-based OA, and neglect the event-based OA.To fill the gap, we construct an Event Ontology Alignment (EventOA) dataset based on FrameNet and Wikidata, which consists of 900+ event type alignments and 8,000+ event argument alignments.Furthermore, we propose a multi-view event ontology alignment (MEOA) method, which utilizes description information (i.e., name, alias and definition) and neighbor information (i.e., subclass and superclass) to obtain richer representation of the event ontologies.Extensive experiments show that our MEOA outperforms the existing entity-based OA methods and can serve as a strong baseline for Even-tOA research.
Exploring the Impact of Vision Features in News Image Captioning

The task of news image captioning aims to generate a detailed caption which describes the specific information of an image in a news article.However, we find that recent state-of-art models can achieve competitive performance even without vision features.To resolve the impact of vision features in the news image captioning task, we conduct extensive experiments with mainstream models based on encoder-decoder framework.From our exploration, we find 1) vision features do contribute to the generation of news image captions; 2) vision features can assist models to better generate entities of captions when the entity information is sufficient in the input textual context of the given article; 3) Regions of specific objects in images contribute to the generation of related entities in captions. 1
Team TheSyllogist at SemEval-2023 Task 3: Language-Agnostic Framing Detection in Multi-Lingual Online News: A Zero-Shot Transfer Approach

We describe our system for SemEval-2022 Task 3 subtask 2 which on detecting the frames used in a news article in a multi-lingual setup.We propose a multi-lingual approach based on machine translation of the input, followed by an English prediction model.Our system demonstrated good zero-shot transfer capability, achieving micro-F1 scores of 53% for Greek (4th on the leaderboard) and 56.1% for Georgian (3rd on the leaderboard), without any prior training on translated data for these languages.Moreover, our system achieved comparable performance on seven other languages, including German, English, French, Russian, Italian, Polish, and Spanish.Our results demonstrate the feasibility of creating a languageagnostic model for automatic framing detection in online news.
LR-Sum: Summarization for Less-Resourced Languages

We introduce LR-Sum, a new permissivelylicensed dataset created with the goal of enabling further research in automatic summarization for less-resourced languages.LR-Sum contains human-written summaries for 40 languages, many of which are less-resourced.We describe our process for extracting and filtering the dataset from the Multilingual Open Text corpus (Palen-Michel et al., 2022).The source data is public domain newswire collected from from Voice of America websites, and LR-Sum is released under a Creative Commons license (CC BY 4.0), making it one of the most openlylicensed multilingual summarization datasets.We describe abstractive and extractive summarization experiments to establish baselines and discuss the limitations of this dataset.
AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese

Despite their successes in NLP, Transformerbased language models still require extensive computing resources and suffer in lowresource or low-compute settings.In this paper, we present AxomiyaBERTa, a novel BERT model for Assamese, a morphologically-rich low-resource language (LRL) of Eastern India.AxomiyaBERTa is trained only on the masked language modeling (MLM) task, without the typical additional next sentence prediction (NSP) objective, and our results show that in resource-scarce settings for very low-resource languages like Assamese, MLM alone can be successfully leveraged for a range of tasks.AxomiyaBERTa achieves SOTA on tokenlevel tasks like Named Entity Recognition and also performs well on "longer-context" tasks like Cloze-style QA and Wiki Title Prediction, with the assistance of a novel embedding disperser and phonological signals respectively.Moreover, we show that AxomiyaBERTa can leverage phonological signals for even more challenging tasks, such as a novel cross-document coreference task on a translated version of the ECB+ corpus, where we present a new SOTA result for an LRL.Our source code and evaluation scripts may be found at https: //github.com/csu-signal/axomiyaberta.
Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents

We investigate semi-structured document classification in a zero-shot setting.Classification of semi-structured documents is more challenging than that of standard unstructured documents, as positional, layout, and style information play a vital role in interpreting such documents.The standard classification setting where categories are fixed during both training and testing falls short in dynamic environments where new document categories could potentially emerge.We focus exclusively on the zero-shot setting where inference is done on new unseen classes.To address this task, we propose a matching-based approach that relies on a pairwise contrastive objective for both pretraining and fine-tuning.Our results show a significant boost in Macro F 1 from the proposed pretraining step in both supervised and unsupervised zero-shot settings.
Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models

In this paper, we propose a table and image generation task to verify how the knowledge about entities acquired from natural language is retained in Vision & Language (V&L) models.This task consists of two parts: the first is to generate a table containing knowledge about an entity and its related image, and the second is to generate an image from an entity with a caption and a table containing related knowledge of the entity.In both tasks, the model must know the entities used to perform the generation properly.We created the Wikipedia Table and Image Generation (WikiTIG) dataset from about 200,000 infoboxes in English Wikipedia articles to perform the proposed tasks.We evaluated the performance on the tasks with respect to the above research question using the V&L model OFA (Wang et al., 2022), which has achieved state-of-the-art results in multiple tasks.Experimental results show that OFA forgets part of its entity knowledge by pre-training as a complement to improve the performance of image related tasks.
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View

We revisit the multimodal entity and relation extraction from a translation point of view.Special attention is paid on the misalignment issue in text-image datasets which may mislead the learning.We are motivated by the fact that the cross-modal misalignment is a similar problem of cross-lingual divergence issue in machine translation.The problem can then be transformed and existing solutions can be borrowed by treating a text and its paired image as the translation to each other.We implement a multimodal back-translation using diffusionbased generative models for pseudo-paralleled pairs and a divergence estimator by constructing a high-resource corpora as a bridge for low-resource learners.Fine-grained confidence scores are generated to indicate both types and degrees of alignments with which better representations are obtained.The method has been validated in the experiments by outperforming 14 state-of-the-art methods in both entity and relation extraction tasks.The source code is available at https://github.com/thecharm/TMR.
Uncovering and Categorizing Social Biases in Text-to-SQL

Content Warning: This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups.Large pre-trained language models are acknowledged to carry social biases towards different demographics, which can further amplify existing stereotypes in our society and cause even more harm.Text-to-SQL is an important task, models of which are mainly adopted by authoritative institutions, where unfair decisions may lead to catastrophic consequences.However, existing Text-to-SQL models are trained on clean, neutral datasets, such as Spider and WikiSQL.This, to some extent, cover up social bias in models under ideal conditions, which nevertheless may emerge in real application scenarios.In this work, we aim to uncover and categorize social biases in Text-to-SQL models.We summarize the categories of social biases that may occur in structured data for Text-to-SQL models.We build test benchmarks and reveal that models with similar task accuracy can contain social biases at very different rates.We show how to take advantage of our methodology to uncover and assess social biases in the downstream Text-to-SQL task 1 .
Modularized Zero-shot VQA with Pre-trained Models

Large-scale pre-trained models (PTMs) show great zero-shot capabilities.In this paper, we study how to leverage them for zero-shot visual question answering (VQA).Our approach is motivated by a few observations.First, VQA questions often require multiple steps of reasoning, which is still a capability that most PTMs lack.Second, different steps in VQA reasoning chains require different skills such as object detection and relational reasoning, but a single PTM may not possess all these skills.Third, recent work on zero-shot VQA does not explicitly consider multi-step reasoning chains, which makes them less interpretable compared with a decomposition-based approach.We propose a modularized zero-shot network that explicitly decomposes questions into sub reasoning steps and is highly interpretable.We convert sub reasoning tasks to acceptable objectives of PTMs and assign tasks to proper PTMs without any adaptation.Our experiments on two VQA benchmarks under the zero-shot setting demonstrate the effectiveness of our method and better interpretability compared with several baselines.