Reviewerly/exHarmony huggingface dataset:
What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary

Dual encoders are now the dominant architecture for dense retrieval.Yet, we have little understanding of how they represent text, and why this leads to good performance.In this work, we shed light on this question via distributions over the vocabulary.We propose to interpret the vector representations produced by dual encoders by projecting them into the model's vocabulary space.We show that the resulting projections contain rich semantic information, and draw connection between them and sparse retrieval.We find that this view can offer an explanation for some of the failure cases of dense retrievers.For example, we observe that the inability of models to handle tail entities is correlated with a tendency of the token distributions to forget some of the tokens of those entities.We leverage this insight and propose a simple way to enrich query and passage representations with lexical information at inference time, and show that this significantly improves performance compared to the original model in zero-shot settings, and specifically on the BEIR benchmark. 1
Contextual Knowledge Learning for Dialogue Generation

Incorporating conversational context and knowledge into dialogue generation models has been essential for improving the quality of the generated responses.The context, comprising utterances from previous dialogue exchanges, is used as a source of content for response generation and as a means of selecting external knowledge.However, to avoid introducing irrelevant content, it is key to enable fine-grained scoring of context and knowledge.In this paper, we present a novel approach to context and knowledge weighting as an integral part of model training.We guide the model training through a Contextual Knowledge Learning (CKL) process which involves Latent Vectors for context and knowledge, respectively.CKL Latent Vectors capture the relationship between context, knowledge, and responses through weak supervision and enable differential weighting of context utterances and knowledge sentences during the training process.Experiments with two standard datasets and human evaluation demonstrate that CKL leads to a significant improvement compared with the performance of six strong baseline models and shows robustness with regard to reduced sizes of training sets.
Silp_nlp at SemEval-2023 Task 2: Cross-lingual Knowledge Transfer for Mono-lingual Learning

Our team silp_nlp participated in SemEval2023 Task 2: MultiCoNER II.Our work made systems for 11 mono-lingual tracks.For leveraging the advantage of all track knowledge we chose transformer-based pretrained models, which have strong cross-lingual transferability.Hence our model trained in two stages, the first stage for multi-lingual learning from all tracks and the second for fine-tuning individual tracks.Our work highlights that the knowledge of all tracks can be transferred to an individual track if the baseline language model has crosslingual features.Our system positioned itself in the top 10 for 4 tracks by scoring 0.7432 macro F1 score for the Hindi track ( 7th rank ) and 0.7322 macro F1 score for the Bangla track ( 9th rank ).
Bhattacharya_Lab at SemEval-2023 Task 12: A Transformer-based Language Model for Sentiment Classification for Low Resource African Languages: Nigerian Pidgin and Yoruba

Sentiment Analysis is an aspect of natural language processing (NLP) that has been a topic of research.While most studies focus on highresource languages with an extensive amount of available data, the study on low-resource languages with insufficient data needs attention.To address this issue, we propose a transformerbased method for sentiment analysis for lowresources African languages, Nigerian Pidgin and Yoruba.To evaluate the effectiveness of our multilingual language models for monolingual sentiment classification, we participated in the AfriSenti SemEval shared task 2023 competition.On the official evaluation set, our group (named as Bhattacharya_Lab) ranked 1 out of 33 participating groups in the Monolingual Sentiment Classification task (i.e., Task A) for Nigerian Pidgin (i.e., Track 4), and in the Top 5 among 33 participating groups in the Monolingual Sentiment Classification task for Yoruba (i.e., Track 2) respectively, demonstrating the potential for our transformer-based language models to improve sentiment analysis in low-resource languages.Overall, our study highlights the importance of exploring the potential of NLP in low-resource languages and the impact of transformer-based multilingual language models in sentiment analysis for the low-resource African languages, Nigerian Pidgin and Yoruba.
Revisiting Event Argument Extraction: Can EAE Models Learn Better When Being Aware of Event Co-occurrences?

Event co-occurrences have been proved effective for event extraction (EE) in previous studies, but have not been considered for event argument extraction (EAE) recently.In this paper, we try to fill this gap between EE research and EAE research, by highlighting the question that "Can EAE models learn better when being aware of event co-occurrences?".To answer this question, we reformulate EAE as a problem of table generation and extend a SOTA prompt-based EAE model into a nonautoregressive generation framework, called TabEAE, which is able to extract the arguments of multiple events in parallel.Under this framework, we experiment with 3 different training-inference schemes on 4 datasets (ACE05, RAMS, WikiEvents and MLEE) and discover that via training the model to extract all events in parallel, it can better distinguish the semantic boundary of each event and its ability to extract single event gets substantially improved.Experimental results show that our method achieves new state-ofthe-art performance on the 4 datasets.
Fine-grained Contrastive Learning for Relation Extraction

Recent relation extraction (RE) works haveshown encouraging improvements by conducting contrastive learning on silver labels generated by distant supervision before fine-tuning on gold labels.Existing methods typically assume all these silver labels are accurate and treat them equally; however, distant supervision is inevitably noisy-some silver labels are more reliable than others.In this paper, we propose fine-grained contrastive learning (FineCL) for RE, which leverages fine-grained information about which silver labels are and are not noisy to improve the quality of learned relationship representations for RE.We first assess the quality of silver labels via a simple and automatic approach we call "learning order denoising," where we train a language model to learn these relations and record the order of learned training instances.We show that learning order largely corresponds to label accuracy-early-learned silver labels have, on average, more accurate labels than later-learned silver labels.Then, during pre-training, we increase the weights of accurate labels within a novel contrastive learning objective.Experiments on several RE benchmarks show that FineCL makes consistent and significant performance gains over state-of-the-art methods.
Natural Language to Code Translation with Execution

success in translating natural language to code (Chen et al., 2021;Austin et al., 2021; Li et al., 2022, inter alia).While these models do not explicitly incorporate program semantics (i.e., execution results) during training, they are able to generate correct solutions for many problems.However, choosing a single correct program from a generated set for each problem remains challenging.In this work, we introduce execution resultbased minimum Bayes risk decoding (MBR-EXEC) for program selection and show that it improves the few-shot performance of pretrained code models on natural-language-tocode tasks.We select output programs from a generated candidate set by marginalizing over program implementations that share the same semantics.Because exact equivalence is intractable, we execute each program on a small number of test inputs to approximate semantic equivalence.Across datasets, execution or simulated execution significantly outperforms the methods that do not involve program semantics.We find that MBR-EXEC consistently improves over all execution-unaware selection methods, suggesting it as an effective approach for natural language to code translation.
Curriculum Prompt Learning with Self-Training for Abstractive Dialogue Summarization

Succinctly summarizing dialogue is a task of growing interest, but inherent challenges, such as insufficient training data and low information density impede our ability to train abstractive models.In this work, we propose a novel curriculum-based prompt learning method with self-training to address these problems.Specifically, prompts are learned using a curriculum learning strategy that gradually increases the degree of prompt perturbation, thereby improving the dialogue understanding and modeling capabilities of our model.Unlabeled dialogue is incorporated by means of self-training so as to reduce the dependency on labeled data.We further investigate topic-aware prompts to better plan for the generation of summaries.Experiments confirm that our model substantially outperforms strong baselines and achieves new state-of-the-art results on the AMI and ICSI datasets.Human evaluations also show the superiority of our model with regard to the summary generation quality.
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations

Recent work on explainable NLP has shown that few-shot prompting can enable large pretrained language models (LLMs) to generate grammatical and factual natural language explanations for data labels.In this work, we study the connection between explainability and sample hardness by investigating the following research question -"Are LLMs and humans equally good at explaining data labels for both easy and hard samples?"We answer this question by first collecting humanwritten explanations in the form of generalizable commonsense rules on the task of Winograd Schema Challenge (Winogrande dataset).We compare these explanations with those generated by GPT-3 while varying the hardness of the test samples as well as the in-context samples.We observe that (1) GPT-3 explanations are as grammatical as human explanations regardless of the hardness of the test samples, (2) for easy examples, GPT-3 generates highly supportive explanations but human explanations are more generalizable, and (3) for hard examples, human explanations are significantly better than GPT-3 explanations both in terms of label-supportiveness and generalizability judgements.We also find that hardness of the in-context examples impacts the quality of GPT-3 explanations.Finally, we show that the supportiveness and generalizability aspects of human explanations are also impacted by sample hardness, although by a much smaller margin than models. 1
Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning

Recent research shows synthetic data as a source of supervision helps pretrained language models (PLM) transfer learning to new target tasks/domains.However, this idea is less explored for spatial language.We provide two new data resources on multiple spatial language processing tasks.The first dataset is synthesized for transfer learning on spatial question answering (SQA) and spatial role labeling (SpRL).Compared to previous SQA datasets, we include a larger variety of spatial relation types and spatial expressions.Our data generation process is easily extendable with new spatial expression lexicons.The second one is a real-world SQA dataset with humangenerated questions built on an existing corpus with SPRL annotations.This dataset can be used to evaluate spatial language processing models in realistic situations.We show pretraining with automatically generated data significantly improves the SOTA results on several SQA and SPRL benchmarks, particularly when the training data in the target domain is small.
Towards Robust k-Nearest-Neighbor Machine Translation

k-Nearest-Neighbor Machine Translation (kNN-MT) becomes an important research direction of NMT in recent years.Its main idea is to retrieve useful key-value pairs from an additional datastore to modify translations without updating the NMT model.However, the underlying retrieved noisy pairs will dramatically deteriorate the model performance.In this paper, we conduct a preliminary study and find that this problem results from not fully exploiting the prediction of the NMT model.To alleviate the impact of noise, we propose a confidence-enhanced kNN-MT model with robust training.Concretely, we introduce the NMT confidence to refine the modeling of two important components of kNN-MT: kNN distribution and the interpolation weight.Meanwhile we inject two types of perturbations into the retrieved pairs for robust training.Experimental results on four benchmark datasets demonstrate that our model not only achieves significant improvements over current kNN-MT models, but also exhibits better robustness.Our code is available at https://github.com/DeepLearnXMU/Robust-knn-mt.
LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation

Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features, which has attracted considerable attention from both natural language processing and computer vision communities.Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases in the real world.In other words, the multilingual multimodal machine translation (Multilingual MMT) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for multiple languages.Besides, the image modality has no language boundaries, which is superior to bridging the semantic gap between languages.To this end, we first propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages.Then, an effective baseline LVP-M 3 using visual prompts is proposed to support translations between different languages, which includes three stages (token encoding, language-aware visual prompt generation, and language translation).Extensive experimental results on our constructed benchmark datasets demonstrate the effectiveness of LVP-M 3 method for Multilingual MMT.
Emoji Sentiment Roles for Sentiment Analysis: A Case Study in Arabic Texts

Emoji (the digital pictograms) are crucial features for textual sentiment analysis.However, analysing the sentiment roles of emoji is very complex.This is due to its dependency on different factors, such as textual context, cultural perspective, interlocutor's personal traits, interlocutors' relationships or a platforms' functional features.This work introduces an approach to analysing the sentiment effects of emoji as textual features.Using an Arabic dataset as a benchmark, our results confirm the borrowed argument that each emoji has three different norms of sentiment role (negative, neutral or positive).Therefore, an emoji can play different sentiment roles depending upon context.It can behave as an emphasizer, an indicator, a mitigator, a reverser or a trigger of either negative or positive sentiment within a text.In addition, an emoji may have neutral effect (i.e., no effect) on the sentiment of the text.
Structural generalization is hard for sequence-to-sequence models

Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks, including ones that require predicting linguistic structure.However, recent work on compositional generalization has shown that seq2seq models achieve very low accuracy in generalizing to linguistic structures that were not seen in training.We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in.We further report on some experiments that give initial answers on the reasons for these limitations.Training Generalization (a) LEX subj_to_obj (common noun) A hedgehog ate the cake.* cake(x4); hedgehog(x1) ∧ eat.agent(x2, x1) ∧eat.theme(x2,x4) The baby liked the hedgehog.* baby(x1); * hedgehog(x4); like.agent(x2,x1) ∧ like.theme(x2,x4) (b) STRUCT PP recursion Ava saw a ball in a bowl on the table.* table(x9); see.agent(x1,Ava) ∧ see.theme(x1, x3) ∧ ball(x3) ∧ ball.nmod.in(x3,x6) ∧ bowl(x6) ∧ bowl.nmod.on(x6,x9) Ava saw a ball in a bowl on the table on the floor.* table(x9); * floor(x12); see.agent(x1,Ava) ∧ see.theme(x1, x3) ∧ ball(x3) ∧ ball.nmod.in(x3,x6) ∧ bowl(x6) ∧ bowl.nmod.on(x6,x9) ∧ table.nmod.on(x9,x12) (c) STRUCT obj_to_subj PP Noah ate the cake on the plate.* cake(x3); * plate(x6); eat.agent(x1,Noah) ∧ eat.theme(x1, x3) ∧ cake.nmod.on(x3,x6) The cake on the table burned.* cake(x1); * table(x4); cake.nmod.on(x1,x4) ∧ burn.theme(x3,x1)
COM-MRC: A COntext-Masked Machine Reading Comprehension Framework for Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) aims to extract sentiment triplets from sentences, which was recently formalized as an effective machine reading comprehension (MRC) based framework.However, when facing multiple aspect terms, the MRC-based methods could fail due to the interference from other aspect terms.In this paper, we propose a novel COntext-Masked MRC (COM-MRC) framework for ASTE.Our COM-MRC framework comprises three closely-related components: a context augmentation strategy, a discriminative model, and an inference method.Specifically, a context augmentation strategy is designed by enumerating all masked contexts for each aspect term.The discriminative model comprises four modules, i.e., aspect and opinion extraction modules, sentiment classification and aspect detection modules.In addition, a two-stage inference method first extracts all aspects and then identifies their opinions and sentiment through iteratively masking the aspects.Extensive experimental results on benchmark datasets show the effectiveness of our proposed COM-MRC framework, which outperforms state-of-the-art methods consistently 1 .
RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning

Conventional visual relationship detection models only use the numeric ids of relation labels for training, but ignore the semantic correlation between the labels, which leads to severe training biases and harms the generalization ability of representations.In this paper, we introduce compact language information of relation labels for regularizing the representation learning of visual relations.Specifically, we propose a simple yet effective visual Relationship prediction framework that transfers natural language knowledge learned from Contrastive Language-Image Pre-training (CLIP) models to enhance the relationship prediction, termed as RelCLIP.Benefiting from the powerful visual-semantic alignment ability of CLIP at image level, we introduce a novel Relational Contrastive Learning (RCL) approach that explores relation-level visual-semantic alignment via learning to match cross-modal relational embeddings.By collaboratively learning the semantic coherence and discrepancy from relation triplets, the model can generate more discriminative and robust representations.Experimental results on the Visual Genome dataset show that RelCLIP achieves significant improvements over strong baselines under full (providing accurate labels) and distant supervision (providing noise labels), demonstrating its powerful generalization ability in learning relationship representations.
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment.This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia.To this end, we introduce TEMPORALWIKI, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively.The benchmark hence allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time.We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning.The dataset and the code is made available at this link.
AraProp at WANLP 2022 Shared Task: Leveraging Pre-Trained Language Models for Arabic Propaganda Detection

This paper presents our approach taken for the shared task on Propaganda Detection in Arabic at the Seventh Arabic Natural Language Processing Workshop (WANLP 2022).We participated in Sub-task 1, where the text of a tweet is provided, and the goal is to identify the different propaganda techniques used in it.This problem belongs to multi-label classification.For our solution, we leveraged different transformer-based pre-trained language models with fine-tuning to solve this problem.In our analysis, we found that MARBERTv2 outperforms in terms of performance, where macro-F1 is 0.08175 and micro-F1 is 0.61116 compared to other language models that we considered.Our method achieved rank 4 in the testing phase of the challenge.
Large-scale Machine Translation for Indian Languages in E-commerce under Low Resource Constraints

The democratization of e-commerce platforms has moved an increasingly diversified Indian user base to shop online. We have deployed reliable and precise large-scale Machine Translation systems for several Indian regional languages in this work. Building such systems is a challenge because of the low-resource nature of the Indian languages. We develop a structured model development pipeline as a closed feedback loop with external manual feedback through an Active Learning component. We show strong synthetic parallel data generation capability and consistent improvements to the model over iterations. Starting with 1.2M parallel pairs for English-Hindi we have compiled a corpus with 400M+ synthetic high quality parallel pairs across different domains. Further, we need colloquial translations to preserve the intent and friendliness of English content in regional languages, and make it easier to understand for our users. We perform robust and effective domain adaptation steps to achieve colloquial such translations. Over iterations, we show 9.02 BLEU points improvement for English to Hindi translation model. Along with Hindi, we show that the overall approach and best practices extends well to other Indian languages, resulting in deployment of our models across 7 Indian Languages.