Reviewerly/exHarmony huggingface dataset:
How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives

Recently, various intermediate layer distillation (ILD) objectives have been shown to improve compression of BERT models via Knowledge Distillation (KD).However, a comprehensive evaluation of the objectives in both taskspecific and task-agnostic settings is lacking.To the best of our knowledge, this is the first work comprehensively evaluating distillation objectives in both settings.We show that attention transfer gives the best performance overall.We also study the impact of layer choice when initializing the student from the teacher layers, finding a significant impact on the performance in task-specific distillation.For vanilla KD and hidden states transfer, initialisation with lower layers of the teacher gives a considerable improvement over higher layers, especially on the task of QNLI (up to an absolute percentage change of 17.8 in accuracy).Attention transfer behaves consistently under different initialisation settings.We release our code as an efficient transformer-based model distillation framework for further studies. 1
Ertim at SemEval-2023 Task 2: Fine-tuning of Transformer Language Models and External Knowledge Leveraging for NER in Farsi, English, French and Chinese

Transformer language models are now a solid baseline for Named Entity Recognition and can be significantly improved by leveraging complementary resources, either by integrating external knowledge or by annotating additional data.In a preliminary step, this work presents experiments on fine-tuning transformer models.Then, a set of experiments has been conducted with a Wikipedia-based reclassification system.Additionally, we conducted a small annotation campaign on the Farsi language to evaluate the impact of additional data.These two methods with complementary resources showed improvements compared to fine-tuning only.
Token-Level Self-Evolution Training for Sequence-to-Sequence Learning

Adaptive training approaches, widely used in sequence-to-sequence models, commonly reweigh the losses of different target tokens based on priors, e.g.word frequency.However, most of them do not consider the variation of learning difficulty in different training steps, and overly emphasize the learning of difficult one-hot labels, making the learning deterministic and sub-optimal.In response, we present Token-Level Self-Evolution Training (SE), a simple and effective dynamic training method to fully and wisely exploit the knowledge from data.SE focuses on dynamically learning the under-explored tokens for each forward pass and adaptively regularizes the training by introducing a novel token-specific label smoothing approach.Empirically, SE yields consistent and significant improvements in three tasks, i.e. machine translation, summarization, and grammatical error correction.Encouragingly, we achieve averaging +0.93 BLEU improvement on three machine translation tasks.Analyses confirm that, besides improving lexical accuracy, SE enhances generation diversity and model generalization.
Extract and Attend: Improving Entity Translation in Neural Machine Translation

While Neural Machine Translation (NMT) has achieved great progress in recent years, it still suffers from inaccurate translation of entities (e.g., person/organization name, location), due to the lack of entity training instances.When we humans encounter an unknown entity during translation, we usually first look up in a dictionary and then organize the entity translation together with the translations of other parts to form a smooth target sentence.Inspired by this translation process, we propose an Extract-and-Attend approach to enhance entity translation in NMT, where the translation candidates of source entities are first extracted from a dictionary and then attended to by the NMT model to generate the target sentence.Specifically, the translation candidates are extracted by first detecting the entities in a source sentence and then translating the entities through looking up in a dictionary.Then, the extracted candidates are added as a prefix of the decoder input to be attended to by the decoder when generating the target sentence through self-attention.Experiments conducted on En-Zh and En-Ru demonstrate that the proposed method is effective on improving both the translation accuracy of entities and the overall translation quality, with up to 35% reduction on entity error rate and 0.85 gain on BLEU and 13.8 gain on COMET.
PoliTo at SemEval-2023 Task 1: CLIP-based Visual-Word Sense Disambiguation Based on Back-Translation

Visual-Word Sense Disambiguation (V-WSD) entails resolving the linguistic ambiguity in a text by selecting a clarifying image from a set of (potentially misleading) candidates.In this paper, we address V-WSD using a stateof-the-art Image-Text Retrieval system, namely CLIP.We propose to alleviate the linguistic ambiguity across multiple domains and languages via text and image augmentation.To augment the textual content we rely on backtranslation with the aid of a variety of auxiliary languages.The approach based on fine-tuning CLIP on the full phrases is effective in accurately disambiguating words and incorporating back-translation enhances the system's robustness and performance on the test samples written in Indo-European languages.
The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics

Many NLP tasks exhibit human label variation, where different annotators give different labels to the same texts.This variation is known to depend, at least in part, on the sociodemographics of annotators.Recent research aims to model individual annotator behaviour rather than predicting aggregated labels, and we would expect that sociodemographic information is useful for these models.On the other hand, the ecological fallacy states that aggregate group behaviour, such as the behaviour of the average female annotator, does not necessarily explain individual behaviour.To account for sociodemographics in models of individual annotator behaviour, we introduce group-specific layers to multi-annotator models.In a series of experiments for toxic content detection, we find that explicitly accounting for sociodemographic attributes in this way does not significantly improve model performance.This result shows that individual annotation behaviour depends on much more than just sociodemographics.
ECOLA: Enhancing Temporal Knowledge Embeddings with Contextualized Language Representations

Since conventional knowledge embedding models cannot take full advantage of the abundant textual information, there have been extensive research efforts in enhancing knowledge embedding using texts.However, existing enhancement approaches cannot apply to temporal knowledge graphs (tKGs), which contain time-dependent event knowledge with complex temporal dynamics.Specifically, existing enhancement approaches often assume knowledge embedding is time-independent.In contrast, the entity embedding in tKG models usually evolves, which poses the challenge of aligning temporally relevant texts with entities.To this end, we propose to study enhancing temporal knowledge embedding with textual data in this paper.As an approach to this task, we propose Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA), which takes the temporal aspect into account and injects textual information into temporal knowledge embedding.To evaluate ECOLA, we introduce three new datasets for training and evaluating ECOLA.Extensive experiments show that ECOLA significantly enhances temporal KG embedding models with up to 287% relative improvements regarding Hits@1 on the link prediction task.The code and models are publicly available ‡ .
The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing

In this work, we analyse the role of output vocabulary for text-to-text (T2T) models on the task of SPARQL semantic parsing. We perform experiments within the the context of knowledge graph question answering (KGQA), where the task is to convert questions in natural language to the SPARQL query language. We observe that the query vocabulary is distinct from human vocabulary. Language Models (LMs) are pre-dominantly trained for human language tasks, and hence, if the query vocabulary is replaced with a vocabulary more attuned to the LM tokenizer, the performance of models may improve. We carry out carefully selected vocabulary substitutions on the queries and find absolute gains in the range of 17% on the GrailQA dataset.
MilaNLP at SemEval-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection

We present the system proposed by the Mi-laNLP team for the Explainable Detection of Online Sexism (EDOS) shared task.We propose an ensemble modeling approach to combine different classifiers trained with domain adaptation objectives and standard fine-tuning.Our results show that the ensemble is more robust than individual models and that regularized models generate more "conservative" predictions, mitigating the effects of lexical overfitting. 1 However, our error analysis also finds that many of the misclassified instances are debatable, raising questions about the objective annotatability of hate speech data.Warning: This paper contains examples of language that some people may find offensive.
LET: Leveraging Error Type Information for Grammatical Error Correction

Grammatical error correction (GEC) aims to correct errors in given sentences and is significant to many downstream natural language understanding tasks.Recent work introduces the idea of grammatical error detection (GED) to improve the GEC task performance.In contrast, these explicit multi-stage works propagate and amplify the problem of misclassification of the GED module.To introduce more convincing error type information, we propose an end-toend framework in this paper, which Leverages Error Type (LET) information in the generation process.First, the input text is fed into a classification module to obtain the error type corresponding to each token.Then, we introduce the category information into the decoder's input and cross-attention module in two ways, respectively.Experiments on various datasets show that our proposed method outperforms existing methods by a clear margin.
Boosting Zero-shot Cross-lingual Retrieval by Training on Artificially Code-Switched Data

Transferring information retrieval (IR) models from a high-resource language (typically English) to other languages in a zero-shot fashion has become a widely adopted approach.In this work, we show that the effectiveness of zero-shot rankers diminishes when queries and documents are present in different languages.Motivated by this, we propose to train ranking models on artificially code-switched data instead, which we generate by utilizing bilingual lexicons.To this end, we experiment with lexicons induced from (1) cross-lingual word embeddings and (2) parallel Wikipedia page titles.We use the mMARCO dataset to extensively evaluate reranking models on 36 language pairs spanning Monolingual IR (MoIR), Cross-lingual IR (CLIR), and Multilingual IR (MLIR).Our results show that code-switching can yield consistent and substantial gains of 5.1 MRR@10 in CLIR and 3.9 MRR@10 in MLIR, while maintaining stable performance in MoIR.Encouragingly, the gains are especially pronounced for distant languages (up to 2x absolute gain).We further show that our approach is robust towards the ratio of codeswitched tokens and also extends to unseen languages.Our results demonstrate that training on code-switched data is a cheap and effective way of generalizing zero-shot rankers for crosslingual and multilingual retrieval.
HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language

This paper presents HaVQA, the first multimodal dataset for visual question-answering (VQA) tasks in the Hausa language.The dataset was created by manually translating 6,022 English question-answer pairs, which are associated with 1,555 unique images from the Visual Genome dataset.As a result, the dataset provides 12,044 gold standard English-Hausa parallel sentences that were translated in a fashion that guarantees their semantic match with the corresponding visual information.We conducted several baseline experiments on the dataset, including visual question answering, visual question elicitation, text-only and multimodal machine translation.
A Weakly Supervised Classifier and Dataset of White Supremacist Language

We present a dataset and classifier for detecting the language of white supremacist extremism, a growing issue in online hate speech.Our weakly supervised classifier is trained on large datasets of text from explicitly white supremacist domains paired with neutral and anti-racist data from similar domains.We demonstrate that this approach improves generalization performance to new domains.Incorporating anti-racist texts as counterexamples to white supremacist language mitigates bias.
Using contradictions improves question answering systems

This work examines the use of contradiction in natural language inference (NLI) for question answering (QA).Typically, NLI systems help answer questions by determining if a potential answer is entailed (supported) by some background context.But is it useful to also determine if an answer contradicts the context?We test this in two settings, multiple choice and extractive QA, and find that systems that incorporate contradiction can do slightly better than entailment-only systems on certain datasets.However, the best performances come from using contradiction, entailment, and QA model confidence scores together.This has implications for the deployment of QA systems in domains such as medicine and science where safety is an issue.
LT at SemEval-2023 Task 1: Effective Zero-Shot Visual Word Sense Disambiguation Approaches using External Knowledge Sources

The objective of the SemEval-2023 Task 1: Visual Word Sense Disambiguation (VWSD) (Raganato et al., 2023) is to identify the correct image illustrating the indented meaning of a target word and some minimal additional context.The omnipresence of textual and visual data in the task strongly suggests the utilization of the recent advances in multi-modal machine learning, i.e., pretrained visiolinguistic models (VLMs).Often referred to as foundation models due to their strong performance on many vision-language downstream tasks, these models further demonstrate powerful zero-shot capabilities.In this work, we utilize various pertained VLMs in a zero-shot fashion for multiple approaches using external knowledge sources to enrich the contextual information.Further, we evaluate our methods on the final test data and extensively analyze the suitability of different knowledge sources, the influence of training data, model sizes, multi-linguality, and different textual prompting strategies.Although we are not among the best-performing systems (rank 20 of 56), our experiments described in this work prove competitive results.Moreover, we aim to contribute meaningful insights and propel multi-modal machine learning tasks like VWSD.
Ellipsis-Dependent Reasoning: a New Challenge for Large Language Models

We propose a novel challenge for large language models: ellipsis-dependent reasoning.We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis.Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.
Incorporating Attribution Importance for Improving Faithfulness Metrics

Feature attribution methods (FAs) are popular approaches for providing insights into the model reasoning process of making predictions.The more faithful a FA is, the more accurately it reflects which parts of the input are more important for the prediction.Widely used faithfulness metrics, such as sufficiency and comprehensiveness use a hard erasure criterion, i.e. entirely removing or retaining the top most important tokens ranked by a given FA and observing the changes in predictive likelihood.However, this hard criterion ignores the importance of each individual token, treating them all equally for computing sufficiency and comprehensiveness.In this paper, we propose a simple yet effective soft erasure criterion.Instead of entirely removing or retaining tokens from the input, we randomly mask parts of the token vector representations proportionately to their FA importance.Extensive experiments across various natural language processing tasks and different FAs show that our soft-sufficiency and softcomprehensiveness metrics consistently prefer more faithful explanations compared to hard sufficiency and comprehensiveness. 1
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation

A major challenge in the field of Text Generation is evaluation: Human evaluations are cost-intensive, and automated metrics often display considerable disagreement with human judgments.In this paper, we propose a statistical model of Text Generation evaluation that accounts for the error-proneness of automated metrics when used to generate preference rankings between system outputs.We show that existing automated metrics are generally overconfident in assigning significant differences between systems in this setting.However, our model enables an efficient combination of human and automated ratings to remedy the errorproneness of the automated metrics.We show that using this combination, we only require about 50% of the human annotations typically used in evaluations to arrive at robust and statistically significant results while yielding the same evaluation outcome as the pure human evaluation in 95% of cases.We showcase the benefits of approach for three text generation tasks: dialogue systems, machine translation, and text summarization.
The State of Profanity Obfuscation in Natural Language Processing Scientific Publications

Work on hate speech has made considering rude and harmful examples in scientific publications inevitable.This situation raises various problems, such as whether or not to obscure profanities.While science must accurately disclose what it does, the unwarranted spread of hate speech can harm readers and increases its internet frequency.While maintaining publications' professional appearance, obfuscating profanities makes it challenging to evaluate the content, especially for non-native speakers.Surveying 150 ACL papers, we discovered that obfuscation is usually used for English but not other languages, and even then, quite unevenly.We discuss the problems with obfuscation and suggest a multilingual community resource called PROF with a Python module to standardize profanity obfuscation processes.We believe PROF can help scientific publication policies to make hate speech work accessible and comparable, irrespective of language.Warning: this paper contains unobfuscated examples some readers may find offensive.
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind

Dialogue participants may have varying levels of knowledge about the topic under discussion.In such cases, it is essential for speakers to adapt their utterances by taking their audience into account.Yet, it is an open question how such adaptation can be modelled in computational agents.In this paper, we model a visually grounded referential game between a knowledgeable speaker and a listener with more limited visual and linguistic experience.Inspired by psycholinguistic theories, we endow our speaker with the ability to adapt its referring expressions via a simulation module that monitors the effectiveness of planned utterances from the listener's perspective.We propose an adaptation mechanism building on plug-andplay approaches to controlled language generation, where utterance generation is steered on the fly by the simulator without finetuning the speaker's underlying language model.Our results and analyses show that our approach is effective: the speaker's utterances become closer to the listener's domain of expertise, which leads to higher communicative success.
Text Generation Model Enhanced with Semantic Information in Aspect Category Sentiment Analysis

Aspect Category Sentiment Analysis (ACSA) is one of the main subtasks of sentiment analysis, which aims at predicting polarity over a given aspect category.Recently, generative methods have emerged as an efficient way to use a pre-trained language model for ACSA.However, those methods fail to model relations between target words and opinion words in a sentence including multiple aspects.To tackle this problem, this paper proposes a method to incorporate Abstract Meaning Representation (AMR), which describes the semantic representation of a sentence as a directed graph, into a text generation model.Furthermore, two regularizers are designed to guide the allocation of cross attention weights over AMR graphs.One is the identical regularizer, which constrains the attention weights of aligned nodes, the other is the entropy regularizer, which helps the decoder generate tokens by only giving a high degree of consideration to a few related nodes in the AMR graph.Experimental results on three datasets show that the proposed method outperforms state-of-the-art methods, proving the effectiveness of our model.
DCU at SemEval-2023 Task 10: A Comparative Analysis of Encoder-only and Decoder-only Language Models with Insights into Interpretability

We conduct a comparison of pre-trained encoder-only and decoder-only language models with and without continued pre-training, to detect online sexism.Our fine-tuning-based classifier system achieved the 16 th rank in the SemEval 2023 Shared Task 10 Subtask A that asks to distinguish sexist and non-sexist texts.Additionally, we conduct experiments aimed at enhancing the interpretability of systems designed to detect online sexism.Our findings provide insights into the features and decisionmaking processes underlying our classifier system, thereby contributing to a broader effort to develop explainable AI models to detect online sexism.
ActiveAED: A Human in the Loop Improves Annotation Error Detection

Manually annotated datasets are crucial for training and evaluating Natural Language Processing models.However, recent work has discovered that even widely-used benchmark datasets contain a substantial number of erroneous annotations.This problem has been addressed with Annotation Error Detection (AED) models, which can flag such errors for human re-annotation.However, even though many of these AED methods assume a final curation step in which a human annotator decides whether the annotation is erroneous, they have been developed as static models without any human-in-the-loop component.In this work, we propose ActiveAED, an AED method that can detect errors more accurately by repeatedly querying a human for error corrections in its prediction loop.We evaluate ActiveAED on eight datasets spanning five different tasks and find that it leads to improvements over the state of the art on seven of them, with gains of up to six percentage points in average precision.
SUT at SemEval-2023 Task 1: Prompt Generation for Visual Word Sense Disambiguation

Visual Word Sense Disambiguation (V-WSD) identifies the correct visual sense of a multisense word in a specific context.This can be challenging as images may need to provide additional context and words may have multiple senses.A proper V-WSD system can benefit applications like image retrieval and captioning.This paper proposes a Prompt Generation approach to solve this challenge.This approach improves the robustness of languageimage models like CLIP to contextual ambiguities and helps them better correlate between textual and visual contexts of different senses of words.