Reviewerly/exHarmony huggingface dataset:
MS@IW at SemEval-2022 Task 4: Patronising and Condescending Language Detection with Synthetically Generated Data

In this description paper we outline the system architecture submitted to Task 4, Subtask 1 at SemEval-2022.We leverage the generative power of state-of-the-art generative pretrained transformer models to increase training set size and remedy class imbalance issues.Our best submitted system is trained on a synthetically enhanced dataset with 10.3 times as many positive samples as the original dataset and reaches an F1 score of 50.62%, which is 10 percentage points higher than our initial system trained on an undersampled version of the original dataset.We explore possible reasons for the comparably low score in the overall task ranking and report on experiments conducted during the post-evaluation phase.
HATE-ITA: Hate Speech Detection in Italian Social Media Text

Warning: This paper contains examples of language that some people may find offensive.Online hate speech is a dangerous phenomenon that can (and should) be promptly counteracted properly.While Natural Language Processing has been successfully used for the purpose, many of the research efforts are directed toward the English language.This choice severely limits the classification power in non-English languages.In this paper, we test several learning frameworks for identifying hate speech in Italian text.We release HATE-ITA, a set of multilanguage models trained on a large set of English data and available Italian datasets.HATE-ITA performs better than mono-lingual models and seems to adapt well also on languagespecific slurs.We believe our findings will encourage research in other mid-to-low resource communities and provide a valuable benchmarking tool for the Italian community.
ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System

This paper introduces our proposed system for the MIA Shared Task on Cross-lingual Openretrieval Question Answering (COQA).In this challenging scenario, given an input question the system has to gather evidence documents from a multilingual pool and generate from them an answer in the language of the question.We devised several approaches combining different model variants for three main components: Data Augmentation, Passage Retrieval, and Answer Generation.For passage retrieval, we evaluated the monolingual BM25 ranker against the ensemble of re-rankers based on multilingual pretrained language models (PLMs) and also variants of the shared task baseline, re-training it from scratch using a recently introduced contrastive loss that maintains a strong gradient signal throughout training by means of mixed negative samples.For answer generation, we focused on languageand domain-specialization by means of continued language model (LM) pretraining of existing multilingual encoders.Additionally, for both passage retrieval and answer generation, we augmented the training data provided by the task organizers with automatically generated question-answer pairs created from Wikipedia passages to mitigate the issue of data scarcity, particularly for the low-resource languages for which no training data were provided.Our results show that language-and domain-specialization as well as data augmentation help, especially for low-resource languages.
What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured

Transformer-based models are now predominant in NLP.They outperform approaches based on static models in many respects.This success has in turn prompted research that reveals a number of biases in the language models generated by transformers.In this paper we utilize this research on biases to investigate to what extent transformer-based language models allow for extracting knowledge about object relations (X occurs in Y ; X consists of Z; action A involves using X).To this end, we compare contextualized models with their static counterparts.We make this comparison dependent on the application of a number of similarity measures and classifiers.Our results are threefold: Firstly, we show that the models combined with the different similarity measures differ greatly in terms of the amount of knowledge they allow for extracting.Secondly, our results suggest that similarity measures perform much worse than classifier-based approaches.Thirdly, we show that, surprisingly, static models perform almost as well as contextualized models -in some cases even better.
reamtchka at SemEval-2022 Task 6: Investigating the effect of different loss functions for Sarcasm detection for unbalanced datasets

This paper describes the system used in SemEval-2022 Task 6: Intended Sarcasm Detection in English and Arabic.Achieving 20th, 3rd places with 34& 47 F1-Sarcastic score for task A, 16th place for task B with 0.0560 F1macro score, and 10, 6th places for task C with 72% and 80% accuracy on the leaderboard.A voting classifier between either multiple different BERT-based models or machine learning models is proposed, as our final model.Multiple key points have been extensively examined to overcome the problem of the unbalance of the dataset as: type of models, suitable architecture, augmentation, loss function, etc.In addition to that, we present an analysis of our results in this work, highlighting its strengths and shortcomings.
Mitra Behzadi at SemEval-2022 Task 5 : Multimedia Automatic Misogyny Identification method based on CLIP

Everyday more users are using memes on social media platforms to convey a message with text and image combined.Although there are many fun and harmless memes being created and posted, there are also ones that are hateful and offensive to particular groups of people.In this article present a novel approach based on the CLIP (Radford et al., 2021) network to detect misogynous memes and find out the types of misogyny in that meme.We participated in Task A and Task B of the Multimedia Automatic Misogyny Identification (MaMi) challenge (Fersini et al., 2022) and our best scores are 0.694 and 0.681 respectively.
Morphology is not just a naive Bayes – UniMelb Submission to SIGMORPHON 2022 ST on Morphological Inflection

The paper describes the Flexica team's submission to the SIGMORPHON 2022 Shared Task 1 Part 1: Typologically Diverse Morphological Inflection.Our team submitted a nonneural system that extracted transformation patterns from alignments between a lemma and inflected forms.For each inflection category, we chose a pattern based on its abstractness score.The system outperformed the non-neural baseline, the extracted patterns covered a substantial part of possible inflections.However, we discovered that such score that does not account for all possible combinations of string segments as well as morphosyntactic features is not sufficient for a certain proportion of inflection cases.
Samsung Research Poland (SRPOL) at SemEval-2022 Task 9: Hybrid Question Answering Using Semantic Roles

In this work we present an overview of our winning system for the R2VQ -Competencebased Multimodal Question Answering task, with the final exact match score of 92.53%.The task is structured as question-answer pairs, querying how well a system is capable of competence-based comprehension of recipes.We propose a hybrid of a rule-based system, Question Answering Transformer, and a neural classifier for N/A answers recognition.The rule-based system focuses on intent identification, data extraction and response generation.
The Limits of Word Level Differential Privacy

As the issues of privacy and trust are receiving increasing attention within the research community, various attempts have been made to anonymize textual data.A significant subset of these approaches incorporate differentially private mechanisms to perturb word embeddings, thus replacing individual words in a sentence.While these methods represent very important contributions, have various advantages over other techniques and do show anonymization capabilities, they have several shortcomings.In this paper, we investigate these weaknesses and demonstrate significant mathematical constraints diminishing the theoretical privacy guarantee as well as major practical shortcomings with regard to the protection against deanonymization attacks, the preservation of content of the original sentences as well as the quality of the language output.Finally, we propose a new method for text anonymization based on transformer based language models fine-tuned for paraphrasing that circumvents most of the identified weaknesses and also offers a formal privacy guarantee.We evaluate the performance of our method via thorough experimentation and demonstrate superior performance over the discussed mechanisms.
EMBEDDIA at SemEval-2022 Task 8: Investigating Sentence, Image, and Knowledge Graph Representations for Multilingual News Article Similarity

In this paper, we present the participation of the EMBEDDIA team in the SemEval-2022 Task 8 (Multilingual News Article Similarity).We cover several techniques and propose different methods for finding the multilingual news article similarity by exploring the dataset in its entirety.We take advantage of the textual content of the articles, the provided metadata (e.g., titles, keywords, topics), the translated articles, the images (those that were available), and knowledge graph-based representations for entities and relations present in the articles.We, then, compute the semantic similarity between the different features and predict through regression the similarity scores.Our findings show that, while our proposed methods obtained promising results, exploiting the semantic textual similarity with sentence representations is unbeatable.Finally, in the official SemEval-2022 Task 8, we ranked fifth in the overall team ranking cross-lingual results, and second in the English-only results.
A Question-Answer Driven Approach to Reveal Affirmative Interpretations from Verbal Negations

This paper explores a question-answer driven approach to reveal affirmative interpretations from verbal negations (i.e., when a negation cue grammatically modifies a verb).We create a new corpus consisting of 4,472 verbal negations and discover that 67.1% of them convey that an event actually occurred.Annotators generate and answer 7,277 questions for the 3,001 negations that convey an affirmative interpretation.We first cast the problem of revealing affirmative interpretations from negations as a natural language inference (NLI) classification task.Experimental results show that state-ofthe-art transformers trained with existing NLI corpora are insufficient to reveal affirmative interpretations.We also observe, however, that fine-tuning brings small improvements.In addition to NLI classification, we also explore the more realistic task of generating affirmative interpretations directly from negations with the T5 transformer.We conclude that the generation task remains a challenge as T5 substantially underperforms humans.
DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre–trained ELECTRA

This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts.Given an instructional text with an omitted token, the task requires models to classify or rank the plausibility of potential fillers.To solve the task, we fine-tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausibility.This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%.However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A.
Partner Personas Generation for Dialogue Response Generation

Incorporating personas information allows diverse and engaging responses in dialogue response generation.Unfortunately, prior works have primarily focused on self personas and have overlooked the value of partner personas.Moreover, in practical applications, the availability of the gold partner personas is often not the case.This paper attempts to tackle these issues by offering a novel framework that leverages automatic partner personas generation to enhance the succeeding dialogue response generation.Our framework employs reinforcement learning with a dedicatedly designed critic network for reward judgement.Experimental results from automatic and human evaluations indicate that our framework is capable of generating relevant, interesting, coherent and informative partner personas, even compared to the ground truth partner personas.This enhances the succeeding dialogue response generation, which surpasses our competitive baselines that condition on the ground truth partner personas.
Learning Open Domain Multi-hop Search Using Reinforcement Learning

We propose a method to teach an automated agent to learn how to search for multi-hop paths of relations between entities in an open domain.The method learns a policy for directing existing information retrieval and machine reading resources to focus on relevant regions of a corpus.The approach formulates the learning problem as a Markov decision process with a state representation that encodes the dynamics of the search process and a reward structure that minimizes the number of documents that must be processed while still finding multihop paths.We implement the method in an actor-critic reinforcement learning algorithm and evaluate it on a dataset of search problems derived from a subset of English Wikipedia.The algorithm finds a family of policies that succeeds in extracting the desired information while processing fewer documents compared to several baseline heuristic algorithms.
Efficient Semi-supervised Consistency Training for Natural Language Understanding

Manually labeled training data is expensive, noisy, and often scarce, such as when developing new features or localizing existing features for a new region.In cases where labeled data is limited but unlabeled data is abundant, semi-supervised learning methods such as consistency training can be used to improve model performance, by training models to output consistent predictions between original and augmented versions of unlabeled data.In this work, we explore different data augmentation methods for consistency training (CT) on Natural Language Understanding (NLU) domain classification (DC) in the limited labeleddata regime.We explore three types of augmentation techniques (human paraphrasing, backtranslation, and dropout) for unlabeled data and train DC models to jointly minimize both the supervised loss and the consistency loss on unlabeled data.Our results demonstrate that DC models trained with CT methods and dropoutbased augmentation on only 0.1% (2,998 instances) of labeled data with the remainder as unlabeled can achieve a top-1 relative accuracy reduction of 12.25% compared to fully supervised model trained with 100% of labeled data, outperforming fully supervised models trained on 10x that amount of labeled data.The dropout-based augmentation achieves similar performance compare to back-translationbased augmentation with much less computational resources.This paves the way for applications of using large scale unlabeled data for semi-supervised learning in production NLU systems.
HITMI&amp;T at SemEval-2022 Task 4: Investigating Task-Adaptive Pretraining And Attention Mechanism On PCL Detection

This paper describes the system for the Semeval-2022 Task4 "Patronizing and Condescending Language Detection".An entity engages in Patronizing and Condescending Language(PCL) when its language use shows a superior attitude towards others or depicts them in a compassionate way.The task contains two parts.The first one is to identify whether the sentence is PCL, and the second one is to categorize PCL.Through experimental verification, the RoBERTa-based model will be used in our system.Respectively, for subtask 1, that is, to judge whether a sentence is PCL, the method of retraining the model with specific task data is adopted, and the method of splicing [CLS] and the keyword representation of the last three layers as the representation of the sentence; for subtask 2, that is, to judge the PCL type of the sentence, in addition to using the same method as task1, the method of selecting a special loss for Multi-label text classification is applied.We give a clear ablation experiment and give the effect of each method on the final result.Our project ranked 11th out of 79 teams participating in subtask 1 and 6th out of 49 teams participating in subtask 2.
Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems

Skill routing is an important component in large-scale conversational systems.In contrast to traditional rule-based skill routing, state-ofthe-art systems use a model-based approach to enable natural conversations.To provide supervision signal required to train such models, ideas such as human annotation, replication of a rule-based system, relabeling based on user paraphrases, and bandit-based learning were suggested.However, these approaches: (a) do not scale in terms of the number of skills and skill on-boarding, (b) require a very costly expert annotation/rule-design, (c) introduce risks in the user experience with each model update.In this paper, we present a scalable self-learning approach to explore routing alternatives without causing abrupt policy changes that break the user experience, learn from the user interaction, and incrementally improve the routing via frequent model refreshes.To enable such robust frequent model updates, we suggest a simple and effective approach that ensures controlled policy updates for individual domains, followed by an off-policy evaluation for making deployment decisions without any need for lengthy A/B experimentation.We conduct various offline and online A/B experiments on a commercial large-scale conversational system to demonstrate the effectiveness of the proposed method in real-world production settings.
Towards a Deep Multi-layered Dialectal Language Analysis: A Case Study of African-American English

Currently, natural language processing (NLP) models proliferate language discrimination leading to potentially harmful societal impacts as a result of biased outcomes.For example, part-of-speech taggers trained on Mainstream American English (MAE) produce noninterpretable results when applied to African American English (AAE) as a result of language features not seen during training.In this work, we incorporate a human-in-the-loop paradigm to gain a better understanding of AAE speakers' behavior and their language use, and highlight the need for dialectal language inclusivity so that native AAE speakers can extensively interact with NLP systems while reducing feelings of disenfranchisement.
SIGMORPHON 2022 Task 0 Submission Description: Modelling Morphological Inflection with Data-Driven and Rule-Based Approaches

This paper describes our participation in the 2022 SIGMORPHON-UniMorph Shared Task on Typologically Diverse and Acquisition-Inspired Morphological Inflection Generation.We present two approaches: one being a modification of the neural baseline encoder-decoder model, the other being hand-coded morphological analyzers using finite-state tools (FST) and outside linguistic knowledge.While our proposed modification of the baseline encoderdecoder model underperforms the baseline for almost all languages, the FST methods outperform other systems in the respective languages by a large margin.This confirms that purely data-driven approaches have not yet reached the maturity to replace trained linguists for documentation and analysis especially considering low-resource and endangered languages.
FarFetched: Entity-centric Reasoning and Claim Validation for the Greek Language based on Textually Represented Environments

Our collective attention span is shortened by the flood of online information.With FarFetched, we address the need for automated claim validation based on the aggregated evidence derived from multiple online news sources.We introduce an entity-centric reasoning framework in which latent connections between events, actions, or statements are revealed via entity mentions and represented in a graph database.Using entity linking and semantic similarity, we offer a way for collecting and combining information from diverse sources in order to generate evidence relevant to the user's claim.Then, we leverage textual entailment recognition to quantitatively determine whether this assertion is credible, based on the created evidence.Our approach tries to fill the gap in automated claim validation for less-resourced languages and is showcased on the Greek language, complemented by the training of relevant semantic textual similarity (STS) and natural language inference (NLI) models that are evaluated on translated versions of common benchmarks.User Claim (Scenario 1) NLI score Denmark and Austria believe that the European Union should increase aid to refugees.(1a) c: 0.014 e: 0.958 n: 0.028 Denmark disagrees with Austria on the management of immigration issues in the European Union.(1b) c: 0.951 e: 0.002 n: 0.047 Candidate Evidence Sequences (↓ similarity)Austria and Denmark also want to increase EU support for countries hosting refugees near crisis hotspots so that they do not travel to Europe.
Hitachi at SemEval-2022 Task 2: On the Effectiveness of Span-based Classification Approaches for Multilingual Idiomaticity Detection

In this paper, we describe our system for SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding.The task aims at detecting idiomaticity in an input sequence (Subtask A) and modeling representation of sentences that contain potential idiomatic multiword expressions (MWEs) (Subtask B) in three languages.We focus on the zero-shot setting of Subtask A and propose two span-based idiomaticity classification methods: MWE span-based classification and idiomatic MWE span prediction-based classification.We use several cross-lingual pre-trained language models (InfoXLM, XLM-R, and others) as our backbone network.Our best-performing system, fine-tuned with the span-based idiomaticity classification, ranked fifth in the zero-shot setting of Subtask A and exhibited a macro F 1 score of 0.7466.
Exploring Contrastive Learning for Multimodal Detection of Misogynistic Memes

Misogynistic memes are rampant on social media, and often convey their messages using multimodal signals (e.g., images paired with derogatory text or captions).However, to date very few multimodal systems have been leveraged for the detection of misogynistic memes.Recently, researchers have turned to contrastive learning solutions for a variety of problems.Most notably, OpenAI's CLIP model has served as an innovative solution for a variety of multimodal tasks.In this work, we experiment with contrastive learning to address the detection of misogynistic memes within the context of SemEval-2022 Task 5.Although our model does not achieve top results, these experiments provide important exploratory findings for this task.We conduct a detailed error analysis, revealing promising clues and offering a foundation for follow-up work.