Reviewerly/exHarmony huggingface dataset:
NER4ID at SemEval-2022 Task 2: Named Entity Recognition for Idiomaticity Detection

Idioms are lexically-complex phrases whose meaning cannot be derived by compositionally interpreting their components.Although the automatic identification and understanding of idioms is essential for a wide range of Natural Language Understanding tasks, they are still largely under-investigated.This motivated the organization of the SemEval-2022 Task 2, which is divided into two multilingual subtasks: one about idiomaticity detection, and the other about sentence embeddings.In this work, we focus on the first subtask and propose a Transformer-based dual-encoder architecture to compute the semantic similarity between a potentially-idiomatic expression and its context and, based on this, predict idiomaticity.Then, we show how and to what extent Named Entity Recognition can be exploited to reduce the degree of confusion of idiom identification systems and, therefore, improve performance.Our model achieves 92.1 F 1 in the one-shot setting and shows strong robustness towards unseen idioms achieving 77.4 F 1 in the zeroshot setting.We release our code at https: //github.com/Babelscape/ner4id.
AliEdalat at SemEval-2022 Task 4: Patronizing and Condescending Language Detection using Fine-tuned Language Models, BERT+BiGRU, and Ensemble Models

This paper presents the AliEdalat team's methodology and results in SemEval-2022 Task 4: Patronizing and Condescending Language (PCL) Detection.This task aims to detect the presence of PCL and PCL categories in text in order to prevent further discrimination against vulnerable communities.We use an ensemble of three basic models to detect the presence of PCL: fine-tuned bigbird, fine-tuned mpnet, and BERT+BiGRU.The ensemble model performs worse than the baseline due to overfitting and achieves an F1-score of 0.3031.We offer another solution to resolve the submitted model's problem.We consider the different categories of PCL separately.To detect each category of PCL, we act like a PCL detector.Instead of BERT+BiGRU, we use fine-tuned roberta in the models.In PCL category detection, our model outperforms the baseline model and achieves an F1-score of 0.2531.We also present new models for detecting two categories of PCL that outperform the submitted models.
HuaAMS at SemEval-2022 Task 8: Combining Translation and Domain Pre-training for Cross-lingual News Article Similarity

This paper describes our submission to SemEval-2022 Multilingual News Article Similarity task.We experiment with different approaches that utilize a pre-trained language model fitted with a regression head to predict similarity scores for a given pair of news articles.Our best performing systems include 2 key steps: 1) pre-training with in-domain data 2) training data enrichment through machine translation.Our final submission is an ensemble of predictions from our top systems.While we show the significance of pre-training and augmentation, we believe the issue of language coverage calls for more attention.
PALI-NLP at SemEval-2022 Task 6: iSarcasmEval- Fine-tuning the Pre-trained Model for Detecting Intended Sarcasm

This paper describes the method we utilized in the SemEval-2022 Task 6 iSarcasmEval: Intended Sarcasm Detection In English and Arabic.Our system has achieved 1st in Sub-taskB, which is to identify the categories of intended sarcasm.The proposed system integrates multiple BERT-based, RoBERTa-based and BERTweet-based models with finetuning.In this task, our contribution is listed as follow: 1) we reveal several large pre-trained models' performance on tasks coping with the tweetlike text.2) Our methods prove that we can still achieve excellent results in this particular task without a complex classifier adopting some proper training method.3) we found there is a hierarchical relationship of sarcasm types in this task.
The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank

The task of inserting text into a specified position in a passage, known as fill in the blank (FITB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text.While previous work has tackled this problem with models trained specifically to do the fill-in-theblank task, a more useful model is one that can effectively perform both FITB and continuation.In this work, we evaluate the feasibility of using a single model to do both tasks.We show that models pre-trained with a FITBstyle objective are capable of both tasks, while models pre-trained for continuation are not.Finally, we show how FITB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.
DataScience-Polimi at SemEval-2022 Task 8: Stacking Language Models to Predict News Article Similarity

In this paper, we describe the approach we designed to solve SemEval-2022 Task 8: Multilingual News Article Similarity.We collect and use exclusively textual features (title, description and body) of articles.Our best model is a stacking of 14 Transformer-based Language models fine-tuned on single or multiple fields, using data in the original language or translated to English.It placed fourth on the original leaderboard, sixth on the complete official one and fourth on the English-subset official one.We observe the data collection as our principal source of error due to a relevant fraction of missing or wrong fields.
Cross-TOP: Zero-Shot Cross-Schema Task-Oriented Parsing

Deep learning methods have enabled taskoriented semantic parsing of increasingly complex utterances.However, a single model is still typically trained and deployed for each task separately, requiring labeled training data for each, which makes it challenging to support new tasks, even within a single business vertical (e.g., food-ordering or travel booking).In this paper we describe Cross-TOP (Cross-Schema Task-Oriented Parsing), a zero-shot method for complex semantic parsing in a given vertical.By leveraging the fact that user requests from the same vertical share lexical and semantic similarities, a single cross-schema parser is trained to service an arbitrary number of tasks, seen or unseen, within a vertical.We show that Cross-TOP can achieve high accuracy on a previously unseen task without requiring any additional training data, thereby providing a scalable way to bootstrap semantic parsers for new tasks.As part of this work we release the FoodOrdering dataset, a task-oriented parsing dataset in the food-ordering vertical, with utterances and annotations derived from five schemas, each from a different restaurant menu.
Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods

The ability to extract entities and their relations from unstructured text is essential for the automated maintenance of large-scale knowledge graphs.To keep a knowledge graph up-to-date, an extractor needs not only the ability to recall the triples it encountered during training, but also the ability to extract the new triples from the context that it has never seen before.In this paper, we show that although existing extraction models are able to easily memorize and recall already seen triples, they cannot generalize effectively for unseen triples.This alarming observation was previously unknown due to the composition of the test sets of the go-to benchmark datasets, which turns out to contain only 2% unseen data, rendering them incapable to measure the generalization performance.To separately measure the generalization performance from the memorization performance, we emphasize unseen data by rearranging datasets, sifting out training instances, or augmenting test sets.In addition to that, we present a simple yet effective augmentation technique to promote generalization of existing extraction models, and experimentally confirm that the proposed method can significantly increase the generalization performance of existing models.
SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

We describe SemEval-2022 Task 7, a shared task on rating the plausibility of clarifications in English-language instructional texts.The dataset for this task consists of manually clarified how-to guides for which we generated alternative clarifications and collected human plausibility judgements. 1 The task of participating systems was to automatically determine the plausibility of a clarification in the respective context.In total, 21 participants took part in this task, with the best system achieving an accuracy of 68.9%.This report summarizes the results and findings from 8 teams and their system descriptions.Finally, we show in an additional evaluation that predictions by the top participating team make it possible to identify contexts with multiple plausible clarifications with an accuracy of 75.2%.
Nikkei at SemEval-2022 Task 8: Exploring BERT-based Bi-Encoder Approach for Pairwise Multilingual News Article Similarity

This paper describes our system in SemEval-2022 Task 8, where participants were required to predict the similarity of two multilingual news articles.In the task of pairwise sentence and document scoring, there are two main approaches: Cross-Encoder, which inputs pairs of texts into a single encoder, and Bi-Encoder, which encodes each input independently.The former method often achieves higher performance, but the latter gave us a better result in SemEval-2022 Task 8.This paper presents our exploration of BERT-based Bi-Encoder approach for this task, and there are several findings such as pretrained models, pooling methods, translation, data separation, and the number of tokens.The weighted average ensemble of the four models achieved the competitive result and ranked in the top 12.
Transformers at SemEval-2022 Task 5: A Feature Extraction based Approach for Misogynous Meme Detection

Social media is an idea created to make the world smaller and more connected.Recently, it has become a hub of fake news and sexist memes that target women.Social Media should ensure proper women's safety and equality.Filtering such information from social media is of paramount importance to achieving this goal.In this paper, we describe the system developed by our team for SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification.We propose a multimodal training methodology that achieves good performance on both the subtasks, ranking 4th for Subtask A (0.718 macro F1-score) and 9th for Subtask B (0.695 macro F1-score) while exceeding the baseline results by good margins.The code will be available here 1
Comparing Distributional and Curated Approaches for Cross-lingual Frame Alignment

Despite advances in statistical approaches to the modeling of meaning, many questions about the ideal way of exploiting both knowledge-based (e.g., FrameNet, WordNet) and data-based methods (e.g., BERT) remain unresolved.This workshop focuses on these questions with three session papers that run the gamut from highly distributional methods (Lekkas et al., 2022), to highly curated methods (Gamonal, 2022), and techniques with statistical methods producing structured semantics (Lawley and Schubert, 2022).In addition, we begin the workshop with a small comparison of cross-lingual techniques for frame semantic alignment for one language pair (Spanish and English).None of the distributional techniques consistently aligns the 1-best frame match from English to Spanish, all failing in at least one case.Predicting which techniques will align which frames crosslinguistically is not possible from any known characteristic of the alignment technique or the frames.Although distributional techniques are a rich source of semantic information for many tasks, at present curated, knowledge-based semantics remains the only technique that can consistently align frames across languages.
Quantifying Synthesis and Fusion and their Impact on Machine Translation

Theoretical work in morphological typology offers the possibility of measuring morphological diversity on a continuous scale.However, literature in Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g.fusional or agglutinative.In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level.We consider Payne (2017)'s approach to classify morphology using two indices: synthesis (e.g.analytic to polysynthetic) and fusion (agglutinative to fusional).For computing synthesis, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study.Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish, and verbs in English-Spanish) and segment level (previous language pairs plus English-German in both directions).We complement the word-level analysis with human evaluation, and overall, we observe a consistent impact of both indexes on machine translation quality.
To Answer or Not To Answer? Improving Machine Reading Comprehension Model with Span-based Contrastive Learning

Machine Reading Comprehension with Unanswerable Questions is a difficult NLP task, challenged by the questions which can not be answered from passages.It is observed that subtle literal changes often make an answerable question unanswerable, however, most MRC models fail to recognize such changes.To address this problem, in this paper, we propose a span-based method of Contrastive Learning (spanCL) which explicitly contrast answerable questions with their answerable and unanswerable counterparts at the answer span level.With spanCL, MRC models are forced to perceive crucial semantic changes from slight literal differences.Experiments on SQuAD 2.0 dataset show that spanCL can improve baselines significantly, yielding 0.86~2.14absolute EM improvements.Additional experiments also show that spanCL is an effective way to utilize generated questions.
Explicit Use of Topicality in Dialogue Response Generation

The current chat dialogue systems implicitly consider the topic given the context, but not explicitly.As a result, these systems often generate inconsistent responses with the topic of the moment.In this study, we propose a dialogue system that responds appropriately following the topic by selecting the entity with the highest "topicality."In topicality estimation, the model is trained through self-supervised learning that regards entities appearing in both context and response as the topic entities.In response generation, the model is trained to generate topic-relevant responses based on the estimated topicality.Experimental results show that our proposed system can follow the topic more than the existing dialogue system that considers only the context.
Free Speech vs. Social Justice: Should Hate Speech be Protected?

Editor’s note: Publication by JIFP does not constitute an endorsement of the content or represent the official position of the Office for Intellectual Freedom (OIF) or the American Library Association (ALA). Commentaries, articles, and book reviews do not necessarily reflect the views of the editorial board of JIFP, OIF, ALA, or their affiliates.There is a troubling trend within the library profession. As a result of blind allegiance to free speech absolutism, librarians are forgetting that we exist in a society. Libraries are social spaces and therefore spaces for social justice.
JSI at SemEval-2022 Task 1: CODWOE - Reverse Dictionary: Monolingual and cross-lingual approaches

The reverse dictionary is a sequence-to-vector task in which a gloss is provided as input, and the model is trained to output a semantically matching word vector.The reverse dictionary is useful in practical applications such as solving the tip-of-the-tongue problem, helping new language learners, etc.In this paper, we evaluate the Transformer-based model with the added LSTM layer for the task at hand in a monolingual, multilingual, and cross-lingual zero-shot setting.Experiments are conducted in five languages in the COD-WOE dataset, namely English, French, Italian, Spanish, and Russian.Our work partially improves the current baseline of the CODWOE competition and offers insight into the feasibility of the cross-lingual methodology for the reverse dictionary task.The code is available at https://github.com/honghanhh/codwoe2021.
Context-Aware Language Modeling for Goal-Oriented Dialogue Systems

Goal-oriented dialogue systems face a tradeoff between fluent language generation and task-specific control.While supervised learning with large language models is capable of producing realistic text, how to steer such responses towards completing a specific task without sacrificing language quality remains an open question.In this work, we formulate goal-oriented dialogue as a partially observed Markov decision process, interpreting the language model as a representation of both the dynamics and the policy.This view allows us to extend techniques from learning-based control, such as task relabeling, to derive a simple and effective method to finetune language models in a goal-aware way, leading to significantly improved task performance.We additionally introduce a number of training strategies that serve to better focus the model on the task at hand.We evaluate our method, Context-Aware Language Models (CALM), on a practical flight-booking task using AirDialogue.Empirically, CALM outperforms the state-of-theart method by 7% in terms of task success, matching human-level task performance.
Investigating the Influence of Users Personality on the Ambiguous Emoji Perception

Emojis are an integral part of Internet communication nowadays.Even though, they are supposed to make the text clearer and less dubious, some emojis are ambiguous and can be interpreted in different ways.One of the factors that determine the perception of emojis is the user's personality.In this work, I conducted an experimental study and investigated how personality traits, measured with a Big Five Inventory (BFI) questionnaire, affect reaction time when interpreting emoji.For a set of emoji, for which there are several possible interpretations, participants had to determine whether the emoji fits the presented context or not.Using regression analysis, I found that conscientiousness and neuroticism significantly predict the reaction time the person needs to decide about the emoji.More conscientious people take longer to resolve ambiguity, while more neurotic people make decisions about ambiguous emoji faster.The knowledge of the relationship between personality and emoji interpretation can lead to effective use of knowledge of people's characters in personalizing interactive computer systems.
AaltoNLP at SemEval-2022 Task 11: Ensembling Task-adaptive Pretrained Transformers for Multilingual Complex NER

This paper presents the system description of team AaltoNLP for SemEval-2022 shared task 11: MultiCoNER.Transformer-based models have produced high scores on standard Named Entity Recognition (NER) tasks.However, accuracy on complex named entities is still low.Complex and ambiguous named entities have been identified as a major error source in NER tasks.The shared task is about multilingual complex named entity recognition.In this paper, we describe an ensemble approach, which increases accuracy across all tested languages.The system ensembles output from multiple same architecture task-adaptive pretrained transformers trained with different random seeds.We notice a large discrepancy between performance on development and test data.Model selection based on limited development data may not yield optimal results on large test data sets.
University of Bucharest Team at Semeval-2022 Task4: Detection and Classification of Patronizing and Condescending Language

This paper details our implementations for finding Patronizing and Condescending Language in texts, as part of the SemEval Workshop Task 4. We have used a variety of methods from simple machine learning algorithms applied on bag of words, all the way to BERT models, in order to solve the binary classification and the multi-label multi-class classification.
Learning to Borrow– Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion

Prior work on integrating text corpora with knowledge graphs (KGs) to improve Knowledge Graph Embedding (KGE) have obtained good performance for entities that co-occur in sentences in text corpora.Such sentences (textual mentions of entity-pairs) are represented as Lexicalised Dependency Paths (LDPs) between two entities.However, it is not possible to represent relations between entities that do not co-occur in a single sentence using LDPs.In this paper, we propose and evaluate several methods to address this problem, where we borrow LDPs from the entity pairs that cooccur in sentences in the corpus (i.e. with mention entity pairs) to represent entity pairs that do not co-occur in any sentence in the corpus (i.e.without mention entity pairs).We propose a supervised borrowing method, Su-perBorrow, that learns to score the suitability of an LDP to represent a without-mention entity pair using pre-trained entity embeddings and contextualised LDP representations.Experimental results show that SuperBorrow improves the link prediction performance of multiple widely-used prior KGE methods such as TransE, DistMult, ComplEx and RotatE.
Constraining word alignments with posterior regularization for label transfer

Unsupervised word alignments offer a lightweight and interpretable method to transfer labels from high- to low-resource languages, as long as semantically related words have the same label across languages. But such an assumption is often not true in industrial NLP pipelines, where multilingual annotation guidelines are complex and deviate from semantic consistency due to various factors (such as annotation difficulty, conflicting ontology, upcoming feature launches etc.);We address this difficulty by constraining the alignments models to remain consistent with both source and target annotation guidelines , leveraging posterior regularization and labeled examples. We illustrate the overall approach using IBM 2 (fast_align) as a base model, and report results on both internal and external annotated datasets. We measure consistent accuracy improvements on the MultiATIS++ dataset over AWESoME, a popular transformer-based alignment model, in the label projection task (+2.7% at word-level and +15% at sentence-level), and show how even a small amount of target language annotations help substantially.
BEIKE NLP at SemEval-2022 Task 4: Prompt-Based Paragraph Classification for Patronizing and Condescending Language Detection

PCL detection task is aimed at identifying and categorizing language that is patronizing or condescending towards vulnerable communities in the general media.Compared to other NLP tasks of paragraph classification, the negative language presented in the PCL detection task is usually more implicit and subtle to be recognized, making the performance of common textclassification approaches disappointed.Targeting the PCL detection problem in SemEval-2022 Task 4, in this paper, we give an introduction to our team's solution, which exploits the power of prompt-based learning on paragraph classification.We reformulate the task as an appropriate cloze prompt and use pretrained Masked Language Models to fill the cloze slot.For the two subtasks, binary classification and multi-label classification, DeBERTa model is adopted and fine-tuned to predict masked label words of task-specific prompts.On the evaluation dataset, for binary classification, our approach achieves an F1-score of 0.6406; for multi-label classification, our approach achieves an macro-F1-score of 0.4689 and ranks first in the leaderboard.