Reviewerly/exHarmony huggingface dataset:
HIT&amp;QMUL at SemEval-2022 Task 9: Label-Enclosed Generative Question Answering (LEG-QA)

This paper presents the second place system for the R2VQ: competence-based multimodal question answering shared task.The task consisted in building question answering systems that could process procedural recipes involving both text and image, and enriched with semantic and cooking roles.We tackled the task by using a text-to-text generative model based on the transformer architecture, with the aim of generalising across different question types.Our proposed architecture incorporates a novel approach for enriching input texts by incorporating semantic and cooking role labels through what we call Label-Enclosed Generative Question Answering (LEG-QA).Our model achieves a score of 91.3, with a significant improvement over the baseline (65.34) and close to the top-ranked system ((92.5).After describing the submitted system, we analyse the impact of the different components of LEG-QA as well as perform an error analysis.
A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation – through the Lens of Semantic Similarity Rating

Evaluating the quality of generated text is difficult, since traditional NLG evaluation metrics, focusing more on surface form than meaning, often fail to assign appropriate scores.This is especially problematic for AMR-to-text evaluation, given the abstract nature of AMR.Our work aims to support the development and improvement of NLG evaluation metrics that focus on meaning, by developing a dynamic CheckList for NLG metrics that is interpreted by being organized around meaning-relevant linguistic phenomena.Each test instance consists of a pair of sentences with their AMR graphs and a human-produced textual semantic similarity or relatedness score.Our Check-List facilitates comparative evaluation of metrics and reveals strengths and weaknesses of novel and traditional metrics.We demonstrate the usefulness of CheckList by designing a new metric GRACO that computes lexical cohesion graphs over AMR concepts.Our analysis suggests that GRACO presents an interesting NLG metric worth future investigation and that meaning-oriented NLG metrics can profit from graph-based metric components using AMR.
Differentially Private Instance Encoding against Privacy Attacks

TextHide was proposed to protect the training data via instance encoding in the natural language domain.Due to the lack of theoretic privacy guarantee, such instance encoding scheme has been shown to be vulnerable to privacy attacks, e.g., reconstruction attacks.To address such limitation, we integrate differential privacy into the instance encoding scheme, and thus provide a provable guarantee against privacy attacks.The experimental results also show that the proposed scheme can defend against privacy attacks while ensuring learning utility (as a trade-off).
OPI at SemEval-2022 Task 10: Transformer-based Sequence Tagging with Relation Classification for Structured Sentiment Analysis

This paper presents our solution for SemEval-2022 Task 10: Structured Sentiment Analysis.The solution consisted of two modules: the first for sequence tagging and the second for relation classification.In both modules we used transformer-based language models.In addition to utilizing language models specific to each of the five competition languages, we also adopted multilingual models.This approach allowed us to apply the solution to both monolingual and cross-lingual sub-tasks, where we obtained average Sentiment Graph F1 of 54.5% and 53.1%, respectively.The source code of the prepared solution is available at https://github.com/rafalposwiata/structuredsentiment-analysis.
UA-KO at SemEval-2022 Task 11: Data Augmentation and Ensembles for Korean Named Entity Recognition

This paper presents the approaches and systems of the UA-KO team for the Korean portion of SemEval-2022 Task 11 on Multilingual Complex Named Entity Recognition.We fine-tuned Korean and multilingual BERT and RoBERTA models, conducted experiments on data augmentation, ensembles, and task-adaptive pretraining.Our final system ranked 8th out of 17 teams with an F1 score of 0.6749 F1.
RIGA at SemEval-2022 Task 1: Scaling Recurrent Neural Networks for CODWOE Dictionary Modeling

Described are our two entries "emukans" and "guntis" for the definition modeling track of CODWOE SemEval-2022 Task 1.Our approach is based on careful scaling of a GRU recurrent neural network, which exhibits double descent of errors, corresponding to significant improvements also per human judgement.Our results are in the middle of the ranking table per official automatic metrics.
When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings

Recent work using word embeddings to model semantic categorization have indicated that static models outperform the more recent contextual class of models (Majewska et al., 2021).In this paper, we consider polysemy as a possible confounding factor, comparing sense-level embeddings with previously studied static embeddings on both coarse-and fine-grained categorization tasks.We find that the effect of polysemy depends on how one defines semantic categorization; while sense-level embeddings dramatically outperform static embeddings in predicting coarse-grained categories derived from a word sorting task, they perform approximately equally in predicting fine-grained categories derived from context-free similarity judgments.Our findings highlight the different processes underlying human behavior on different types of semantic tasks.
Stanford MLab at SemEval 2022 Task 7: Tree- and Transformer-Based Methods for Clarification Plausibility

In this paper, we detail the methods we used to determine the idiomaticity and plausibility of candidate words or phrases into an instructional text as part of the SemEval Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts.Given a set of steps in an instructional text, there are certain phrases that most plausibly fill that spot.We explored various possible architectures, including tree-based methods over GloVe embeddings, ensembled BERT and ELECTRA models, and GPT 2-based infilling methods.
UoR-NCL at SemEval-2022 Task 6: Using ensemble loss with BERT for intended sarcasm detection

Sarcasm has gained notoriety for being difficult to detect by machine learning systems due to its figurative nature.In this paper, Bidirectional Encoder Representations from Transformers (BERT) model has been used with ensemble loss made of cross-entropy loss and negative log-likelihood loss to classify whether a given sentence is in English and Arabic tweets are sarcastic or not.From the results obtained in the experiments, our proposed BERT with ensemble loss achieved superior performance when applied to English and Arabic test datasets.For the validation dataset, our model performed better on the Arabic dataset but failed to outperform the baseline method (made of BERT with only a single loss function) when applied on the English validation set.
PALI at SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

This paper describes our system used in the SemEval-2022 Task 7(Roth et al.): Identifying Plausible Clarifications of Implicit and Underspecified Phrases.Semeval Task7 is an more complex cloze task, different than normal cloze task, only requiring NLP system could find the best fillers for sentence.In Semeval Task7, NLP system not only need to choose the best fillers for each input instance, but also evaluate the quality of all possible fillers and give them a relative score according to context semantic information.We propose an ensemble of different state-of-the-art transformer-based language models(i.e., RoBERTa and Deberta) with some plug-and-play tricks, such as Grouped Layerwise Learning Rate Decay (GLLRD) strategy, contrastive learning loss, different pooling head and an external input data preprecess block before the information came into pretrained language models, which improve performance significantly.The main contributions of our system are 1) revealing the performance discrepancy of different transformer-based pretraining models on the downstream task; 2) presenting an efficient learning-rate and parameter attenuation strategy when fintuning pretrained language models; 3) adding different constrative learning loss to improve model performance; 4) showing the useful of the different pooling head structure.Our system achieves a test accuracy of 0.654 on subtask1(ranking 4th on the leaderboard) and a test Spearman's rank correlation coefficient of 0.785 on subtask2(ranking 2nd on the leaderboard).
Even the Simplest Baseline Needs Careful Re-investigation: A Case Study on XML-CNN

The power and the potential of deep learning models attract many researchers to design advanced and sophisticated architectures.Nevertheless, the progress is sometimes unreal due to various possible reasons.In this work, through an astonishing example we argue that more efforts should be paid to ensure the progress in developing a new deep learning method.For a highly influential multi-label text classification method XML-CNN, we show that the superior performance claimed in the original paper was mainly due to some unbelievable coincidences.We re-examine XML-CNN and make a re-implementation which reveals some contradictory findings to the claims in the original paper.Our study suggests suitable baselines for multi-label text classification tasks and confirms that the progress on a new architecture cannot be confidently justified without a cautious investigation.• Our investigation suggests that instead of XML-CNN, Kim-CNN or a simpler variant of XML-CNN should be considered as a baseline in XMTC tasks.The paper is organized as follows: in Section 2, we introduce Kim-CNN, XML-CNN, and their differences.We conduct an investigation on XML-CNN in Section 3. The investigation includes inspection of the authors' code and our analysis on why it coincidentally works.We then conduct a fair and thorough comparison between Kim-CNN and XML-CNN in Section 4. Finally, we conclude this work in Section 5.
CSECU-DSG at SemEval-2022 Task 3: Investigating the Taxonomic Relationship Between Two Arguments using Fusion of Multilingual Transformer Models

Recognizing lexical relationships between words is one of the formidable tasks in computational linguistics.It plays a vital role in the improvement of various NLP tasks.However, the diversity of word semantics, sentence structure as well as word order information make it challenging to distill the relationship effectively.To address these challenges, SemEval-2022 Task 3 introduced a shared task PreTENS focusing on semantic competence to determine the taxonomic relations between two nominal arguments.This paper presents our participation in this task where we proposed an approach through exploiting an ensemble of multilingual transformer models.We employed two finetuned multilingual transformer models including XLM-RoBERTa and mBERT to train our model.To enhance the performance of individual models, we fuse the predicted probability score of these two models using weighted arithmetic mean to generate a unified probability score.The experimental results showed that our proposed method achieved competitive performance among the participants' methods.
Few-shot Learning for Sumerian Named Entity Recognition

This paper presents our study in exploring the task of named entity recognition (NER) in a low resource setting, focusing on few-shot learning on the Sumerian NER task.The Sumerian language is deemed as an extremely low-resource language due to that (1) it is a long dead language, (2) highly skilled language experts are extremely scarce.NER on Sumerian text is important in that it helps identify the actors and entities active in a given period of time from the collections of tens of thousands of texts in building socio-economic networks of the archives of interest.As a text classification task, NER tends to become challenging when the amount of annotated data is limited or the model is required to handle new classes.The Sumerian NER is no exception.In this work, we propose to use two few-shot learning systems, ProtoBERT and NNShot, to the Sumerian NER task.Our experiments show that the Pro-toBERT NER generally outperforms both the NNShot NER and the fully supervised BERT NER in low resource settings on the predictions of rare classes.In particular, F1-score of Pro-toBERT on unseen entity types on our test set has achieved 89.6% that is significantly better than the F1-score of 84.3% of the BERT NER.
Beyond emojis: an insight into the IKON language

This paper presents a new iconic language, the IKON language, and its philosophical, linguistic, and graphical principles.We examine some case studies to highlight the semantic complexity of the visual representation of meanings.We also introduce the Iconometer test to validate our icons and their application to the medical domain, through the creation of iconic sentences.
niksss at SemEval-2022 Task 6: Are Traditionally Pre-Trained Contextual Embeddings Enough for Detecting Intended Sarcasm ?

This paper presents the 10th and 11th place system for Subtask A -English and Subtask A -Arabic respectively of the SemEval 2022 -Task 6.The purpose of the Subtask A was to classify a given text sequence into sarcastic and nonsarcastic.We also breifly cover our method for Subtask B which performed subpar when compared with most of the submissions on the official leaderboard .All of the developed solutions used a transformers based language model for encoding the text sequences with necessary changes of the pretrained weights and classifier according to the language and subtask at hand .
BL.Research at SemEval-2022 Task 8: Using various Semantic Information to evaluate document-level Semantic Textual Similarity

Sebastien Dufour, Mohamed Mehdi Kandi, Karim Boutamine, Camille Gosse, Mokhtar Boumedyen Billami, Christophe Bortolaso, Youssef Miloudi. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). 2022.
Web-based Annotation Interface for Derivational Morphology

The paper presents a visual interface for manual annotation of language resources for derivational morphology.The interface is webbased and created using relatively simple programming techniques, and yet it rapidly facilitates and speeds up the annotation process, especially in languages with rich derivational morphology.As such, it can reduce the cost of the process.After introducing manual annotation tasks in derivational morphology, the paper describes the new visual interface and a case study that compares the current annotation method to the annotation using the interface.In addition, it also demonstrates the opportunity to use the interface for manual annotation of syntactic trees.
Targeted Identity Group Prediction in Hate Speech Corpora

The past decade has seen an abundance of work seeking to detect, characterize, and measure online hate speech.A related, but less studied problem, is the specification of identity groups targeted by that hate speech.Predictive accuracy on this task can supplement additional analyses beyond hate speech detection, motivating its study.Using the Measuring Hate Speech corpus, which provided annotations for targeted identity groups on roughly 50,000 social media comments, we create neural network models to perform multi-label binary prediction of identity groups targeted by a social media comment.Specifically, we study 8 broad identity groups and 12 identity sub-groups within race and gender identity.We find that these networks exhibited good predictive performance, achieving ROC AUCs of greater than 0.9 and PR AUCs of greater than 0.7 on several identity groups.At the same time, we find performance suffered on identity groups less represented in the dataset.We validate model performance on the HateCheck and Gab Hate Corpora, finding that predictive performance generalizes in most settings.We additionally examine the performance of the model on comments targeting multiple identity groups.Lastly, we discuss issues with a standardized conceptualization of a "target" in hate speech corpora, and its relation to intersectionality.Our results demonstrate the feasibility of simultaneously detecting a broad range of targeted groups in social media comments, and offer suggestions for future work on modeling and dataset annotation for this task.
Explicit Graph Reasoning Fusing Knowledge and Contextual Information for Multi-hop Question Answering

Current graph-neural-network-based (GNNbased) approaches to multi-hop questions integrate clues from scattered paragraphs in an entity graph, achieving implicit reasoning by synchronous update of graph node representations using information from neighbours; this is poorly suited for explaining how clues are passed through the graph in hops.In this paper, we describe a structured Knowledge and contextual Information Fusion GNN (KIFGraph) whose explicit multi-hop graph reasoning mimics human step by step reasoning.Specifically, we first integrate clues at multiple levels of granularity (question, paragraph, sentence, entity) as nodes in the graph, connected by edges derived using structured semantic knowledge, then use a contextual encoder to obtain the initial node representations, followed by stepby-step two-stage graph reasoning that asynchronously updates node representations.Each node can be related to its neighbour nodes through fused structured knowledge and contextual information, reliably integrating their answer clues.Moreover, a masked attention mechanism (MAM) filters out noisy or redundant nodes and edges, to avoid ineffective clue propagation in graph reasoning.Experimental results show performance competitive with published models on the HotpotQA dataset.
A Sequence Modelling Approach to Question Answering in Text-Based Games

Interactive Question Answering (IQA) requires an intelligent agent to interact with a dynamic environment in order to gather information necessary to answer a question.IQA tasks have been proposed as means of training systems to develop language or visual comprehension abilities.To this end, the Question Answering with Interactive Text (QAit) task was created to produce and benchmark interactive agents capable of seeking information and answering questions in unseen environments.While prior work has exclusively focused on IQA as a reinforcement learning problem, such methods suffer from low sample efficiency and poor accuracy in zero-shot evaluation.In this paper, we propose the use of the recently proposed Decision Transformer architecture to provide improvements upon prior baselines.By utilising a causally masked GPT-2 Transformer for command generation and a BERT model for question answer prediction, we show that the Decision Transformer achieves performance greater than or equal to current state-of-the-art RL baselines on the QAit task in a sample efficient manner.In addition, these results are achievable by training on sub-optimal random trajectories, therefore not requiring the use of online agents to gather data.
Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification

Many linguistic expressions have idiomatic and literal interpretations, and the automatic distinction of these two interpretations has been studied for decades.Recent research has shown that contextualized word embeddings derived from masked language models (MLMs) can give promising results for idiom token classification.This indicates that contextualized word embedding alone contains information about whether the word is being used in a literal sense or not.However, we believe that more types of information can be derived from MLMs and that leveraging such information can improve idiom token classification.In this paper, we leverage three types of embeddings from MLMs; uncontextualized token embeddings and masked token embeddings in addition to the standard contextualized word embeddings and show that the newly added embeddings significantly improve idiom token classification for both English and Japanese datasets.
Short Text Classification via Term Graph

Short text classi cation is a method for classifying short sentence with prede ned labels. However, short text is limited in shortness in text length that leads to a challenging problem of sparse features. Most of existing methods treat each short sentences as independently and identically distributed (IID), local context only in the sentence itself is focused and the relational information between sentences are lost. To overcome these limitations, we propose a PathWalk model that combine the strength of graph networks and short sentences to solve the sparseness of short text. Experimental results on four different available datasets show that our PathWalk method achieves the state-of-the-art results, demonstrating the efficiency and robustness of graph networks for short text classification.
Mubody, an astrodynamics open-source Python library focused on libration points

Mubody is an astrodynamics open-source Python library focused on the libration points. Such points result from the equilibrium of the gravitational forces between two massive bodies as the Sun and Earth, for example. The library is mainly intended for the generation of orbits in these regions, which is not a straightforward process, specially if perturbations are considered. Currently, the library allows to generate Lissajous orbits in the second Lagrange point of the Sun-Earth system under the influence of perturbations such as the Earth orbit eccentricity. The next milestone, as a result of a master student work, is the incorporation of Halo orbits and the expansion to all three collinear libration points from any two massive bodies of the Solar System. This tool has been developed as part of a PhD, motivated by the need of performing mission analysis in libration point regions. Nevertheless, since its creation it has also proven to be an excellent academic tool for both enhancing the library itself and using its results for further studies (collision risk, thermal analysis, formation flight control, etc). As a result, the tool has rapidly evolved, building onto the knowledge and experience that the students gather while working on their academic projects (bachelor’s degree dissertations, master theses, subjects, internships). The participation on the library development provides students with experience in orbital mechanics, software design, version control and it compels them to ensure that their work can be readily used by others as it is properly documented. The project is hosted in GitLab under a MIT licence