Reviewerly/exHarmony huggingface dataset:
Aggregating Multiple Heuristic Signals as Supervision for Unsupervised Automated Essay Scoring

Automated Essay Scoring (AES) aims to evaluate the quality score for input essays.In this work, we propose a novel unsupervised AES approach ULRA, which does not require groundtruth scores of essays for training.The core idea of our ULRA is to use multiple heuristic quality signals as the pseudo-groundtruth, and then train a neural AES model by learning from the aggregation of these quality signals.To aggregate these inconsistent quality signals into a unified supervision, we view the AES task as a ranking problem, and design a special Deep Pairwise Rank Aggregation (DPRA) loss for training.In the DPRA loss, we set a learnable confidence weight for each signal to address the conflicts among signals, and train the neural AES model in a pairwise way to disentangle the cascade effect among partialorder pairs.Experiments on eight prompts of ASPA dataset show that ULRA achieves the state-of-the-art performance compared with previous unsupervised methods in terms of both transductive and inductive settings.Further, our approach achieves comparable performance with many existing domain-adapted supervised models, showing the effectiveness of ULRA.The code is available at https: //github.com/tenvence/ulra.
niceNLP at SemEval-2023 Task 10: Dual Model Alternate Pseudo-labeling Improves Your Predictions

Sexism is a growing online problem.It harms women who are targeted and makes online spaces inaccessible and unwelcoming.In this paper, we present our approach for Task A of SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS), which aims to perform binary sexism detection on textual content.To solve this task, we fine-tune the pre-trained model based on several popular natural language processing methods to improve the generalization ability in the face of different data.According to the experimental results, the effective combination of multiple methods enables our approach to achieve excellent performance gains.
Reasoning Circuits: Few-shot Multi-hop Question Generation with Structured Rationales

Multi-hop Question Generation is the task of generating questions which require the reader to reason over and combine information spread across multiple passages employing several reasoning steps. Chain-of-thought rationale generation has been shown to improve performance on multi-step reasoning tasks and make model predictions more interpretable. However, few-shot performance gains from including rationales have been largely observed only in +100B language models, and otherwise require large-scale manual rationale annotation. In this paper, we introduce a new framework for applying chain-of-thought inspired structured rationale generation to multi-hop question generation under a very low supervision regime (8- to 128-shot). We propose to annotate a small number of examples following our proposed multi-step rationale schema, treating each reasoning step as a separate task to be performed by a generative language model. We show that our framework leads to improved control over the difficulty of the generated questions and better performance compared to baselines trained without rationales, both on automatic evaluation metrics and in human evaluation. Importantly, we show that this is achievable with a modest model size.
Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion

This paper describes our zero-shot approaches for the Visual Word Sense Disambiguation (VWSD) Task in English.Our preliminary study shows that the simple approach of matching candidate images with the phrase using CLIP suffers from the many-to-many nature of image-text pairs.We find that the CLIP text encoder may have limited abilities in capturing the compositionality in natural language.Conversely, the descriptive focus of the phrase varies from instance to instance.We address these issues in our two systems, Augment-CLIP and Stable Diffusion Sampling (SD Sampling).Augment-CLIP augments the text prompt by generating sentences that contain the context phrase with the help of large language models (LLMs).We further explore CLIP models in other languages, as the an ambiguous word may be translated into an unambiguous one in the other language.SD Sampling uses textto-image Stable Diffusion to generate multiple images from the given phrase, increasing the likelihood that a subset of images match the one that paired with the text.
Contrastive Loss is All You Need to Recover Analogies as Parallel Lines

While static word embedding models are known to represent linguistic analogies as parallel lines in high-dimensional space, the underlying mechanism as to why they result in such geometric structures remains obscure.We find that an elementary contrastive-style optimization employed over distributional information performs competitively with popular word embedding models on analogy recovery tasks, while achieving dramatic speedups in training time.Further, we demonstrate that a contrastive loss is sufficient to create these parallel structures in word embeddings, and establish a precise relationship between the cooccurrence statistics and the geometric structure of the resulting word embeddings.
Privacy-Preserving Knowledge Transfer through Partial Parameter Sharing

Valuable datasets that contain sensitive information are not shared due to privacy and copyright concerns.This hinders progress in many areas and prevents the use of machine learning solutions to solve relevant tasks.One possible solution is sharing models that are trained on such datasets.However, this is also associated with potential privacy risks due to data extraction attacks.In this work, we propose a solution based on sharing parts of the model's parameters, and using a proxy dataset for complimentary knowledge transfer.Our experiments show encouraging results, and reduced risk to potential training data identification attacks.We present a viable solution to sharing knowledge with data-disadvantaged parties, that do not have the resources to produce high-quality data, with reduced privacy risks to the sharing parties.We make our code publicly available. 1
HULAT at SemEval-2023 Task 9: Data Augmentation for Pre-trained Transformers Applied to Multilingual Tweet Intimacy Analysis

This paper describes our participation in SemEval-2023 Task 9, Intimacy Analysis of Multilingual Tweets.We fine-tune some of the most popular transformer models with the training dataset and synthetic data generated by different data augmentation techniques.During the development phase, our best results were obtained by using XLM-T.Data augmentation techniques provide a very slight improvement in the results.Our system ranked in the 27th position out of the 45 participating systems.Despite its modest results, our system shows promising results in languages such as Portuguese, English, and Dutch.All our code is available in the repository https: //github.com/isegura/hulat_intimacy.
MarsEclipse at SemEval-2023 Task 3: Multi-lingual and Multi-label Framing Detection with Contrastive Learning

This paper describes our system for SemEval-2023 Task 3 Subtask 2 on Framing Detection.We used a multi-label contrastive loss for finetuning large pre-trained language models in a multi-lingual setting, achieving very competitive results: our system was ranked first on the official test set and on the official shared task leaderboard for five of the six languages for which we had training data and for which we could perform fine-tuning.Here, we describe our experimental setup, as well as various ablation studies.The code of our system is available at https://github.com/QishengL/SemEval2023
Character Coreference Resolution in Movie Screenplays

Movie screenplays have a distinct narrative structure.It segments the story into scenes containing interleaving descriptions of actions, locations, and character dialogues.A typical screenplay spans several scenes and can include long-range dependencies between characters and events.A holistic document-level understanding of the screenplay requires several natural language processing capabilities, such as parsing, character identification, coreference resolution, action recognition, summarization, and attribute discovery.In this work, we develop scalable and robust methods to extract the structural information and character coreference clusters from full-length movie screenplays.We curate two datasets for screenplay parsing and character coreference -MovieParse and MovieCoref, respectively.We build a robust screenplay parser to handle inconsistencies in screenplay formatting and leverage the parsed output to link co-referring character mentions.Our coreference models can scale to long screenplay documents without drastically increasing their memory footprints.We open-source our model and datasets here. 1
RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations

Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to taskspecific perturbations, e.g., replacing key question entities or shuffling table columns.To systematically study the robustness of Table QA models, we propose a benchmark called ROBUT, which builds upon existing Table QA datasets (WTQ, WIKISQL-WEAK, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question.Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets.We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models.
Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT

Well pre-trained contextualized representations from pre-trained language models (PLM) have been shown helpful for enhancing various natural language processing tasks, surely including neural machine translation (NMT).However, existing methods either consider encoder-only enhancement or rely on specific multilingual PLMs, which leads to a much larger model or give up potentially helpful knowledge from target PLMs.In this paper, we propose a new monolingual PLM-sponsored NMT model to let both encoder and decoder enjoy PLM enhancement to alleviate such obvious inconvenience.Especially, incorporating a newly proposed frequency-weighted embedding transformation algorithm, PLM embeddings can be effectively exploited in terms of the representations of the NMT decoder.We evaluate our model on IWSLT14 En-De, De-En, WMT14 En-De, and En-Fr tasks, and the results show that our proposed PLM enhancement gives significant improvement and even helps achieve new state-of-the-art.
Arizonans at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis with XLM-T

This paper presents the systems and approaches of the Arizonans team for SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis.We finetune the Multilingual RoBERTa model trained with about 200M tweets, XLM-T.Our final model ranked 9 th out of 45 overall, 13 th in seen languages, and 8 th in unseen languages.
Synthetic Pre-Training Tasks for Neural Machine Translation

Pre-training models with large crawled corpora can lead to issues such as toxicity and bias, as well as copyright and privacy concerns.A promising way of alleviating such concerns is to conduct pre-training with synthetic tasks and data, since no real-world information is ingested by the model.Our goal in this paper is to understand the factors that contribute to the effectiveness of pre-training models when using synthetic resources, particularly in the context of neural machine translation.We propose several novel approaches to pre-training translation models that involve different levels of lexical and structural knowledge, including: 1) generating obfuscated data from a large parallel corpus 2) concatenating phrase pairs extracted from a small word-aligned corpus, and 3) generating synthetic parallel data without real human language corpora.Our experiments on multiple language pairs reveal that pre-training benefits can be realized even with high levels of obfuscation or purely synthetic parallel data.We hope the findings from our comprehensive empirical analysis will shed light on understanding what matters for NMT pre-training, as well as pave the way for the development of more efficient and less toxic models.
CLaC at SemEval-2023 Task 2: Comparing Span-Prediction and Sequence-Labeling Approaches for NER

This paper summarizes the CLaC submission for the MultiCoNER 2 task which concerns the recognition of complex, fine-grained named entities.We compare two popular approaches for NER, namely Sequence Labeling and Span Prediction.We find that our best Span Prediction system performs slightly better than our best Sequence Labeling system on test data.Moreover, we find that using the larger version of XLM RoBERTa significantly improves performance.Post-competition experiments show that Span Prediction and Sequence Labeling approaches improve when they use special input tokens (
 and 
) of XLM-RoBERTa.The code for training all models, preprocessing, and post-processing is available at this Github repo.
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing stateof-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications.We show that going beyond Englishcentric bitexts, coupled with a novel sampling strategy aimed at reducing under-utilization of training data, substantially boosts performance across model sizes for both Electra and MLM pre-training objectives.We introduce XY-LENT: X-Y bitext enhanced Language ENcodings using Transformers which not only achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands, is also competitive across bands.Our XY-LENT XL variant outperforms XLM-R XXL and exhibits competitive performance with mT5 XXL while being 5x and 6x smaller respectively.We then show that our proposed method helps ameliorate the curse of multilinguality, with the XY-LENT XL achieving 99.3% GLUE performance and 98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same size band.We then analyze our models performance on extremely low resource languages and posit that scaling alone may not be sufficient for improving the performance in this scenario.
UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysis

This paper describes our system designed for SemEval-2023 Task 12: Sentiment analysis for African languages.The challenge faced by this task is the scarcity of labeled data and linguistic resources in low-resource settings.To alleviate these, we propose a generalized multilingual system SACL-XLMR for sentiment analysis on low-resource languages.Specifically, we design a lexicon-based multilingual BERT to facilitate language adaptation and sentimentaware representation learning.Besides, we apply a supervised adversarial contrastive learning technique to learn sentiment-spread structured representations and enhance model generalization.Our system achieved competitive results, largely outperforming baselines on both multilingual and zero-shot sentiment classification subtasks.Notably, the system obtained the 1st rank on the zero-shot classification subtask in the official ranking.Extensive experiments demonstrate the effectiveness of our system.
lilGym: Natural Language Visual Reasoning with Reinforcement Learning

We present lilGym, a new benchmark for language-conditioned reinforcement learning in visual environments.lilGym is based on 2,661 highly-compositional human-written natural language statements grounded in an interactive visual environment.We introduce a new approach for exact reward computation in every possible world state by annotating all statements with executable Python programs.Each statement is paired with multiple start states and reward functions to form thousands of distinct Markov Decision Processes of varying difficulty.We experiment with lilGym with different models and learning regimes.Our results and analysis show that while existing methods are able to achieve non-trivial performance, lilGym forms a challenging open problem.
Augustine of Hippo at SemEval-2023 Task 4: An Explainable Knowledge Extraction Method to Identify Human Values in Arguments with SuperASKE

In this paper we present and discuss the results achieved by the "Augustine of Hippo" team at SemEval-2023 Task 4 about human value detection.In particular, we provide a quantitative and qualitative reviews of the results obtained by SuperASKE, discussing respectively performance metrics and classification errors.Finally, we present our main contribution: an explainable and unsupervised approach mapping arguments to concepts, followed by a supervised classification model mapping concepts to human values.
UL &amp; UM6P at SemEval-2023 Task 10: Semi-Supervised Multi-task Learning for Explainable Detection of Online Sexism

This paper introduces our participating system to the Explainable Detection of Online Sexism (EDOS) SemEval-2023 -Task 10: Explainable Detection of Online Sexism.The EDOS shared task covers three hierarchical sub-tasks for sexism detection, coarse-grained and finegrained categorization.We have investigated both single-task and multi-task learning based on RoBERTa transformer-based language models.For improving the results, we have performed further pre-training of RoBERTa on the provided unlabeled data.Besides, we have employed a small sample of the unlabeled data for semi-supervised learning using the minimum class-confusion loss.Our system has achieved macro F1 scores of 82.25%, 67.35%, and 49.8% on Tasks A, B, and C, respectively.
PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation

Syntactically controlled paraphrase generation requires language models to generate paraphrases for sentences according to specific syntactic structures.Existing fine-tuning methods for this task are costly as all the parameters of the model need to be updated during the training process.Inspired by recent studies on parameter-efficient learning, we propose Parse-Instructed Prefix (PIP), a novel adaptation of prefix-tuning to tune large pre-trained language models on syntactically controlled paraphrase generation task in a low-data setting with significantly less training cost.We introduce two methods to instruct a model's encoder prefix to capture syntax-related knowledge: direct initiation (PIP-Direct) and indirect optimization (PIP-Indirect).In contrast to traditional finetuning methods for this task, PIP is a computeefficient alternative with 10× times less learnable parameters.Compared to existing prefixtuning methods, PIP excels at capturing syntax control information, achieving significantly higher performance at the same level of learnable parameter count.
SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition

Distantly-Supervised Named Entity Recognition effectively alleviates the burden of timeconsuming and expensive annotation in the supervised setting.But the context-free matching process and the limited coverage of knowledge bases introduce inaccurate and incomplete annotation noise respectively.Previous studies either considered only incomplete annotation noise or indiscriminately handle two types of noise with the same strategy.In this paper, we argue that the different causes of two types of noise bring up the requirement of different strategies in model architecture.Therefore, we propose the SANTA to handle these two types of noise separately with (1) Memory-smoothed Focal Loss and Entity-aware KNN to relieve the entity ambiguity problem caused by inaccurate annotation, and (2) Boundary Mixup to alleviate decision boundary shifting problem caused by incomplete annotation and a noisetolerant loss to improve the robustness.Benefiting from our separate tailored strategies, we confirm in the experiment that the two types of noise are well mitigated.SANTA also achieves a new state-of-the-art on five public datasets.
Joint Constrained Learning with Boundary-adjusting for Emotion-Cause Pair Extraction

Emotion-Cause Pair Extraction (ECPE) aims to identify the document's emotion clauses and corresponding cause clauses.Like other relation extraction tasks, ECPE is closely associated with the relationship between sentences.Recent methods based on Graph Convolutional Networks focus on how to model the multiplex relations between clauses by constructing different edges.However, the data of emotions, causes, and pairs are extremely unbalanced, but current methods get their representation using the same graph structure.In this paper, we propose a Joint Constrained Learning framework with Boundary-adjusting for Emotion-Cause Pair Extraction (JCB).Specifically, through constrained learning, we summarize the prior rules existing in the data and force the model to take them into consideration in optimization, which helps the model learn a better representation from unbalanced data.Furthermore, we adjust the decision boundary of classifiers according to the relations between subtasks, which have always been ignored.No longer working independently as in the previous framework, the classifiers corresponding to three subtasks cooperate under the relation constraints.Experimental results show that JCB obtains competitive results compared with state-of-theart methods and prove its robustness on unbalanced data.
Dynamic Regularization in UDA for Transformers in Multimodal Classification

Multimodal machine learning is a cutting-edge field that explores ways to incorporate information from multiple sources into models.As more multimodal data becomes available, this field has become increasingly relevant.This work focuses on two key challenges in multimodal machine learning.The first is finding efficient ways to combine information from different data types.The second is that often, one modality (e.g., text) is stronger and more relevant, making it difficult to identify meaningful patterns in the weaker modality (e.g., image).Our approach focuses on more effectively exploiting the weaker modality while dynamically regularizing the loss function.First, we introduce a new two-stream model called Multimodal BERT-ViT, which features a novel intra-CLS token fusion.Second, we utilize a dynamic adjustment that maintains a balance between specialization and generalization during the training to avoid overfitting, which we devised.We add this dynamic adjustment to the Unsupervised Data Augmentation (UDA) framework.We evaluate the effectiveness of these proposals on the task of multi-label movie genre classification using the Moviescope and MM-IMDb datasets.The evaluation revealed that our proposal offers substantial benefits, while simultaneously enabling us to harness the weaker modality without compromising the information provided by the stronger.