Reviewerly/exHarmony huggingface dataset:
Unsupervised Graph-Text Mutual Conversion with a Unified Pretrained Language Model

Graph-to-text (G2T) generation and text-tograph (T2G) triple extraction are two essential tasks for knowledge graphs.Existing unsupervised approaches become suitable candidates for jointly learning the two tasks due to their avoidance of using graph-text parallel data.However, they adopt multiple complex modules and still require entity information or relation type for training.To this end, we propose INFINITY, a simple yet effective unsupervised method with a unified pretrained language model that does not introduce external annotation tools or additional parallel information.It achieves fully unsupervised graph-text mutual conversion for the first time.Specifically, INFINITY treats both G2T and T2G as a bidirectional sequence generation task by finetuning only one pretrained seq2seq model.A novel back-translation-based framework is then designed to generate synthetic parallel data automatically.Besides, we investigate the impact of graph linearization and introduce the structure-aware fine-tuning strategy to alleviate possible performance deterioration via retaining structural information in graph sequences.As a fully unsupervised framework, INFINITY is empirically verified to outperform state-ofthe-art baselines for G2T and T2G tasks.Additionally, we also devise a new training setting called cross learning for low-resource unsupervised information extraction.
JUST-KM at SemEval-2023 Task 7: Multi-evidence Natural Language Inference using Role-based Double Roberta-Large

In recent years, there has been a vast increase in the available clinical data. Variant Deep learning techniques are used to enhance the retrieval and interpretation of these data. This task deployed Natural language inference (NLI) in Clinical Trial Reports (CTRs) to provide individualized care that is supported by evidence. A collection of breast cancer clinical trial records, statements, annotations, and labels from experienced domain experts. NLI presents a chance to advance the widespread understanding and retrieval of medical evidence, leading to significant improvements in connecting the most recent evidence to personalized care. The primary objective is to identify the inference relationship (entailment or contradiction) between pairs of clinical trial records and statements. In this research, we used different transformer-based models, and The proposed model, “Role-based Double Roberta-Large,” achieved the best result on the testing dataset with F1-score equal to 67.0%
SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives

Dual encoders have been used for retrieval tasks and representation learning with good results.A standard way to train dual encoders is using a contrastive loss with in-batch negatives.In this work, we propose an improved contrastive learning objective by adding queries or documents from the same encoder towers to the negatives, for which we name it as "contrastive loss with SAMe TOwer NEgatives" (SamToNe).By evaluating on question answering retrieval benchmarks from MS MARCO and MultiReQA, and heterogenous zero-shot information retrieval benchmarks (BEIR), we demonstrate that SamToNe can effectively improve the retrieval quality for both symmetric and asymmetric dual encoders.By directly probing the embedding spaces of the two encoding towers via the t-SNE algorithm (van der Maaten and Hinton, 2008), we observe that SamToNe ensures the alignment between the embedding spaces from the two encoder towers.Based on the analysis of the embedding distance distributions of the top-1 retrieved results, we further explain the efficacy of the method from the perspective of regularisation.
Another Dead End for Morphological Tags? Perturbed Inputs and Parsing

The usefulness of part-of-speech tags for parsing has been heavily questioned due to the success of word-contextualized parsers.Yet, most studies are limited to coarse-grained tags and high quality written content; while we know little about their influence when it comes to models in production that face lexical errors.We expand these setups and design an adversarial attack to verify if the use of morphological information by parsers: (i) contributes to error propagation or (ii) if on the other hand it can play a role to correct mistakes that word-only neural parsers make.The results on 14 diverse UD treebanks show that under such attacks, for transition-and graph-based models their use contributes to degrade the performance even faster, while for the (lower-performing) sequence labeling parsers they are helpful.We also show that if morphological tags were utopically robust against lexical perturbations, they would be able to correct parsing mistakes.
Modelling the Reduplicating Lushootseed Morphology with an FST and LSTM

In this paper, we present an FST based approach for conducting morphological analysis, lemmatization and generation of Lushootseed words.Furthermore, we use the FST to generate training data for an LSTM based neural model and train this model to do morphological analysis.The neural model reaches a 71.9% accuracy on the test data.Furthermore, we discuss reduplication types in the Lushootseed language forms.The approach involves the use of both attested instances of reduplication and bare stems for applying a variety of reduplications to, as it is unclear just how much variation can be attributed to the individual speakers and authors of the source materials.That is, there may be areal factors that can be aligned with certain types of reduplication and their frequencies.
Listener Model for the PhotoBook Referential Game with CLIPScores as Implicit Reference Chain

PhotoBook is a collaborative dialogue game where two players receive private, partiallyoverlapping sets of images and resolve which images they have in common.It presents machines with a great challenge to learn how people build common ground around multimodal context to communicate effectively.Methods developed in the literature, however, cannot be deployed to real gameplay since they only tackle some subtasks of the game, and they require additional reference chains inputs, whose extraction process is imperfect.Therefore, we propose a reference chain-free listener model that directly addresses the game's predictive task, i.e., deciding whether an image is shared with partner.Our DeBERTa-based listener model reads the full dialogue, and utilizes CLIPScore features to assess utterance-image relevance.We achieve >77% accuracy on unseen sets of images/game themes, outperforming baseline by >17 points.
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control

Pretrained language models have demonstrated extraordinary capabilities in language generation.However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization.Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.However, many important distributions, such as personal preferences, are unquantified.In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing NANO, a fewshot human-in-the-loop training algorithm that continuously learns from human feedback.NANO achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works.We also show that NANO is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals' personal preferences with high sample efficiency.Our code is available at https: //github.com/sfanxiang/Nano. but also improves performance on quantified distributions.Our contribution is summarized as follows:• We introduce a human-in-the-loop reward learning algorithm that learns to generate text following arbitrary distribution through human feedback.We demonstrate that our method works for all of the following types of distributions: single-topic/attribute, quantified distributions, and unquantified distributions.• We show that NANO is able to learn unquantified distributions, successfully achieves personalization, and captures differences between different individuals' personal preferences with only 64 labels from each person (RQ1).• We achieve state-of-the-art result on controlling quantified distributions (RQ2) as well as single topic/attribute generation (RQ3) compared to previous works, while using only few-shot samples.• Through ablation studies, we demonstrate the necessity of multi-iteration human feedback for high sample efficiency (RQ4) and justify our architecture's design choices (RQ5).We also show that our method extends to newer and larger language models than GPT-2.An illustration of our method is shown in Figure 1, and a comparison of NANO's capabilities to previous works is provided in Table 2.
Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports

With the increasing number of clinical trial reports generated every day, it is becoming hard to keep up with novel discoveries that inform evidence-based healthcare recommendations.To help automate this process and assist medical experts, NLP solutions are being developed.This motivated the SemEval-2023 Task 7, where the goal was to develop an NLP system for two tasks: evidence retrieval and natural language inference from clinical trial data.In this paper, we describe our two developed systems.The first one is a pipeline system that models the two tasks separately, while the second one is a joint system that learns the two tasks simultaneously with a shared representation and a multi-task learning approach.The final system combines their outputs in an ensemble system.We formalize the models, present their characteristics and challenges, and provide an analysis of achieved results.Our system ranked 3rd out of 40 participants with a final submission.
RIGA at SemEval-2023 Task 2: NER Enhanced with GPT-3

The following is a description of the RIGA team's submissions for the English track of the SemEval-2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER) II.Our approach achieves 17% boost in results by utilizing pre-existing Large-scale Language Models (LLMs), such as GPT-3, to gather additional contexts.We then fine-tune a pre-trained neural network utilizing these contexts.The final step of our approach involves meticulous model and compute resource scaling, which results in improved performance.Our results placed us 12th out of 34 teams in terms of overall ranking and 7th in terms of the noisy subset ranking.The code for our method is available on GitHub 1 .
Multi-doc Hybrid Summarization via Salient Representation Learning

Multi-document summarization is gaining more and more attention recently and serves as an invaluable tool to obtain key facts among a large information pool. In this paper, we proposed a multi-document hybrid summarization approach, which simultaneously generates a human-readable summary and extracts corresponding key evidences based on multi-doc inputs. To fulfill that purpose, we crafted a salient representation learning method to induce latent salient features, which are effective for joint evidence extraction and summary generation. In order to train this model, we conducted multi-task learning to optimize a composited loss, constructed over extractive and abstractive sub-components in a hierarchical way. We implemented the system based on a ubiquiotously adopted transformer architecture and conducted experimental studies on multiple datasets across two domains, achieving superior performance over the baselines.
Domain Generalization via Switch Knowledge Distillation for Robust Review Representation

Applying neural models injected with indomain user and product information to learn review representations of unseen or anonymous users incurs an obvious obstacle in contentbased recommender systems.For the generalization of the in-domain classifier, most existing models train an extra plain-text model for the unseen domain.Without incorporating historical user and product information, such a schema makes unseen and anonymous users dissociate from the recommender system.To simultaneously learn the review representation of both existing and unseen users, this study proposed a switch knowledge distillation for domain generalization.A generalization-switch (GSwitch) model was initially applied to inject user and product information by flexibly encoding both domain-invariant and domainspecific features.By turning the status ON or OFF, the model introduced a switch knowledge distillation to learn a robust review representation that performed well for either existing or anonymous unseen users.The empirical experiments were conducted on IMDB, Yelp-2013, and Yelp-2014 by masking out users in test data as unseen and anonymous users.The comparative results indicate that the proposed method enhances the generalization capability of several existing baseline models.
Training Trajectories of Language Models Across Scales

Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger.How do language models of different sizes learn during pre-training?Why do larger language models demonstrate more desirable behaviors?In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al., 2022)-from 125M to 175B parameters-on next-token prediction, sequence-level generation and downstream tasks.We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior (Nakkiran et al., 2020); 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; and 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independently of the model size.Together, these results show that perplexity is more predictive of model behaviors than model size or training computation. 1
Multi-VALUE: A Framework for Cross-Dialectal English NLP

Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users.Inclusive and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts.Current systems often fall short of this ideal since they are designed and tested on a single dialect: Standard American English (SAE).We introduce a suite of resources for evaluating and achieving English dialect invariance.The resource is called Multi-VALUE, a controllable rule-based translation system spanning 50 English dialects and 189 unique linguistic features.Multi-VALUE maps SAE to synthetic forms of each dialect.First, we use this system to stress tests question answering, machine translation, and semantic parsing.Stress tests reveal significant performance disparities for leading models on nonstandard dialects.Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems.Finally, we partner with native speakers of Chicano and Indian English to release new goldstandard variants of the popular CoQA task.To execute the transformation code, run model checkpoints, and download both synthetic and gold-standard dialectal benchmark datasets, see
Theoretical Linguistics Rivals Embeddings in Language Clustering for Multilingual Named Entity Recognition

While embedding-based methods have been dominant in language clustering for multilingual tasks, clustering based on linguistic features has not yet been explored much, as it remains baselines (Tan et al., 2019;Shaffer, 2021).This study investigates whether and how theoretical linguistics improves language clustering for multilingual named entity recognition (NER).We propose two types of language groupings: one based on morpho-syntactic features in a nominal domain and one based on a head parameter.Our NER experiments show that the proposed methods largely outperform a state-of-the-art embedding-based model, suggesting that theoretical linguistics plays a significant role in multilingual learning tasks.
LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning

In recent years, there has been significant progress in developing pre-trained language models for NLP.However, these models often struggle when fine-tuned on small datasets.To address this issue, researchers have proposed various adaptation approaches.Promptbased tuning is arguably the most common way, especially for larger models.Previous research shows that adding contrastive learning to prompt-based fine-tuning is effective as it helps the model generate embeddings that are more distinguishable between classes, and it can also be more sample-efficient as the model learns from positive and negative examples simultaneously.One of the most important components of contrastive learning is data augmentation, but unlike computer vision, effective data augmentation for NLP is still challenging.This paper proposes LM-CPPF, Contrastive Paraphrasing-guided Prompt-based Fine-tuning of Language Models, which leverages promptbased few-shot paraphrasing using generative language models, especially large language models such as GPT-3 and OPT-175B, for data augmentation.Our experiments on multiple text classification benchmarks show that this augmentation method outperforms other methods, such as easy data augmentation, back translation, and multiple templates. 1
Semi-Supervised Domain Adaptation for Emotion-Related Tasks

Semi-supervised domain adaptation (SSDA) adopts a model trained from a label-rich source domain to a new but related domain with a few labels of target data.It is shown that, in an SSDA setting, a simple combination of domain adaptation (DA) with semi-supervised learning (SSL) techniques often fails to effectively utilize the target supervision and cannot address distribution shifts across different domains due to the training data bias toward the source-labeled samples.In this paper, inspired by the co-learning of multiple classifiers for the computer vision tasks, we propose to decompose the SSDA framework for emotion-related tasks into two subcomponents of unsupervised domain adaptation (UDA) from the source to the target domain and semi-supervised learning (SSL) in the target domain where the two models iteratively teach each other by interchanging their high confident predictions.We further propose a novel data cartography-based regularization technique for pseudo-label denoising that employs training dynamics to further hone our models' performance.We release our code. 1
Entropy-guided Vocabulary Augmentation of Multilingual Language Models for Low-resource Tasks

Multilingual language models (MLLMs) like mBERT promise to extend the benefits of NLP research to low-resource languages (LRLs).However, LRL words are under-represented in the wordpiece/subword vocabularies of MLLMs.This leads to many LRL words getting replaced by UNK, or concatenated from morphologically unrelated wordpieces, leading to low task accuracy.(Pre)-training MLLMs after including LRL documents is resource-intensive in terms of both human inputs and computational resources.In response, we propose EVALM (entropy-based vocabulary augmented language model), which uses a new task-cognizant measurement to detect the most vulnerable LRL words, whose wordpiece segmentations are undesirable.EVALM then provides reasonable initializations of their embeddings, followed by limited fine-tuning using the small LRL task corpus.Our experiments show significant performance improvements and also some surprising limits to such vocabulary augmentation strategies in various classification tasks for multiple diverse LRLs, as well as code-mixed texts.We will release the code and data to enable further research 1 .
Learning with Partial Annotations for Event Detection

Event detection (ED) seeks to discover and classify event instances in plain texts.Previous methods for ED typically adopt supervised learning, requiring fully labeled and high-quality training data.However, in a realworld application, we may not obtain clean training data but only partially labeled one, which could substantially impede the learning process.In this work, we conduct a seminal study for learning with partial annotations for ED.We propose a new trigger localization formulation using contrastive learning to distinguish ground-truth triggers from contexts, showing a decent robustness for addressing partial annotation noise.Impressively, in an extreme scenario where more than 90% of events are unlabeled, our approach achieves an F1 score of over 60%.In addition, we reannotate and make available two fully annotated subsets of ACE 2005 to serve as an unbiased benchmark for event detection.We hope our approach and data will inspire future studies on this vital yet understudied problem.
Pretrained Bidirectional Distillation for Machine Translation

Knowledge transfer can boost neural machine translation (NMT), for example, by finetuning a pretrained masked language model (LM).However, it may suffer from the forgetting problem and the structural inconsistency between pretrained LMs and NMT models.Knowledge distillation (KD) may be a potential solution to alleviate these issues, but few studies have investigated language knowledge transfer from pretrained language models to NMT models through KD.In this paper, we propose Pretrained Bidirectional Distillation (PBD) for NMT, which aims to efficiently transfer bidirectional language knowledge from masked language pretraining to NMT models.Its advantages are reflected in efficiency and effectiveness through a globally defined and bidirectional context-aware distillation objective.Bidirectional language knowledge of the entire sequence is transferred to an NMT model concurrently during translation training.Specifically, we propose self-distilled masked language pretraining to obtain the PBD objective.We also design PBD losses to efficiently distill the language knowledge, in the form of token probabilities, to the encoder and decoder of an NMT model using the PBD objective.Extensive experiments reveal that pretrained bidirectional distillation can significantly improve machine translation performance and achieve competitive or even better results than previous pretrain-finetune or unified multilingual translation methods in supervised, unsupervised, and zero-shot scenarios.Empirically, it is concluded that pretrained bidirectional distillation is an effective and efficient method for transferring language knowledge from pretrained language models to NMT models.
Leveraging Prefix Transfer for Multi-Intent Text Revision

Text revision is a necessary process to improve text quality.During this process, writers constantly edit texts out of different edit intentions.Identifying edit intention for a raw text is always an ambiguous work, and most previous work on revision systems mainly focuses on editing texts according to one specific edit intention.In this work, we aim to build a multiintent text revision system that could revise texts without explicit intent annotation.Our system is based on prefix-tuning, which first gets prefixes for every edit intent, and then trains a prefix transfer module, enabling the system to selectively leverage the knowledge from various prefixes according to the input text.We conduct experiments on the ITER-ATER dataset, and the results show that our system outperforms baselines.The system can significantly improve the SARI score with more than 3% improvements, which thrives on the learned editing intention prefixes.