Reviewerly/exHarmony huggingface dataset:
Reconstruction Probing

We propose reconstruction probing, a new analysis method for contextualized representations based on reconstruction probabilities in masked language models (MLMs).This method relies on comparing the reconstruction probabilities of tokens in a given sequence when conditioned on the representation of a single token that has been fully contextualized and when conditioned on only the decontextualized lexical prior of the model.This comparison can be understood as quantifying the contribution of contextualization towards reconstruction-the difference in the reconstruction probabilities can only be attributed to the representational change of the single token induced by contextualization.We apply this analysis to three MLMs and find that contextualization boosts reconstructability of tokens that are close to the token being reconstructed in terms of linear and syntactic distance.Furthermore, we extend our analysis to finer-grained decomposition of contextualized representations, and we find that these boosts are largely attributable to static and positional embeddings at the input layer.
UMSE: Unified Multi-scenario Summarization Evaluation

Summarization quality evaluation is a nontrivial task in text summarization.Contemporary methods can be mainly categorized into two scenarios: (1) reference-based: evaluating with human-labeled reference summary; (2) reference-free: evaluating the summary consistency of the document.Recent studies mainly focus on one of these scenarios and explore training neural models built on pre-trained language models (PLMs) to align with human criteria.However, the models from different scenarios are optimized individually, which may result in sub-optimal performance since they neglect the shared knowledge across different scenarios.Besides, designing individual models for each scenario caused inconvenience to the user.Inspired by this, we propose Unified Multi-scenario Summarization Evaluation Model (UMSE).More specifically, we propose a perturbed prefix tuning method to share cross-scenario knowledge between scenarios and use a self-supervised training paradigm to optimize the model without extra human labeling.Our UMSE is the first unified summarization evaluation framework engaged with the ability to be used in three evaluation scenarios.Experimental results across three typical scenarios on the benchmark dataset SummEval indicate that our UMSE can achieve comparable performance with several existing strong methods which are specifically designed for each scenario. 1
Friedrich Nietzsche at SemEval-2023 Task 4: Detection of Human Values from Text Using Machine Learning

Literature permeates almost every facet of our lives, whether through books, magazines, or internet articles.Moreover, every piece of written work contains ideas and opinions that we tend to relate to, accept or disregard, the debate over, or enlighten ourselves.However, the existence of subtle themes that are difficult to discern inspired us to utilize four machine learning algorithms: Decision Trees, Random Forest, Logistic Regression, and Support Vector Classifier (SVC) to aid in their detection.Trained on the ValueEval data set as a multi-label classification problem, the supervised machine learning models did not perform as well as expected, with F1 metrics hovering from 0.0 to 0.04 for each value.Noting this, our paper discusses our approach's limitations and weaknesses.
Bf3R at SemEval-2023 Task 7: a text similarity model for textual entailment and evidence retrieval in clinical trials and animal studies

We describe our participation on the Multievidence Natural Language Inference for Clinical Trial Data (NLI4CT) of SemEval'23.The organizers provided a collection of clinical trials as training data and a set of statements, which can be related to either a single trial or to a comparison of two trials.The task consisted of two sub-tasks: (i) textual entailment (Task 1) for predicting whether the statement is supported (Entailment) or not (Contradiction) by the corresponding trial(s); and (ii) evidence retrieval (Task 2) for selecting the evidences (sentences in the trials) that support the decision made for Task 1.We built a model based on a sentence-based BERT similarity model which was pre-trained on ClinicalBERT embeddings.Our best results on the official test sets were f-scores of 0.64 and 0.67 for Tasks 1 and 2, respectively.
RadLing: Towards Efficient Radiology Report Understanding

Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus.There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks.We present RadLing, a continuously pretrained language model using ELECTRA-small (Clark et al., 2020) architecture, trained using over 500K radiology reports, that can compete with state-of-the-art results for fine tuning tasks in radiology domain.Our main contribution in this paper is knowledge-aware masking which is a taxonomic knowledge-assisted pretraining task that dynamically masks tokens to inject knowledge during pretraining.In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.
Document-Level Multi-Event Extraction with Event Proxy Nodes and Hausdorff Distance Minimization

Document-level multi-event extraction aims to extract the structural information from a given document automatically.Most recent approaches usually involve two steps: (1) modeling entity interactions; (2) decoding entity interactions into events.However, such approaches ignore a global view of inter-dependency of multiple events.Moreover, an event is decoded by iteratively merging its related entities as arguments, which might suffer from error propagation and is computationally inefficient.In this paper, we propose an alternative approach for document-level multi-event extraction with event proxy nodes and Hausdorff distance minimization.The event proxy nodes, representing pseudo-events, are able to build connections with other event proxy nodes, essentially capturing global information.The Hausdorff distance makes it possible to compare the similarity between the set of predicted events and the set of ground-truth events.By directly minimizing Hausdorff distance, the model is trained towards the global optimum directly, which improves performance and reduces training time.Experimental results show that our model outperforms previous state-of-the-art method in F1-score on two datasets with only a fraction of training time. 1 1 Code is available at https://github.com/xnyuwg/procnet... [5]
Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment

In-context learning (ICL) unfolds as large language models become capable of inferring test labels conditioned on a few labeled samples without any gradient update.ICL-enabled large language models provide a promising step forward toward bypassing recurrent annotation costs in a low-resource setting.Yet, only a handful of past studies have explored ICL in a cross-lingual setting, in which the need for transferring label-knowledge from a high-resource language to a low-resource one is immensely crucial.To bridge the gap, we provide the first in-depth analysis of ICL for cross-lingual text classification.We find that the prevalent mode of selecting random inputlabel pairs to construct the prompt-context is severely limited in the case of cross-lingual ICL, primarily due to the lack of alignment in the input as well as the output spaces.To mitigate this, we propose a novel prompt construction strategy -Cross-lingual In-context Source-Target Alignment (X-InSTA).With an injected coherence in the semantics of the input examples and a task-based alignment across the source and target languages, X-InSTA is able to outperform random prompt selection by a large margin across three different tasks using 44 different cross-lingual pairs.
Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction?

Model ensemble has been in widespread use for Grammatical Error Correction (GEC), boosting model performance.We hypothesize that model ensemble based on the perplexity (PPL) computed by pre-trained language models (PLMs) should benefit the GEC system.To this end, we explore several ensemble strategies based on strong PLMs with four sophisticated single models.However, the performance does not improve but even gets worse after the PLM-based ensemble.This surprising result sets us doing a detailed analysis on the data and coming up with some insights on GEC.The human references of correct sentences is far from sufficient in the test data, and the gap between a correct sentence and an idiomatic one is worth our attention.Moreover, the PLM-based ensemble strategies provide an effective way to extend and improve GEC benchmark data.
BIT’s System for Multilingual Track

This paper describes the system we submitted to the IWSLT 2023 multilingual speech translation track, with the input is speech from one language, and the output is text from 10 target languages.Our system consists of CNN and Transformer, convolutional neural networks downsample speech features and extract local information, while transformer extract global features and output the final results.In our system, we use speech recognition tasks to pre-train encoder parameters, and then use speech translation corpus to train the multilingual speech translation model.We have also adopted other methods to optimize the model, such as data augmentation, model ensemble, etc.Our system can obtain satisfactory results on test sets of 10 languages in the MUST-C corpus.
Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning

Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks, but designing prompts that generalize well to diverse problem types can be challenging (Zhou et al., 2022), especially in the context of math word problem (MWP) solving.Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques.To address these issues, we investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation.Our approach is largely inspired by Gao et al. (2022), where they proposed to replace the CoT with the programs as the intermediate reasoning step.Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving.Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the programannotated training data.Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning.Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of finetuned small models in MWP solving 1 .
An Empirical Study of Sentiment-Enhanced Pre-Training for Aspect-Based Sentiment Analysis

Aspect-Based Sentiment Analysis (ABSA) aims to recognize fine-grained opinions and sentiments of users, which is an important problem in sentiment analysis.Recent work has shown that Sentiment-enhanced Pre-Training (SPT) can substantially improve the performance of various ABSA tasks.However, there is currently a lack of comprehensive evaluation and fair comparison of existing SPT approaches.Therefore, this paper performs an empirical study to investigate the effectiveness of different SPT approaches.First, we develop an effective knowledge-mining method and leverage it to build a large-scale knowledgeannotated SPT corpus.Second, we systematically analyze the impact of integrating sentiment knowledge and other linguistic knowledge in pre-training.For each type of sentiment knowledge, we also examine and compare multiple integration methods.Finally, we conduct extensive experiments on a wide range of ABSA tasks to see how much SPT can facilitate the understanding of aspect-level sentiments.
Large Language Models Are Partially Primed in Pronoun Interpretation

While a large body of literature suggests that large language models (LLMs) acquire rich linguistic representations, little is known about whether they adapt to linguistic biases in a human-like way.The present study probes this question by asking whether LLMs display human-like referential biases using stimuli and procedures from real psycholinguistic experiments.Recent psycholinguistic studies suggest that humans adapt their referential biases with recent exposure to referential patterns; closely replicating three relevant psycholinguistic experiments from Johnson and Arnold (2022) in an in-context learning (ICL) framework, we found that InstructGPT adapts its pronominal interpretations in response to the frequency of referential patterns in the local discourse, though in a limited fashion: adaptation was only observed relative to syntactic but not semantic biases.By contrast, FLAN-UL2 fails to generate meaningful patterns.Our results provide further evidence that contemporary LLMs discourse representations are sensitive to syntactic patterns in the local context but less so to semantic patterns.Our data and code are available at https://github.com/zkx06111/llm_priming.
Critic-Guided Decoding for Controlled Text Generation

Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. In this work, we propose a novel critic decoding method for controlled language generation (CriticControl) that combines the strengths of reinforcement learning and weighted decoding. Specifically, we adopt the actor-critic framework and train an LM-steering critic from reward models. Similar to weighted decoding, our method freezes the language model and manipulates the output token distribution using a critic to improve training efficiency and stability. Evaluation of our method on three controlled generation tasks, topic control, sentiment control, and detoxification, shows that our approach generates more coherent and well-controlled texts than previous methods. In addition, CriticControl demonstrates superior generalization ability in zero-shot settings. Human evaluation studies also corroborate our findings.
Unsupervised Open-domain Keyphrase Generation

In this work, we study the problem of unsupervised open-domain keyphrase generation, where the objective is a keyphrase generation model that can be built without using human-labeled data and can perform consistently across domains.To solve this problem, we propose a seq2seq model that consists of two modules, namely phraseness and informativeness module, both of which can be built in an unsupervised and open-domain fashion.The phraseness module generates phrases, while the informativeness module guides the generation towards those that represent the core concepts of the text.We thoroughly evaluate our proposed method using eight benchmark datasets from different domains.Results on in-domain datasets show that our approach achieves stateof-the-art results compared with existing unsupervised models, and overall narrows the gap between supervised and unsupervised methods down to about 16%.Furthermore, we demonstrate that our model performs consistently across domains, as it overall surpasses the baselines on out-of-domain datasets. 1.
Language Modeling with Latent Situations

Language models (LMs) often generate incoherent outputs: they refer to events and entity states that are incompatible with the state of the world described in inputs.We introduce SITU-ATIONSUPERVISION, a family of approaches for improving coherence in LMs by training them to construct and condition on explicit representations of entities and their states.SITU-ATIONSUPERVISION has two components: an auxiliary situation modeling task that trains models to predict entity state representations in context, and a latent state inference procedure that imputes these states from partially annotated training data.SITUATIONSUPERVISION can be applied via fine-tuning (by supervising LMs to encode state variables in their hidden representations) and prompting (by inducing LMs to interleave textual descriptions of entity states with output text).In both cases, it requires only a small number of state annotations to produce substantial coherence improvements (up to an 16% reduction in errors), showing that standard LMs can be efficiently adapted to explicitly model language and aspects of its meaning.
UBC-DLNLP at SemEval-2023 Task 12: Impact of Transfer Learning on African Sentiment Analysis

We describe our contribution to the SemEVAl 2023 AfriSenti-SemEval shared task, where we tackle the task of sentiment analysis in 14 different African languages.We develop both monolingual and multilingual models under a full supervised setting (subtasks A and B).We also develop models for the zero-shot setting (subtask C).Our approach involves experimenting with transfer learning using six language models, including further pretraining of some of these models as well as a final finetuning stage.Our best performing models achieve an F 1 -score of 70.36 on development data and an F 1 -score of 66.13 on test data.Unsurprisingly, our results demonstrate the effectiveness of transfer learning and finetuning techniques for sentiment analysis across multiple languages.Our approach can be applied to other sentiment analysis tasks in different languages and domains.
Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCo

This paper contains content that can be offensive or disturbing.Annotator disagreement is common whenever human judgment is needed for supervised learning.It is conventional to assume that one label per item represents ground truth.However, this obscures minority opinions, if present.We regard "ground truth" as the distribution of all labels that a population of annotators could produce, if asked (and of which we only have a small sample).We next introduce DisCo (Distribution from Context), a simple neural model that learns to predict this distribution.The model takes annotator-item pairs, rather than items alone, as input, and performs inference by aggregating over all annotators.Despite its simplicity, our experiments show that, on six benchmark datasets, our model is competitive with, and frequently outperforms, other, more complex models that either do not model specific annotators or were not designed for label distribution learning.
Zero-Shot Text Classification via Self-Supervised Tuning

Existing solutions to zero-shot text classification either conduct prompting with pre-trained language models, which is sensitive to the choices of templates, or rely on large-scale annotated data of relevant tasks for meta-tuning.In this work, we propose a new paradigm based on self-supervised learning to solve zeroshot text classification tasks by tuning the language models with unlabeled data, called selfsupervised tuning.By exploring the inherent structure of free texts, we propose a new learning objective called first sentence prediction to bridge the gap between unlabeled data and text classification tasks.After tuning the model to learn to predict the first sentence in a paragraph based on the rest, the model is able to conduct zero-shot inference on unseen tasks such as topic classification and sentiment analysis.Experimental results show that our model outperforms the state-of-the-art baselines on 7 out of 10 tasks.Moreover, the analysis reveals that our model is less sensitive to the prompt design.
Categorial grammar induction from raw data

Grammar induction, the task of learning a set of grammatical rules from raw or minimally labeled text data, can provide clues about what kinds of syntactic structures are learnable without prior knowledge.Recent work (e.g., Kim et al., 2019;Zhu et al., 2020; Jin et al., 2021a) has achieved advances in unsupervised induction of probabilistic context-free grammars (PCFGs).However, categorial grammar induction has received less recent attention, despite allowing inducers to support a larger set of syntactic categories-due to restrictions on how categories can combine-and providing a transparent interface with compositional semantics, opening up possibilities for models that jointly learn form and meaning.Motivated by this, we propose a new model for inducing a basic (Ajdukiewicz, 1935;Bar-Hillel, 1953) categorial grammar.In contrast to earlier categorial grammar induction systems (e.g., Bisk and Hockenmaier, 2012), our model learns from raw data without any part-of-speech information.Experiments on child-directed speech show that our model attains a recall-homogeneity of 0.33 on average, which dramatically increases to 0.59 when a bias toward forward function application is added to the model.
On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing

We provide a survey and empirical comparison of the state-of-the-art in neural selective classification for NLP tasks.We also provide a methodological blueprint, including a novel metric called refinement that provides a calibrated evaluation of confidence functions for selective prediction.Finally, we supply documented, open-source code to support the future development of selective prediction techniques.
Debiasing should be Good and Bad: Measuring the Consistency of Debiasing Techniques in Language Models

Debiasing methods that seek to mitigate the tendency of Language Models (LMs) to occasionally output toxic or inappropriate text have recently gained traction.In this paper, we propose a standardized protocol which distinguishes methods that yield not only desirable results, but are also consistent with their mechanisms and specifications.For example, we ask, given a debiasing method that is developed to reduce toxicity in LMs, if the definition of toxicity used by the debiasing method is reversed, would the debiasing results also be reversed?We used such considerations to devise three criteria for our new protocol: Specification Polarity, Specification Importance, and Domain Transferability.As a case study, we apply our protocol to a popular debiasing method, Self-Debiasing, and compare it to one we propose, called Instructive Debiasing, and demonstrate that consistency is as important an aspect to debiasing viability as is simply a desirable result.We show that our protocol provides essential insights into the generalizability and interpretability of debiasing methods that may otherwise go overlooked.
WKU_NLP at SemEval-2023 Task 9: Translation Augmented Multilingual Tweet Intimacy Analysis

This paper presents a system designed for the SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis.This system is composed of a pretrained multilingual masked language model as a text encoder and a neural network as a regression model.To enhance model performance in low-resource scenarios, the system employs data augmentation techniques leveraging neural machine translation models.Furthermore, I demonstrate the system can be further improved through the ensemble of topperforming models in each language.This system ranks 4 th in languages unseen in the training data and 16 th in languages seen in the training data.The code and data are accessible vias the following link: