Reviewerly/exHarmony huggingface dataset:
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models

Controlling the text generated by language models and customizing the content has been a long-standing challenge.Existing prompting techniques proposed in pursuit of providing control are task-specific and lack generality; this provides overwhelming choices for non-expert users to find a suitable method for their task.The effort associated with those techniques, such as in writing examples, explanations, instructions, etc. further limits their adoption among non-expert users.In this paper, we propose a simple prompting strategy HELP ME THINK where we encourage large language models (such as GPT3 and ChatGPT) to help non-expert users by asking a set of relevant questions and leveraging user answers to execute the task.We demonstrate the efficacy of our technique HELP ME THINK on a variety of tasks.Specifically, we focus on tasks that are hard for average humans and require significant thinking to perform.We hope our work will encourage the development of unconventional ways to harness the power of large language models.
Distantly Supervised Course Concept Extraction in MOOCs with Academic Discipline

With the rapid growth of Massive Open Online Courses (MOOCs), it is expensive and time-consuming to extract high-quality knowledgeable concepts taught in the course by human effort to help learners grasp the essence of the course.In this paper, we propose to automatically extract course concepts using distant supervision to eliminate the heavy work of human annotations, which generates labels by matching them with an easily accessed dictionary.However, this matching process suffers from severe noisy and incomplete annotations because of the limited dictionary and diverse MOOCs.To tackle these challenges, we present a novel three-stage framework DS-MOCE, which leverages the power of pretrained language models explicitly and implicitly and employs discipline-embedding models with a self-train strategy based on label generation refinement across different domains.We also provide an expert-labeled dataset spanning 20 academic disciplines.Experimental results demonstrate the superiority of DS-MOCE over the state-of-the-art distantly supervised methods (with 7% absolute F1 score improvement).Code and data are now available at https: //github.com/THU-KEG/MOOC-NER.
UIO at SemEval-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource Languages

Our contribution to the 2023 AfriSenti-SemEval shared task 12: Sentiment Analysis for African Languages, provides insight into how a multilingual large language model can be a resource for sentiment analysis in languages not seen during pretraining.The shared task provides datasets of a variety of African languages from different language families.The languages are to various degrees related to languages used during pretraining, and the language data contain various degrees of codeswitching.We experiment with both monolingual and multilingual datasets for the final finetuning, and find that with the provided datasets that contain samples in the thousands, monolingual fine-tuning yields the best results.
GersteinLab at MEDIQA-Chat 2023: Clinical Note Summarization from Doctor-Patient Conversations through Fine-tuning and In-context Learning

This paper presents our contribution to the MEDIQA-2023 Dialogue2Note shared task, encompassing both subtask A and subtask B. We approach the task as a dialogue summarization problem and implement two distinct pipelines: (a) a fine-tuning of a pre-trained dialogue summarization model and GPT-3, and (b) few-shot in-context learning (ICL) using a large language model, GPT-4.Both methods achieve excellent results in terms of ROUGE-1 F1, BERTScore F1 (deberta-xlarge-mnli), and BLEURT, with scores of 0.4011, 0.7058, and 0.5421, respectively.Additionally, we predict the associated section headers using RoBERTa and SciBERT based classification models.Our team ranked fourth among all teams, while each team is allowed to submit three runs as part of their submission.We also utilize expert annotations to demonstrate that the notes generated through the ICL GPT-4 are better than all other baselines.The code for our submission is available 1 .
Medical knowledge-enhanced prompt learning for diagnosis classification from clinical text

Artificial intelligence based diagnosis systems have emerged as powerful tools to reform traditional medical care.Each clinician now wants to have his own intelligent diagnostic partner to expand the range of services he can provide.When reading a clinical note, experts make inferences with relevant knowledge.However, medical knowledge appears to be heterogeneous, including structured and unstructured knowledge.Existing approaches are incapable of uniforming them well.Besides, the descriptions of clinical findings in clinical notes, which are reasoned to diagnosis, vary a lot for different diseases or patients.To address these problems, we propose a Medical Knowledgeenhanced Prompt Learning (MedKPL) model for diagnosis classification.First, to overcome the heterogeneity of knowledge, given the knowledge relevant to diagnosis, MedKPL extracts and normalizes the relevant knowledge into a prompt sequence.Then, MedKPL integrates the knowledge prompt with the clinical note into a designed prompt for representation.Therefore, MedKPL can integrate medical knowledge into the models to enhance diagnosis and effectively transfer learned diagnosis capacity to unseen diseases using alternating relevant disease knowledge.The experimental results on two medical datasets show that our method can obtain better medical text classification results and can perform better in transfer and few-shot settings among datasets of different diseases.
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question

Existing metrics for evaluating the quality of automatically generated questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and predicted questions, providing a high score when there is a considerable lexical overlap or semantic similarity between the candidate and the reference questions.This approach has two major shortcomings.First, we need expensive human-provided reference questions.Second, it penalises valid questions that may not have high lexical or semantic similarity to the reference questions.In this paper, we propose a new metric, RQUGE, based on the answerability of the candidate question given the context.The metric consists of a question-answering and a span scorer modules, using pre-trained models from existing literature, thus it can be used without any further training.We demonstrate that RQUGE has a higher correlation with human judgment without relying on the reference question.Additionally, RQUGE is shown to be more robust to several adversarial corruptions.Furthermore, we illustrate that we can significantly improve the performance of QA models on out-of-domain datasets by fine-tuning on synthetic data generated by a question generation model and reranked by RQUGE.
Exploring Zero and Few-shot Techniques for Intent Classification

Conversational NLU providers often need to scale to thousands of intent-classification models where new customers often face the coldstart problem.Scaling to so many customers puts a constraint on storage space as well.In this paper, we explore four different zero and few-shot intent classification approaches with this low-resource constraint: 1) domain adaptation, 2) data augmentation, 3) zero-shot intent classification using descriptions large language models (LLMs), and 4) parameter-efficient fine-tuning of instruction-finetuned language models.Our results show that all these approaches are effective to different degrees in low-resource settings.Parameter-efficient finetuning using T-few recipe (Liu et al., 2022) on Flan-T5 (Chung et al., 2022) yields the best performance even with just one sample per intent.We also show that the zero-shot method of prompting LLMs using intent descriptions is also very competitive.
Efficient Transformers with Dynamic Token Pooling

Transformers achieve unrivalled performance in modelling language, but remain inefficient in terms of memory and time complexity.A possible remedy is to reduce the sequence length in the intermediate layers by pooling fixed-length segments of tokens.Nevertheless, natural units of meaning, such as words or phrases, display varying sizes.To address this mismatch, we equip language models with a dynamic-pooling mechanism, which predicts segment boundaries in an autoregressive fashion.We compare several methods to infer boundaries, including end-to-end learning through stochastic re-parameterisation, supervised learning (based on segmentations from subword tokenizers or spikes in conditional entropy), as well as linguistically motivated boundaries.We perform character-level evaluation on texts from multiple datasets and morphologically diverse languages.The results demonstrate that dynamic pooling, which jointly segments and models language, is both faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget.
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages

The rapid growth of machine translation (MT) systems necessitates meta-evaluations of evaluation metrics to enable selection of those that best reflect MT quality.Unfortunately, most meta-evaluation studies focus on European languages, the observations for which may not always apply to other languages.Indian languages, having over a billion speakers, are linguistically different from them, and to date, there are no such systematic studies focused solely on English to Indian language MT.This paper fills this gap through a Multidimensional Quality Metric (MQM) dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems.We evaluate 16 metrics and show that, pre-trained metrics like COMET have the highest correlations with annotator scores as opposed to n-gram metrics like BLEU.We further leverage our MQM annotations to develop an Indic-COMET metric and show that it outperforms COMET counterparts in both human scores correlations and robustness scores in Indian languages.Additionally, we show that the Indic-COMET can outperform COMET on some unseen Indian languages.We hope that our dataset and analysis will facilitate further research in Indic MT evaluation.
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions

Modern machine learning relies on datasets to develop and validate research ideas.Given the growth of publicly available data, finding the right dataset to use is increasingly difficult.Any research question imposes explicit and implicit constraints on how well a given dataset will enable researchers to answer this question, such as dataset size, modality, and domain.We operationalize the task of recommending datasets given a short natural language description of a research idea, to help people find relevant datasets for their needs.Dataset recommendation poses unique challenges as an information retrieval problem; datasets are hard to directly index for search and there are no corpora readily available for this task.To facilitate this task, we build the DataFinder Dataset which consists of a larger automatically-constructed training set (17.5K queries) and a smaller expertannotated evaluation set (392 queries).Using this data, we compare various information retrieval algorithms on our test set and present a superior bi-encoder retriever for text-based dataset recommendation.This system, trained on the DataFinder Dataset, finds more relevant search results than existing third-party dataset search engines.To encourage progress on dataset recommendation, we release our dataset and models to the public. 11 Code and data: https://github.com/viswavi/datafinder Natural Language Query "I want to use adversarial learning to perform domain adaptation for semantic segmentation of images."
People and Places of Historical Europe: Bootstrapping Annotation Pipeline and a New Corpus of Named Entities in Late Medieval Texts

Although pre-trained named entity recognition (NER) models are highly accurate on modern corpora, they underperform on historical texts due to differences in language OCR errors.In this work, we develop a new NER corpus of 3.6M sentences from late medieval charters written mainly in Czech, Latin, and German.We show that we can start with a list of known historical figures and locations and an unannotated corpus of historical texts, and use information retrieval techniques to automatically bootstrap a NER-annotated corpus.Using our corpus, we train a NER model that achieves entity-level Precision of 72.81-93.98%with 58.14-81.77%Recall on a manually-annotated test dataset.Furthermore, we show that using a weighted loss function helps to combat class imbalance in token classification tasks.To make it easy for others to reproduce and build upon our work, we publicly release our corpus, models, and experimental code.
Semantic-Oriented Unlabeled Priming for Large-Scale Language Models

Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning.Unfortunately, for in-context learning there is currently no way to leverage unlabeled data, which is often much easier to obtain in large quantities than labeled examples.In this work, we therefore investigate ways to make use of unlabeled examples to improve the zero-shot performance of pretrained language models without any finetuning: We introduce Semantic-Oriented Unlabeled Priming (SOUP), a method that classifies examples by retrieving semantically similar unlabeled examples, assigning labels to them in a zero-shot fashion, and then using them for in-context learning.We also propose bag-of-contexts priming, a new priming strategy that is more suitable for our setting and enables the usage of more examples than fit into the context window.
Entailment as Robust Self-Learner

Entailment has been recognized as an important metric for evaluating natural language understanding (NLU) models, and recent studies have found that entailment pretraining benefits weakly supervised fine-tuning.In this work, we design a prompting strategy that formulates a number of different NLU tasks as contextual entailment.This approach improves the zero-shot adaptation of pretrained entailment models.Secondly, we notice that self-training entailment-based models with unlabeled data can significantly improve the adaptation performance on downstream tasks.To achieve more stable improvement, we propose the Simple Pseudo-Label Editing (SimPLE) algorithm for better pseudo-labeling quality in self-training.We also found that both pretrained entailmentbased models and the self-trained models are robust against adversarial evaluation data.Experiments on binary and multi-class classification tasks show that SimPLE leads to more robust self-training results, indicating that the self-trained entailment models are more efficient and trustworthy than large language models on language understanding tasks.
IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning

In the field of machine reading comprehension (MRC), existing systems have surpassed the average performance of human beings in many tasks like SQuAD. However, there is still a long way to go when it comes to logical reasoning. Although some methods for it have been put forward, they either are designed in a quite complicated way or rely too much on external structures. In this paper, we proposed IDOL (InDicator-Oriented Logic Pre-training), an easy-to-understand but highly effective further pre-training task which logically strengthens the pre-trained models with the help of 6 types of logical indicators and a logically rich dataset LoGic Pre-training (LGP). IDOL achieves state-of-the-art performance on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC, and is proven to be capable of generalizing to different pre-trained models and other types of MRC benchmarks like RACE and SQuAD 2.0 while keeping competitive general language understanding ability through testing on tasks in GLUE. Besides, at the beginning of the era of large language models, we take several of them like ChatGPT into comparison and find that IDOL still shows its advantage.
Probing Physical Reasoning with Counter-Commonsense Context

In this study, we create a CConS (Countercommonsense Contextual Size comparison) dataset to investigate how physical commonsense affects the contextualized size comparison task; the proposed dataset consists of both contexts that fit physical commonsense and those that do not.This dataset tests the ability of language models to predict the size relationship between objects under various contexts generated from our curated noun list and templates.We measure the ability of several masked language models and generative models.The results show that while large language models can use prepositions such as "in" and "into" in the provided context to infer size relationships, they fail to use verbs and thus make incorrect judgments led by their prior physical commonsense.
Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora

Grammatical error correction (GEC) is the task of correcting typos, spelling, punctuation and grammatical issues in text.Approaching the problem as a sequence-to-sequence task, we compare the use of a common subword unit vocabulary and byte-level encoding.Initial synthetic training data is created using an error-generating pipeline, and used for finetuning two subword-level models and one bytelevel model.Models are then finetuned further on hand-corrected error corpora, including texts written by children, university students, dyslexic and second-language writers, and evaluated over different error types and origins.We show that a byte-level model enables higher correction quality than a subword approach, not only for simple spelling errors, but also for more complex semantic, stylistic and grammatical issues.In particular, initial training on synthetic corpora followed by finetuning on a relatively small parallel corpus of real-world errors helps the byte-level model correct a wide range of commonly occurring errors.Our experiments are run for the Icelandic language but should hold for other similar languages, particularly morphologically rich ones.
The Larger they are, the Harder they Fail: Language Models do not Recognize Identifier Swaps in Python

Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming.Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers.We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size.Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.* Equal contribution. 1 Since model capacity in number of parameters and pretraining dataset size are balanced according to a design law that is fixed for each model family and is intended to empirically maximize the pretraining set likelihood given a compute budget (Kaplan et al., 2020), for the reminder of this paper we will jointly refer to them as "model size".
Not Enough Data to Pre-train Your Language Model? MT to the Rescue!

In recent years, pre-trained transformer-based language models (LM) have become a key resource for implementing most NLP tasks.However, pre-training such models demands large text collections not available in most languages.In this paper, we study the use of machinetranslated corpora for pre-training LMs.We answer the following research questions: RQ1: Is MT-based data an alternative to real data for learning a LM?; RQ2: Can real data be complemented with translated data and improve the resulting LM?In order to validate these two questions, several BERT models for Basque have been trained, combining real data and synthetic data translated from Spanish.The evaluation carried out on 9 NLU tasks indicates that models trained exclusively on translated data offer competitive results.Furthermore, models trained with real data can be improved with synthetic data, although further research is needed on the matter.
Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction

We study the influence of different activation functions in the output layer of pre-trained transformer models for soft and hard label prediction in the learning with disagreement task. In this task, the goal is to quantify the amount of disagreement via predicting soft labels. To predict the soft labels, we use BERT-based preprocessors and encoders and vary the activation function used in the output layer, while keeping other parameters constant. The soft labels are then used for the hard label prediction. The activation functions considered are sigmoid as well as a step-function that is added to the model post-training and a sinusoidal activation function, which is introduced for the first time in this paper.
A Unified One-Step Solution for Aspect Sentiment Quad Prediction

Aspect sentiment quad prediction (ASQP) is a challenging yet significant subtask in aspectbased sentiment analysis as it provides a complete aspect-level sentiment structure.However, existing ASQP datasets are usually small and low-density, hindering technical advancement.To expand the capacity, in this paper, we release two new datasets for ASQP, which contain the following characteristics: larger size, more words per sample, and higher density.With such datasets, we unveil the shortcomings of existing strong ASQP baselines and therefore propose a unified one-step solution for ASQP, namely One-ASQP, to detect the aspect categories and to identify the aspectopinion-sentiment (AOS) triplets simultaneously.Our One-ASQP holds several unique advantages: (1) by separating ASQP into two subtasks and solving them independently and simultaneously, we can avoid error propagation in pipeline-based methods and overcome slow training and inference in generation-based methods; (2) by introducing sentiment-specific horns tagging schema in a token-pair-based two-dimensional matrix, we can exploit deeper interactions between sentiment elements and efficiently decode the AOS triplets; (3) we design "[NULL]" token can help us effectively identify the implicit aspects or opinions.Experiments on two benchmark datasets and our released two datasets demonstrate the advantages of our One-ASQP.The two new datasets are publicly released at https://www.github.