Reviewerly/exHarmony huggingface dataset:
RISE: Leveraging Retrieval Techniques for Summarization Evaluation

Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and a long document summarization benchmark. The results show that RISE consistently achieves higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.
TAGPRIME: A Unified Framework for Relational Structure Extraction

Many tasks in natural language processing require the extraction of relationship information for a given condition, such as event argument extraction, relation extraction, and taskoriented semantic parsing.Recent works usually propose sophisticated models for each task independently and pay less attention to the commonality of these tasks and to have a unified framework for all the tasks.In this work, we propose to take a unified view of all these tasks and introduce TAGPRIME to address relational structure extraction problems.TAGPRIME is a sequence tagging model that appends priming words about the information of the given condition (such as an event trigger) to the input text.With the self-attention mechanism in pre-trained language models, the priming words make the output contextualized representations contain more information about the given condition, and hence become more suitable for extracting specific relationships for the condition.Extensive experiments and analyses on three different tasks that cover ten datasets across five different languages demonstrate the generality and effectiveness of TAGPRIME.
vONTSS: vMF based semi-supervised neural topic modeling with optimal transport

Recently, Neural Topic Models (NTM), inspired by variational autoencoders, have attracted a lot of research interest; however, these methods have limited applications in the real world due to the challenge of incorporating human knowledge. This work presents a semi-supervised neural topic modeling method, vONTSS, which uses von Mises-Fisher (vMF) based variational autoencoders and optimal transport. When a few keywords per topic are provided, vONTSS in the semi-supervised setting generates potential topics and optimizes topic-keyword quality and topic classification. Experiments show that vONTSS outperforms existing semi-supervised topic modeling methods in classification accuracy and diversity. vONTSS also supports unsupervised topic modeling. Quantitative and qualitative experiments show that vONTSS in the unsupervised setting outperforms recent NTMs on multiple aspects: vONTSS discovers highly clustered and coherent topics on benchmark datasets. It is also much faster than the state-of-the-art weakly supervised text classification method while achieving similar classification performance. We further prove the equivalence of optimal transport loss and cross-entropy loss at the global minimum.
Human-in-the-loop Abstractive Dialogue Summarization

Abstractive dialogue summarization has received increasing attention recently.Despite the fact that most of the current dialogue summarization systems are trained to maximize the likelihood of human-written summaries and have achieved significant results, there is still a huge gap in generating high-quality summaries as determined by humans, such as coherence and faithfulness, partly due to the misalignment in maximizing a single human-written summary.To this end, we propose to incorporate different levels of human feedback into the training process.This will enable us to guide the models to capture the behaviors humans care about for summaries.Specifically, we ask humans to highlight the salient information to be included in summaries to provide the local feedback, and to make overall comparisons among summaries in terms of coherence, accuracy, coverage, concise and overall quality, as the global feedback.We then combine both local and global feedback to fine-tune the dialog summarization policy with Reinforcement Learning.Experiments conducted on multiple datasets demonstrate the effectiveness and generalization of our methods over the stateof-the-art supervised baselines, especially in terms of human judgments 1 .
Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training

Language tasks involving character-level manipulations (e.g., spelling corrections, arithmetic operations, word games) are challenging for models operating on subword units.To address this, we develop a causal intervention framework to learn robust and interpretable character representations inside subword-based language models.Our method treats each character as a typed variable in a causal model and learns such causal structures by adapting the interchange intervention training method of Geiger et al. (2022b).We additionally introduce a suite of character-level tasks that systematically vary in their dependence on meaning and sequence-level context.While characterlevel models still perform best on purely formbased tasks like string reversal, our method outperforms character-level models on more complex tasks that blend form, meaning, and context, such as spelling correction in context and word search games.Compared with standard subword-based models, our approach also significantly improves robustness on unseen token sequences and leads to human-interpretable internal representations of characters.
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale.

In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale.The majority of recent scaling laws studies focused on high-compute high-parameter count settings, leaving the question of when these abilities begin to emerge largely unanswered.In this paper, we investigate whether the effects of pre-training can be observed when the problem size is reduced, modeling a smaller, reduced-vocabulary language.We show the benefits of pre-training with masked language modeling (MLM) objective in models as small as 1.25M parameters, and establish a strong correlation between pre-training perplexity and downstream performance (GLUE benchmark).We examine downscaling effects, extending scaling laws to models as small as 1M parameters.At this scale, we observe a break of the power law for compute-optimal models and show that the MLM loss does not scale smoothly with compute-cost (FLOPs) below 2.2 × 10 15 FLOPs.We also find that adding layers does not always benefit downstream performance. 1
NLPeople at SemEval-2023 Task 2: A Staged Approach for Multilingual Named Entity Recognition

The MultiCoNER II shared task aims at detecting complex, ambiguous named entities with fine-grained types in a low context setting.Previous winning systems incorporated external knowledge bases to retrieve helpful contexts.In our submission we additionally propose splitting the NER task into two stages, a Span Extraction Step, and an Entity Classification step.Our results show that the former does not suffer from the low context setting comparably, and in so leading to a higher overall performance for an external KB-assisted system.We achieve 3 rd place on the multilingual track and an average of 6 th place overall.
Distinguishing Address vs. Reference Mentions of Personal Names in Text

Detecting named entities in text has long been a core NLP task.However, not much work has gone into distinguishing whether an entity mention is addressing the entity vs. referring to the entity; e.g., John, would you turn the light off? vs. John turned the light off.While this distinction is marked by a vocative case marker in some languages, many modern Indo-European languages such as English do not use such explicit vocative markers, and the distinction is left to be interpreted in context.In this paper, we present a new annotated dataset that captures the address vs. reference distinction in English, 1 an automatic tagger that performs at 85% accuracy in making this distinction, and demonstrate how this distinction is important in NLP and computational social science applications in English language.
A New Task and Dataset on Detecting Attacks on Human Rights Defenders

The ability to conduct retrospective analyses of attacks on human rights defenders over time and by location is important for humanitarian organizations to better understand historical or ongoing human rights violations and thus better manage the global impact of such events. We hypothesize that NLP can support such efforts by quickly processing large collections of news articles to detect and summarize the characteristics of attacks on human rights defenders. To that end, we propose a new dataset for detecting Attacks on Human Rights Defenders (HRDsAttack) consisting of crowdsourced annotations on 500 online news articles. The annotations include fine-grained information about the type and location of the attacks, as well as information about the victim(s). We demonstrate the usefulness of the dataset by using it to train and evaluate baseline models on several sub-tasks to predict the annotated characteristics.
Rahul Patil at SemEval-2023 Task 1: V-WSD: Visual Word Sense Disambiguation

Semeval 2023 task 1: VWSD, In this paper, we propose an ensemble of two Neural network systems that ranks 10 images given a word and limited textual context. We have used openAI Clip based models for the English language and multilingual text-to-text translation models for Farsi-to-English and Italian-to-English. Additionally, we propose a system that learns from multilingual bert-base embeddings for text and resnet101 embeddings for the image. Considering all the three languages into account this system has achieved the fourth rank.
Large Language Models are Built-in Autoregressive Search Engines

Document retrieval is a key stage of standard Web search engines.Existing dual-encoder dense retrievers obtain representations for questions and documents independently, allowing for only shallow interactions between them.To overcome this limitation, recent autoregressive search engines replace the dual-encoder architecture by directly generating identifiers for relevant documents in the candidate pool.However, the training cost of such autoregressive search engines rises sharply as the number of candidate documents increases.In this paper, we find that large language models (LLMs) can follow human instructions to directly generate URLs for document retrieval.Surprisingly, when providing a few Query-URL pairs as in-context demonstrations, LLMs can generate Web URLs where nearly 90% of the corresponding documents contain correct answers to open-domain questions.In this way, LLMs can be thought of as built-in search engines, since they have not been explicitly trained to map questions to document identifiers.Experiments demonstrate that our method can consistently achieve better retrieval performance than existing retrieval approaches by a significant margin on three open-domain question answering benchmarks, under both zero and few-shot settings.The code for this work can be found at https://github.com/Ziems/llm-url.
Intermediate Domain Finetuning for Weakly Supervised Domain-adaptive Clinical NER

Accurate human-annotated data for real-world use cases can be scarce and expensive to obtain.In the clinical domain, obtaining such data is even more difficult due to privacy concerns which not only restrict open access to quality data but also require that the annotation be done by domain experts.In this paper, we propose a novel framework -Inter-DAPT -that leverages Intermediate Domain Finetuning to allow language models to adapt to narrow domains with small, noisy datasets.By making use of peripherally-related, unlabeled datasets, this framework circumvents domain-specific data scarcity issues.Our results show that this weakly supervised framework provides performance improvements in downstream clinical named entity recognition tasks.
ZeroAE: Pre-trained Language Model based Autoencoder for Transductive Zero-shot Text Classification

Many text classification tasks require handling unseen domains with plenty of unlabeled data, thus giving rise to the self-adaption or the socalled transductive zero-shot learning (TZSL) problem.However, current methods based solely on encoders or decoders overlook the possibility that these two modules may promote each other.As a first effort to bridge this gap, we propose an autoencoder named ZeroAE.Specifically, the text is encoded with two separate BERT-based encoders into two disentangled spaces, i.e., label-relevant (for classification) and label-irrelevant respectively.The two latent spaces are then decoded by prompting GPT-2 to recover the text as well as to further generate text with labels in the unseen domains to train the encoder in turn.To better exploit the unlabeled data, a novel indirect uncertaintyaware sampling (IUAS) approach is proposed to train ZeroAE.Extensive experiments show that ZeroAE largely surpasses the SOTA methods by 15.93% and 8.70% on average respectively in the label-partially-unseen and labelfully-unseen scenario.Notably, the label-fullyunseen ZeroAE even possesses superior performance to the label-partially-unseen SOTA methods.
The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources

Many state-of-the-art natural language understanding (NLU) models are based on pretrained neural language models.These models often make inferences using information from multiple sources.An important class of such inferences are those that require both background knowledge, presumably contained in a model's pretrained parameters, and instance-specific information that is supplied at inference time.However, the integration and reasoning abilities of NLU models in the presence of multiple knowledge sources have been largely understudied.In this work, we propose a test suite of coreference resolution subtasks that require reasoning over multiple facts.These subtasks differ in terms of which knowledge sources contain the relevant facts.We also introduce subtasks where knowledge is present only at inference time using fictional knowledge.We evaluate state-of-the-art coreference resolution models on our dataset.Our results indicate that several models struggle to reason on-thefly over knowledge observed both at pretrain time and at inference time.However, with taskspecific training, a subset of models demonstrates the ability to integrate certain knowledge types from multiple sources.Still, even the best performing models seem to have difficulties with reliably integrating knowledge presented only at inference time.
Exploiting Knowledge about Discourse Relations for Implicit Discourse Relation Classification

In discourse relation recognition, the classification labels are typically represented as one-hot vectors.However, the categories are in fact not all independent of one another -on the contrary, there are several frameworks that describe the labels' similarities (by e.g.sorting them into a hierarchy or describing them in terms of features (Sanders et al., 2021)).Recently, several methods for representing the similarities between labels have been proposed (Zhang et al., 2018;Wang et al., 2018;Xiong et al., 2021).We here explore and extend the Label Confusion Model (Guo et al., 2021) for learning a representation for discourse relation labels.We explore alternative ways of informing the model about the similarities between relations, by representing relations in terms of their names (and parent category), their typical markers, or in terms of CCR features that describe the relations.Experimental results show that exploiting label similarity improves classification results.
Spontaneous gestures encoded by hand positions improve language models: An Information-Theoretic motivated study

The multi-modality nature of human communication has been utilized to enhance the performance of language modeling-related tasks.Driven by the development of large-scale endto-end learning techniques and the availability of multi-modal data, it becomes possible to represent non-verbal communication behaviors through joint-learning, and directly study their interaction with verbal communication.However, there are still gaps in existing studies to better address the underlying mechanism of how non-verbal expression contributes to the overall communication purpose.Therefore, we explore two questions using mixedmodal language models trained against monologue video data: first, whether incorporating gesture representations can improve the language model's performance (perplexity); second, whether spontaneous gestures demonstrate entropy rate constancy (ERC), which is an empirical pattern found in most verbal language data that supports the rational communication assumption from Information Theory.We have positive and interesting findings for both questions: speakers indeed use spontaneous gestures to convey "meaningful" information that enhances verbal communication, which can be captured with a simple spatial encoding scheme.More importantly, gestures are produced and organized rationally in a similar way as words, which optimizes communication efficiency.
Leveraging Structural Discourse Information for Event Coreference Resolution in Dutch

Structural information is known to be important in resolving coreferential relations.We directly embed discourse structure information (subsection, paragraph and text location) in a transformer-based Dutch event coreference resolution model in order to more explicitly provide it with structural information.Results reveal that integrating this type of knowledge leads to a significant improvement in CONLL F1 for within-document settings (+ 8.6%) and a minor improvement for cross-document settings (+ 1.1%).
Is a Video worth n n Images? A Highly Efficient Approach to Transformer-based Video Question Answering

Conventional Transformer-based Video Question Answering (VideoQA) approaches generally encode frames independently through one or more image encoders followed by interaction between frames and question.However, such schema incur significant memory use and inevitably slow down the training and inference speed.In this work, we present a highly efficient approach for VideoQA based on existing vision-language pre-trained models where we concatenate video frames to a n × n matrix and then convert it to one image.By doing so, we reduce the use of the image encoder from n 2 to 1 while maintaining the temporal structure of the original video.Experimental results on MSRVTT and TrafficQA show that our proposed approach achieves state-of-theart performance with nearly 4× faster speed and only 30% memory use.We show that by integrating our approach into VideoQA systems we can achieve comparable, even superior, performance with a significant speed up for training and inference.We believe the proposed approach can facilitate VideoQA-related research by reducing the computational requirements for those who have limited access to budgets and resources.Our code is publicly available at https://github.com/lyuchenyang/Efficient-VideoQA for research use.
Tübingen at SemEval-2023 Task 4: What Can Stance Tell? A Computational Study on Detecting Human Values behind Arguments

This paper describes the performance of a system which uses stance as an output instead of taking it as an input to identify 20 human values behind given arguments, based on two datasets for SemEval-2023 Task 4. The rationale was to draw a conclusion on whether predicting stance would help predict the given human values better.For this setup-predicting 21 labels-a pre-trained language model, RoBERTa-Large was used.The system had an F 1 -score of 0.50 for predicting these human values for the main test set while this score was 0.35 on the secondary test set, and through further analysis, this paper aims to give insight into the problem of human value identification.
The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

This paper describes the results of the first shared task on generation of teacher responses in educational dialogues.The goal of the task was to benchmark the ability of generative language models to act as AI teachers, replying to a student in a teacher-student dialogue.Eight teams participated in the competition hosted on CodaLab and experimented with a wide variety of state-of-the-art models, including Alpaca, Bloom, DialoGPT, DistilGPT-2, Flan-T5, GPT-2, GPT-3, GPT-4, LLaMA, OPT-2.7B, and T5base.Their submissions were automatically scored using BERTScore and DialogRPT metrics, and the top three among them were further manually evaluated in terms of pedagogical ability based on Tack and Piech (2022).The NAISTeacher system, which ranked first in both automated and human evaluation, generated responses with GPT-3.5 Turbo using an ensemble of prompts and DialogRPT-based ranking of responses for given dialogue contexts.Despite promising achievements of the participating teams, the results also highlight the need for evaluation metrics better suited to educational contexts.
XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations

Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural languages (NLs) into meaning representations (MRs) such as SQL, lambda calculus, and logic forms.However, existing CLSP models are separately proposed and evaluated on datasets of limited tasks and applications, impeding a comprehensive and unified evaluation of CLSP on a diverse range of NLs and MRs.To this end, we present XSEMPLR, a unified benchmark for cross-lingual semantic parsing featured with 22 natural languages and 8 meaning representations by examining and selecting 9 existing datasets to cover 5 tasks and 164 domains.We use XSEMPLR to conduct a comprehensive benchmark study on a wide range of multilingual language models including encoder-based models (mBERT, XLM-R), encoder-decoder models (mBART, mT5), and decoder-based models (Codex, BLOOM).We design 6 experiment settings covering various lingual combinations (monolingual, multilingual, cross-lingual) and numbers of learning samples (full dataset, few-shot, and zero-shot).Our experiments show that encoder-decoder models (mT5) achieve the highest performance compared with other popular models, and multilingual training can further improve the average performance.Notably, multilingual large language models (e.g., BLOOM) are still inadequate to perform CLSP tasks.We also find that the performance gap between monolingual training and cross-lingual transfer learning is still significant for multilingual models, though it can be mitigated by cross-lingual fewshot training.Our dataset and code are available at https://github.com/psunlpgroup/XSemPLR.