Reviewerly/exHarmony huggingface dataset:
StoryWars: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generation

Collaborative stories, which are texts created through the collaborative efforts of multiple authors with different writing styles and intentions, pose unique challenges for NLP models.Understanding and generating such stories remains an underexplored area due to the lack of open-domain corpora.To address this, we introduce STORYWARS, a new dataset of over 40,000 collaborative stories written by 9,400 different authors from an online platform.We design 12 task types, comprising 7 understanding and 5 generation task types, on STORY-WARS, deriving 101 diverse story-related tasks in total as a multi-task benchmark covering all fully-supervised, few-shot, and zero-shot scenarios.Furthermore, we present our instructiontuned model, INSTRUCTSTORY, for the story tasks showing that instruction tuning, in addition to achieving superior results in zero-shot and few-shot scenarios, can also obtain the best performance on the fully-supervised tasks in STORYWARS, establishing strong multi-task benchmark performances on STORYWARS. 1
CoMave: Contrastive Pre-training with Multi-scale Masking for Attribute Value Extraction

Attribute Value Extraction (AVE) aims to automatically obtain attribute value pairs from product descriptions to aid e-commerce.Despite the progressive performance of existing approaches in e-commerce platforms, they still suffer from two challenges: 1) difficulty in identifying values at different scales simultaneously; 2) easy confusion by some highly similar fine-grained attributes.This paper proposes a pre-training technique for AVE to address these issues.In particular, we first improve the conventional token-level masking strategy, guiding the language model to understand multi-scale values by recovering spans at the phrase and sentence level.Second, we apply clustering to build a challenging negative set for each example and design a pre-training objective based on contrastive learning to force the model to discriminate similar attributes.Comprehensive experiments show that our solution provides a significant improvement over traditional pretrained models in the AVE task, and achieves state-of-the-art on four benchmarks 1 .
Incorporating Factuality Inference to Identify Document-level Event Factuality

Document-level Event Factuality Identification (DEFI) refers to identifying the degree of certainty that a specific event occurs in a document.Previous studies on DEFI failed to link the document-level event factuality with various sentence-level factuality values in the same document.In this paper, we innovatively propose an event factuality inference task to bridge the sentence-level and the document-level event factuality semantically.Specifically, we present a Sentence-to-Document Inference Network (SDIN) that contains a multi-layer interaction module and a gated aggregation module to integrate the above two tasks, and employ a multitask learning framework to improve the performance of DEFI.The experimental results on the public English and Chinese DLEF datasets show that our model outperforms the SOTA baselines significantly.
DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection

This paper presents the submissions of the DH-FBK team for the three tasks of Task 10 at SemEval 2023.The Explainable Detection of Online Sexism (EDOS) task aims at detecting sexism in English text in an accurate and explainable way, thanks to a fine-grained annotation that follows a three-level schema: sexist or not (Task A), category of sexism (Task B) and vector of sexism (Task C) exhibited.We use a multi-task learning approach in which models share representations from all three tasks, allowing for knowledge to be shared across them.Notably, with our approach a single model can solve all three tasks.In addition, motivated by the subjective nature of the task, we incorporate inter-annotator agreement information in our multi-task architecture.Although disaggregated annotations are not available, we artificially estimate them using a 5-classifier ensemble, and show that ensemble agreement can be a good approximation of crowd agreement.Our approach achieves competitive results, ranking 32 nd out of 84, 24 th out of 69 and 11 th out of 63 for Tasks A, B and C respectively.We finally show that low inter-annotator agreement levels are associated with more challenging examples for models, making agreement information useful for this kind of task.
Unsupervised Mapping of Arguments of Deverbal Nouns to Their Corresponding Verbal Labels

Deverbal nouns are nominal forms of verbs commonly used in written English texts to describe events or actions, as well as their arguments.However, many NLP systems, and in particular pattern-based ones, neglect to handle such nominalized constructions.The solutions that do exist for handling arguments of nominalized constructions are based on semantic annotation and require semantic ontologies, making their applications restricted to a small set of nouns.We propose to adopt instead a more syntactic approach, which maps the arguments of deverbal nouns to the universal-dependency relations of the corresponding verbal construction.We present an unsupervised mechanismbased on contextualized word representationswhich allows to enrich universal-dependency trees with dependency arcs denoting arguments of deverbal nouns, using the same labels as the corresponding verbal cases.By sharing the same label set as in the verbal case, patterns that were developed for verbs can be applied without modification but with high accuracy also to the nominal constructions.
Retrieval-augmented Video Encoding for Instructional Captioning

Instructional videos make learning knowledge more efficient, by providing a detailed multimodal context of each procedure in instruction.A unique challenge posed by instructional videos is key-object degeneracy, where any single modality fails to sufficiently capture the key objects referred to in the procedure.For machine systems, such degeneracy can disturb the performance of a downstream task such as dense video captioning, leading to the generation of incorrect captions omitting key objects.To repair degeneracy, we propose a retrievalbased framework to augment the model representations in the presence of such key-object degeneracy.We validate the effectiveness and generalizability of our proposed framework over baselines using modalities with key-object degeneracy.
Explainable Recommendation with Personalized Review Retrieval and Aspect Learning

Explainable recommendation is a technique that combines prediction and generation tasks to produce more persuasive results.Among these tasks, textual generation demands large amounts of data to achieve satisfactory accuracy.However, historical user reviews of items are often insufficient, making it challenging to ensure the precision of generated explanation text.To address this issue, we propose a novel model, ERRA (Explainable Recommendation by personalized Review retrieval and Aspect learning).With retrieval enhancement, ERRA can obtain additional information from the training sets.With this additional information, we can generate more accurate and informative explanations.Furthermore, to better capture users' preferences, we incorporate an aspect enhancement component into our model.By selecting the top-n aspects that users are most concerned about for different items, we can model user representation with more relevant details, making the explanation more persuasive.To verify the effectiveness of our model, extensive experiments on three datasets show that our model outperforms state-of-theart baselines (for example, 3.4% improvement in prediction and 15.8% improvement in explanation for TripAdvisor).
ArgAnalysis35K : A large-scale dataset for Argument Quality Analysis

Argument Quality Detection is an emerging field in NLP which has seen significant recent development.However, existing datasets in this field suffer from a lack of quality, quantity and diversity of topics and arguments, specifically the presence of vague arguments that are not persuasive in nature.In this paper, we leverage a combined experience of 10+ years of Parliamentary Debating to create a dataset that covers significantly more topics and has a wide range of sources to capture more diversity of opinion.With 34,890 high-quality argument-analysis pairs (a term we introduce in this paper), this is also the largest dataset of its kind to our knowledge.In addition to this contribution, we introduce an innovative argument scoring system based on instance-level annotator reliability and propose a quantitative model of scoring the relevance of arguments to a range of topics.
Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models

The bias-variance tradeoff is the idea that learning methods need to balance model complexity with data size to minimize both under-fitting and over-fitting.Recent empirical work and theoretical analyses with over-parameterized neural networks challenge the classic bias-variance trade-off notion suggesting that no such trade-off holds: as the width of the network grows, bias monotonically decreases while variance initially increases followed by a decrease.In this work, we first provide a variance decomposition-based justification criteria to examine whether large pretrained neural models in a fine-tuning setting are generalizable enough to have low bias and variance.We then perform theoretical and empirical analysis using ensemble methods explicitly designed to decrease variance due to optimization.This results in essentially a two-stage fine-tuning algorithm that first ratchets down bias and variance iteratively, and then uses a selected fixed-bias model to further reduce variance due to optimization by ensembling.We also analyze the nature of variance change with the ensemble size in low-and high-resource classes.Empirical results show that this two-stage method obtains strong results on SuperGLUE tasks and clinical information extraction tasks.Code and settings are available: https://github.com/christa60/bias-var-fine-tuning-plms.git
Analyzing Text Representations by Measuring Task Alignment

Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios.What makes a representation good for text classification?Is it due to the geometric properties of the space or because it is well aligned with the task?We hypothesize the second claim.To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity.Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.
MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning

Recent advances in pre-trained language models (PLMs) have facilitated the development of commonsense reasoning tasks.However, existing methods rely on multi-hop knowledge retrieval and thus suffer low accuracy due to embedded noise in the acquired knowledge.In addition, these methods often attain high computational costs and nontrivial knowledge loss because they encode the knowledge independently of the PLM, making it less relevant to the task and resulting in a poor local optimum.In this work, we propose Multi-View Knowledge Retrieval with Prompt Tuning (MVP-Tuning).Our MVP-Tuning leverages similar questionanswer pairs in training set to improve knowledge retrieval and employs a single prompttuned PLM to model knowledge and input text jointly.We conduct our experiments on five commonsense reasoning QA benchmarks to show that MVP-Tuning outperforms all other baselines in 4 out of 5 datasets with only as most 2% trainable parameters.The ensemble of our MVP-Tuning models even gets a new state-of-the-art performance on OpenBookQA and is ranked first place on the leaderboard 1 .Our code and data are available 2 .
A Weakly-Supervised Learning Approach to the Identification of “Alternative Lexicalizations” in Shallow Discourse Parsing

Recently, the identification of free connective phrases as signals for discourse relations has received new attention with the introduction of statistical models for their automatic extraction.The limited amount of annotations makes it still challenging to develop well-performing models.In our work, we want to overcome this limitation with semi-supervised learning from unlabeled news texts.We implement a self-supervised sequence labeling approach and filter its predictions by a second model trained to disambiguate signal candidates.With our novel model design, we report state-of-the-art results and in addition, achieve an average improvement of about 5% for both exactly and partially matched alternatively-lexicalized discourse signals due to weak supervision.
BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation

We present a large-scale video subtitle translation dataset, BIGVIDEO, to facilitate the study of multi-modality machine translation.Compared with the widely used HOW2 and VATEX datasets, BIGVIDEO is more than 10 times larger, consisting of 4.5 million sentence pairs and 9,981 hours of videos.We also introduce two deliberately designed test sets to verify the necessity of visual information: AMBIGUOUS with the presence of ambiguous words, and UNAMBIGUOUS in which the text context is self-contained for translation.To better model the common semantics shared across texts and videos, we introduce a contrastive learning method in the cross-modal encoder.Extensive experiments on the BIGVIDEO show that: a) Visual information consistently improves the NMT model in terms of BLEU, BLEURT, and COMET on both AMBIGUOUS and UNAMBIGUOUS test sets.b) Visual information helps disambiguation, compared to the strong text baseline on terminologytargeted scores and human evaluation.Dataset and our implementations are available at https://github.com/DeepLearnXMU/BigVideo-VMT.
QUESPA Submission for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks

This article describes the QUESPA team speech translation (ST) submissions for the Quechua to Spanish (QUE-SPA) track featured in the Evaluation Campaign of IWSLT 2023: lowresource and dialect speech translation.Two main submission types were supported in the campaign: constrained and unconstrained.We submitted six total systems of which our best (primary) constrained system consisted of an ST model based on the Fairseq S2T framework where the audio representations were created using log mel-scale filter banks as features and the translations were performed using a transformer.The best (primary) unconstrained system used a pipeline approach which combined automatic speech recognition (ASR) with machine translation (MT).The ASR transcriptions for the best unconstrained system were computed using a pre-trained XLS-R-based model along with a fine-tuned language model.Transcriptions were translated using a MT system based on a fine-tuned, pre-trained language model (PLM).The four other submissions are presented in this article (2 constrained and 2 unconstrained) for comparison because they consist of various architectures.Our results show that direct ST (ASR and MT combined together) can be more effective than a PLM in a low-resource (constrained) setting for Quechua to Spanish.On the other hand, we show that fine-tuning of any type on both the ASR and MT system is worthwhile, resulting in nearly 16 BLEU for the unconstrained task.
Triggering Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks

Despite readily memorizing world knowledge about entities, pre-trained language models (LMs) struggle to compose together two or more facts to perform multi-hop reasoning in question-answering tasks.In this work, we propose techniques that improve upon this limitation by relying on random walks over structured knowledge graphs.Specifically, we use soft prompts to guide LMs to chain together their encoded knowledge by learning to map multi-hop questions to random walk paths that lead to the answer.Applying our methods on two T5 LMs shows substantial improvements over standard tuning approaches in answering questions that require 2-hop reasoning.
How Do In-Context Examples Affect Compositional Generalization?

Compositional generalization-understanding unseen combinations of seen primitives-is an essential reasoning capability in human intelligence.The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning-the prevailing few-shot paradigm based on large language models-exhibits compositional generalization.In this paper, we present COFE, a test suite to investigate in-context compositional generalization.We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization.We study three potential factors: similarity, diversity and complexity.Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple.Furthermore, two strong limitations are observed: in-context compositional generalization on fictional words is much weaker than that on commonly used ones; it is still critical that the incontext examples should cover required linguistic structures, even though the backbone model has been pre-trained on large corpus.We hope our analysis would facilitate the understanding and utilization of in-context learning paradigm.
AUGUST: an Automatic Generation Understudy for Synthesizing Conversational Recommendation Datasets

High-quality data is essential for conversational recommendation systems and serves as the cornerstone of the network architecture development and training strategy design.Existing works contribute heavy human efforts to manually labeling or designing and extending recommender dialogue templates.However, they suffer from: (i) the limited number of human annotators results in that datasets can hardly capture rich and large-scale cases in the real world, (ii) the limited experience and knowledge of annotators accounts for the uninformative corpus and inappropriate recommendations.In this paper, we propose a novel automatic dataset synthesis approach that can generate both large-scale and high-quality recommendation dialogues through a data2text generation process, where unstructured recommendation conversations are generated from structured graphs based on user-item information from the real world.In doing so, we comprehensively exploit: (i) rich personalized user profiles from traditional recommendation datasets, (ii) rich external knowledge from knowledge graphs, and (iii) the conversation ability contained in human-to-human conversational recommendation datasets.Extensive experiments validate the benefit brought by the automatically synthesised data under the low-resource scenarios, and demonstrates the promising potential to facilitate developing a more effective conversational recommendation system 1 .
Benchmarking Large Language Model Capabilities for Conditional Generation

Pre-trained large language models (PLMs) underlie most new developments in natural language processing.They have shifted the field from application-specific model pipelines to a single model that is adapted to a wide range of tasks.Autoregressive PLMs like GPT-3 or PaLM, alongside techniques like few-shot learning, have additionally shifted the output modality to generation instead of classification or regression.Despite their ubiquitous use, the generation quality of language models is rarely evaluated when these models are introduced.Additionally, it is unclear how existing generation tasks--while they can be used to compare systems at a high level-relate to the real world use cases for which people have been adopting them.In this work, we discuss how to adapt existing applicationspecific generation benchmarks to PLMs and provide an in-depth, empirical study of the limitations and capabilities of PLMs in natural language generation tasks along dimensions such as scale, architecture, input and output language.Our results show that PLMs differ in their applicability to different data regimes and their generalization to multiple languages and inform which PLMs to use for a given generation task setup.We share best practices to be taken into consideration when benchmarking generation capabilities during the development of upcoming PLMs.
Generating medically-accurate summaries of patient-provider dialogue: A multi-stage approach using large language models

A medical provider's summary of a patient visit serves several critical purposes, including clinical decision-making, facilitating handoffs between providers, and as a reference for the patient.An effective summary is required to be coherent and accurately capture all the medically relevant information in the dialogue, despite the complexity of patient-generated language.Even minor inaccuracies in visit summaries (for example, summarizing "patient does not have a fever" when a fever is present) can be detrimental to the outcome of care for the patient.This paper tackles the problem of medical conversation summarization by discretizing the task into several smaller dialogueunderstanding tasks that are sequentially built upon.First, we identify medical entities and their affirmations within the conversation to serve as building blocks.We study dynamically constructing few-shot prompts for tasks by conditioning on relevant patient information and use GPT-3 (Brown et al., 2020) as the backbone for our experiments.We also develop GPT-derived summarization metrics to measure performance against reference summaries quantitatively.Both our human evaluation study and metrics for medical correctness show that summaries generated using this approach are clinically accurate and outperform the baseline approach of summarizing the dialog in a zero-shot, single-prompt setting.
Decipherment of Lost Ancient Scripts as Combinatorial Optimisation Using Coupled Simulated Annealing

This paper presents a new approach to the ancient scripts decipherment problem based on combinatorial optimisation and coupled simulated annealing, an advanced non-convex optimisation procedure.Solutions are encoded by using k-permutations allowing for null, oneto-many, and many-to-one mappings between signs.The proposed system is able to produce enhanced results in cognate identification when compared to the state-of-the-art systems on standard evaluation benchmarks used in literature.
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding

State-of-the-art few-shot learning (FSL) methods leverage prompt-based fine-tuning to obtain remarkable results for natural language understanding (NLU) tasks.While much of the prior FSL methods focus on improving downstream task performance, there is a limited understanding of the adversarial robustness of such methods.In this work, we conduct an extensive study of several state-of-the-art FSL methods to assess their robustness to adversarial perturbations.To better understand the impact of various factors towards robustness (or the lack of it), we evaluate prompt-based FSL methods against fully fine-tuned models for aspects such as the use of unlabeled data, multiple prompts, number of few-shot examples, model size and type.Our results on six GLUE tasks indicate that compared to fully fine-tuned models, vanilla FSL methods lead to a notable relative drop in task performance (i.e., are less robust) in the face of adversarial perturbations.However, using (i) unlabeled data for promptbased FSL and (ii) multiple prompts flip the trend.We further demonstrate that increasing the number of few-shot examples and model size lead to increased adversarial robustness of vanilla FSL methods.Broadly, our work sheds light on the adversarial robustness evaluation of prompt-based FSL methods for NLU tasks.