Reviewerly/exHarmony huggingface dataset:
Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning

Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with.A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step.We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting.Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval.Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set.We show this methodology better emulates realworld requirements than artificial challenges. 1
On Complementarity Objectives for Hybrid Retrieval

Dense retrieval has shown promising results in various information retrieval tasks, and hybrid retrieval, combined with the strength of sparse retrieval, has also been actively studied.A key challenge in hybrid retrieval is to make sparse and dense complementary to each other.Existing models have focused on dense models to capture "residual" features neglected in the sparse models.Our key distinction is to show how this notion of residual complementarity is limited, and propose a new objective, denoted as RoC (Ratio of Complementarity), which captures a fuller notion of complementarity.We propose a two-level orthogonality designed to improve RoC, then show that the improved RoC of our model, in turn, improves the performance of hybrid retrieval.Our method outperforms all state-of-the-art methods on three representative IR benchmarks: MSMARCO-Passage, Natural Questions, and TREC Ro-bust04, with statistical significance.Our finding is also consistent in various adversarial settings.
Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23

This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task.Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50).We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT.After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation.Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets.Our best single model obtains 31.2BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023.
Saama AI Research at SemEval-2023 Task 7: Exploring the Capabilities of Flan-T5 for Multi-evidence Natural Language Inference in Clinical Trial Data

The goal of the NLI4CT task is to build a Natural Language Inference system for Clinical Trial Reports that will be used for evidence interpretation and retrieval.Large Language models have demonstrated state-of-the-art performance in various natural language processing tasks across multiple domains.We suggest using an instruction-finetuned Large Language Models (LLMs) to take on this particular task in light of these developments.We have evaluated the publicly available LLMs under zeroshot setting, and finetuned the best performing Flan-T5 model for this task.On the leaderboard, our system ranked second, with an F1 Score of 0.834 on the official test set.
A Static Evaluation of Code Completion by Large Language Models

Large language models trained on code have shown great potential to increase productivity of software developers.Several executionbased benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems.Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost.On the contrary, static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models.In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees.Compared with executionbased evaluation, our method is not only more efficient, but also applicable to code in the wild.For experiments, we collect code context from open source repos to generate one million function bodies using public models.Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models.Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.
CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding

Story generation and understanding-as with all NLG/NLU tasks-has seen a surge in neurosymbolic work.Researchers have recognized that, while large language models (LLMs) have tremendous utility, they can be augmented with symbolic means to be even better and to make up for many flaws that neural networks have.However, symbolic methods are extremely costly in terms of the amount of time and expertise needed to create them.In this work, we capitalize on state-of-the-art Code-LLMs, such as Codex, to bootstrap the use of symbolic methods for tracking the state of stories and aiding in story understanding.We show that our CoRRPUS system and abstracted prompting procedures can beat current stateof-the-art structured LLM techniques on preexisting story understanding tasks (bAbI Task 2 and Re 3 ) with minimal hand engineering.This work highlights the usefulness of code-based symbolic representations for enabling LLMs to better perform story reasoning tasks.
BpHigh at SemEval-2023 Task 7: Can Fine-tuned Cross-encoders Outperform GPT-3.5 in NLI Tasks on Clinical Trial Data?

Many nations and organizations have begun collecting and storing clinical trial records for storage and analytical purposes so that medical and clinical practitioners can refer to them on a centralized database over the internet and stay updated with the current clinical information.Clinical trial records have increased significantly, making it difficult for many medical and clinical practitioners to stay updated with the latest information.To help and support medical and clinical practitioners, there is a need to build intelligent systems that can update them with the latest information in a byte-sized condensed format and, at the same time, leverage their understanding capabilities to help them make decisions.This paper describes our contribution to SemEval 2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT).Our results show that there is still a need to build domainspecific models as smaller transformer-based models can be finetuned on that data and outperform foundational large language models like GPT-3.5.We also demonstrate how the performance of GPT-3.5 can be increased using few-shot prompting by leveraging the semantic similarity of the text samples and the few-shot train snippets.We will also release our code and models on open-source hosting platforms, GitHub and HuggingFace.
Controllable Conversation Generation with Conversation Structures via Diffusion Models

Generating coherent conversation is an important and challenging long text generation task, as it has various applications such as daily entertainment, education, or building conversational AI to facilitate human-computer interaction.However, current generation models often fail to effectively utilize rich linguistic and world knowledge to generate conversations just like humans.In this work, we introduce a novel conversation generation framework to effectively incorporate human knowledge and conversation structures with both controllability and interpretability for better conversation generation.Specifically, we first generate the prototype conversations from short descriptions.We then gradually and strategically incorporate different levels of conversation structures including the action triples, dialogue acts, and discourse relations via diffusion models to directly edit the prototype conversations.We demonstrate the effectiveness of our framework through experiments on two datasets by comparing our method with the state-of-the-art baseline models 1 .
Eliciting Affective Events from Language Models by Multiple View Co-prompting

Prior research on affective event classification showed that exploiting weakly labeled data for training can improve model performance.In this work, we propose a simpler and more effective approach for generating training data by automatically acquiring and labeling affective events with Multiple View Co-prompting, which leverages two language model prompts that provide independent views of an event.The approach starts with a modest amount of gold data and prompts pre-trained language models to generate new events.Next, information about the probable affective polarity of each event is collected from two complementary language model prompts and jointly used to assign polarity labels.Experimental results on two datasets show that the newly acquired events improve a state-of-the-art affective event classifier.We also present analyses which show that using multiple views produces polarity labels of higher quality than either view on its own.
Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL

The task of text-to-SQL aims to convert a natural language question into its corresponding SQL query within the context of relational tables.Existing text-to-SQL parsers generate a "plausible" SQL query for an arbitrary user question, thereby failing to correctly handle problematic user questions.To formalize this problem, we conduct a preliminary study on the observed ambiguous and unanswerable cases in text-to-SQL and summarize them into 6 feature categories.Correspondingly, we identify the causes behind each category and propose requirements for handling ambiguous and unanswerable questions.Following this study, we propose a simple yet effective counterfactual example generation approach that automatically produces ambiguous and unanswerable text-to-SQL examples.Furthermore, we propose a weakly supervised DTE (Detecting-Then-Explaining) model for error detection, localization, and explanation.Experimental results show that our model achieves the best result on both real-world examples and generated examples compared with various baselines.We release our data and code at: https://github.com/wbbeyourself/DTE.
Towards Higher Pareto Frontier in Multilingual Machine Translation

Multilingual neural machine translation has witnessed remarkable progress in recent years.However, the long-tailed distribution of multilingual corpora poses a challenge of Pareto optimization, i.e., optimizing for some languages may come at the cost of degrading the performance of others.Existing balancing training strategies are equivalent to a series of Pareto optimal solutions, which trade off on a Pareto frontier 1 .In this work, we propose a new training framework, Pareto Mutual Distillation (Pareto-MD), towards pushing the Pareto frontier outwards rather than making trade-offs.Specifically, Pareto-MD collaboratively trains two Pareto optimal solutions that favor different languages and allows them to learn from the strengths of each other via knowledge distillation.Furthermore, we introduce a novel strategy to enable stronger communication between Pareto optimal solutions and broaden the applicability of our approach.Experimental results on the widely-used WMT and TED datasets show that our method significantly pushes the Pareto frontier and outperforms baselines by up to +2.46 BLEU 2 .
OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models

We conduct a thorough investigation into the reasoning capabilities of Large Language Models (LLMs), focusing specifically on the Open Pretrained Transformers (OPT) models as a representative of such models. Our study entails finetuning three different sizes of OPT on a carefully curated reasoning corpus, resulting in two sets of finetuned models: OPT-R, finetuned without explanations, and OPT-RE, finetuned with explanations. We then evaluate all models on 57 out-of-domain tasks drawn from the Super-NaturalInstructions benchmark, covering 26 distinct reasoning skills, utilizing three prompting techniques. Through a comprehensive grid of 27 configurations and 6,156 test evaluations, we investigate the dimensions of finetuning, prompting, and scale to understand the role of explanations on different reasoning skills. Our findings reveal that having explanations in the fewshot exemplar has no significant impact on the model’s performance when the model is finetuned, while positively affecting the non-finetuned counterpart. Moreover, we observe a slight yet consistent increase in classification accuracy as we incorporate explanations during prompting and finetuning, respectively. Finally, we offer insights on which reasoning skills benefit the most from incorporating explanations during finetuning and prompting, such as Numerical (+20.4%) and Analogical (+13.9%) reasoning, as well as skills that exhibit negligible or negative effects.
ClaimDiff: Comparing and Contrasting Claims on Contentious Issues

With the growing importance of detecting misinformation, many studies have focused on verifying factual claims by retrieving evidence.However, canonical fact verification tasks do not apply to catching subtle differences in factually consistent claims, which might still bias the readers, especially on contentious political or economic issues.Our underlying assumption is that among the trusted sources, one's argument is not necessarily more true than the other, requiring comparison rather than verification.In this study, we propose ClaimDiff, a novel dataset that primarily focuses on comparing the nuance between claim pairs.In ClaimDiff, we provide 2,941 annotated claim pairs from 268 news articles.We observe that while humans are capable of detecting the nuances between claims, strong baselines struggle to detect them, showing over a 19% absolute gap with the humans.We hope this initial study could help readers to gain an unbiased grasp of contentious issues through machine-aided comparison.
Robust Natural Language Understanding with Residual Attention Debiasing

Natural language understanding (NLU) models often suffer from unintended dataset biases.Among bias mitigation methods, ensemblebased debiasing methods, especially productof-experts (PoE), have stood out for their impressive empirical success.However, previous ensemble-based debiasing methods typically apply debiasing on top-level logits without directly addressing biased attention patterns.Attention serves as the main media of feature interaction and aggregation in PLMs and plays a crucial role in providing robust prediction.In this paper, we propose REsidual Attention Debiasing (READ), an end-to-end debiasing method that mitigates unintended biases from attention.Experiments on three NLU tasks show that READ significantly improves the performance of BERT-based models on OOD data with shortcuts removed, including +12.9% accuracy on HANS, +11.0%accuracy on FEVER-Symmetric, and +2.7% F1 on PAWS.Detailed analyses demonstrate the crucial role of unbiased attention in robust NLU models and that READ effectively mitigates biases in attention. 1
GVdoc - Graph-based Visual DOcument Classification

The robustness of a model for real-world deployment is decided by how well it performs on unseen data and distinguishes between in-domain and out-of-domain samples.Visual document classifiers have shown impressive performance on in-distribution test sets.However, they tend to have a hard time correctly classifying and differentiating out-ofdistribution examples.Image-based classifiers lack the text component, whereas multimodality transformer-based models face the token serialization problem in visual documents due to their diverse layouts.They also require a lot of computing power during inference, making them impractical for many real-world applications.We propose, GVdoc, a graph-based document classification model that addresses both of these challenges.Our approach generates a document graph based on its layout, and then trains a graph neural network to learn node and graph embeddings.Through experiments, we show that our model, even with fewer parameters, outperforms state-of-the-art models on out-of-distribution data while retaining comparable performance on the in-distribution test set.
Plug-and-Play Document Modules for Pre-trained Models

Large-scale pre-trained models (PTMs) have been widely used in document-oriented NLP tasks, such as question answering.However, the encoding-task coupling requirement results in the repeated encoding of the same documents for different tasks and queries, which is highly computationally inefficient.To this end, we target to decouple document encoding from downstream tasks, and propose to represent each document as a plug-and-play document module, i.e., a document plugin, for PTMs (PlugD).By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simultaneously encode documents and input queries using task-specific encoders.Extensive experiments on 8 datasets of 4 typical NLP tasks show that PlugD enables models to encode documents once and for all across different scenarios.Especially, PlugD can save 69% computational costs while achieving comparable performance to state-of-the-art encoding-task coupling methods.Additionally, we show that PlugD can serve as an effective post-processing way to inject knowledge into task-specific models, improving model performance without any additional model training.Our code and checkpoints can be found in https://github.com/thunlp/Document-Plugin.
Exploiting Commonsense Knowledge about Objects for Visual Activity Recognition

Situation recognition is the task of recognizing the activity depicted in an image, including the people and objects involved.Previous models for this task typically train a classifier to identify the activity using a backbone image feature extractor.We propose that commonsense knowledge about the objects depicted in an image can also be a valuable source of information for activity identification.Previous NLP research has argued that knowledge about the prototypical functions of physical objects is important for language understanding, and NLP techniques have been developed to acquire this knowledge.Our work investigates whether this prototypical function knowledge can also be beneficial for visual situation recognition.We build a framework that incorporates this type of commonsense knowledge in a transformer-based model that is trained to predict the action verb for situation recognition.Our experimental results show that adding prototypical function knowledge about physical objects does improve performance for the visual activity recognition task.
NL4IA at SemEval-2023 Task 3: A Comparison of Sequence Classification and Token Classification to Detect Persuasive Techniques

The following system description presents our approach to the detection of persuasion techniques in online news.The given task has been framed as a multi-label classification problem.In a multi-label classification problem, each input chunk-in this case paragraph-is assigned one of several class labels.Span level annotations were also provided.In order to assign class labels to the given documents, we opted for RoBERTa (A Robustly Optimized BERT Pretraining Approach) for both approaches-sequence and token classification.Starting off with a pre-trained model for language representation, we fine-tuned this model on the given classification task with the provided annotated data in supervised training steps.
A Holistic Approach to Reference-Free Evaluation of Machine Translation

Traditional machine translation evaluation relies on references written by humans.While reference-free evaluation gets rid of the constraints of labor-intensive annotations, it can pivot easily to new domains and is more scalable.In this paper, we propose a referencefree evaluation approach that characterizes evaluation as two aspects: (1) fluency: how well the candidate translation conforms to normal human language usage; (2) faithfulness: how well the candidate translation reflects the source data.We further split the faithfulness into word-level and sentence-level.Extensive experiments spanning WMT18/19/21 Metrics segment-level daRR and MQM datasets demonstrate that our proposed reference-free approach, ReFreeEval, outperforms SOTA reference-free metrics like YiSi-2, SentSim and BERTScore-MKD in most language directions.The code can be found at ReFreeEval Repo 1 .
Revisiting Pathologies of Neural Models under Input Reduction

We revisit the question of why neural models tend to produce high-confidence predictions on inputs that appear nonsensical to humans.Previous work has suggested that the models fail to assign low probabilities to such inputs due to model overconfidence.We evaluate various regularization methods on fact verification benchmarks and find that this problem persists even with well-calibrated or underconfident models, suggesting that overconfidence is not the only underlying cause.We also find that regularizing the models with reduced examples helps improve interpretability but comes with the cost of miscalibration.We show that although these reduced examples are incomprehensible to humans, they can contain valid statistical patterns in the dataset utilized by the model. 1
On the Compositional Generalization in Versatile Open-domain Dialogue

Previous research has demonstrated the potential of multi-task learning to foster a conversational agent's ability to acquire a variety of skills.However, these approaches either suffer from interference among different datasets (also known as negative transfer), or fail to effectively reuse knowledge and skills learned from other datasets.In contrast to previous works, we develop a sparsely activated modular network: (1) We propose a wellrounded set of operators and instantiate each operator with an independent module; (2) We formulate dialogue generation as the execution of a generated programme which recursively composes and assembles modules.Extensive experiments on 9 datasets verify the efficacy of our methods through automatic evaluation and human evaluation.Notably, our model outperforms state-of-the-art supervised approaches on 4 datasets with only 10% training data thanks to the modular architecture and multi-task learning.
In-sample Curriculum Learning by Sequence Completion for Natural Language Generation

Curriculum learning has shown promising improvements in multiple domains by training machine learning models from easy samples to hard ones.Previous works which either design rules or train models for scoring the difficulty highly rely on task-specific expertise, and cannot generalize.Inspired by the "easy-to-hard" intuition, we propose to do in-sample curriculum learning for natural language generation tasks.Our learning strategy starts training the model to generate the last few words, i.e., do sequence completion, and gradually extends to generate the whole output sequence.Comprehensive experiments show that it generalizes well to different tasks and achieves significant improvements over strong baselines.