Reviewerly/exHarmony huggingface dataset:
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

Large-scale language models have shown the ability to adapt to a new task via conditioning on a few demonstrations (i.e., in-context learning).Large-scale language models have shown the ability to adapt to a new task via conditioning on a few demonstrations (i.e., in-context learning).However, in the vision-language domain, most large-scale pre-trained visionlanguage (VL) models do not possess the ability to conduct in-context learning.How can we enable in-context learning for VL models?In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to the VL domain?Specifically, we first meta-trains a language model to perform in-context learning on NLP tasks (as in MetaICL); then we transfer this model to perform VL tasks by attaching a visual encoder.Our experiments suggest that indeed in-context learning ability can be transferred cross modalities: our model considerably improves the in-context learning capability on VL tasks and can even compensate for the size of the model significantly.On VQA, OK-VQA, and GQA, our method could outperform the baseline model while having ∼20 times fewer parameters.
A Multi-dimensional study on Bias in Vision-Language models

In recent years, joint Vision-Language (VL) models have increased in popularity and capability.Very few studies have attempted to investigate bias in VL models, even though it is a well-known issue in both individual modalities.This paper presents the first multi-dimensional analysis of bias in English VL models, focusing on gender, ethnicity, and age as dimensions.When subjects are input as images, pre-trained VL models complete a neutral template with a hurtful word 5% of the time, with higher percentages for female and young subjects.Bias presence in downstream models has been tested on Visual Question Answering.We developed a novel bias metric called the Vision-Language Association Test based on questions designed to elicit biased associations between stereotypical concepts and targets.Our findings demonstrate that pre-trained VL models contain biases that are perpetuated in downstream tasks.
Automatic Annotation of Direct Speech in Written French Narratives

The automatic annotation of direct speech (AADS) in written text has been often used in computational narrative understanding.Methods based on either rules or deep neural networks have been explored, in particular for English or German languages.Yet, for French, our target language, not many works exist.Our goal is to create a unified framework to design and evaluate AADS models in French.For this, we consolidated the largest-to-date French narrative dataset annotated with DS per word; we adapted various baselines for sequence labelling or from AADS in other languages; and we designed and conducted an extensive evaluation focused on generalisation.Results show that the task still requires substantial efforts and emphasise characteristics of each baseline.Although this framework could be improved, it is a step further to encourage more research on the topic.
Multilingual Summarization with Factual Consistency Evaluation

Abstractive summarization has enjoyed renewed interest in recent years, thanks to pretrained language models and the availability of large-scale datasets.Despite promising results, current models still suffer from generating factually inconsistent summaries, reducing their utility for real-world application.Several recent efforts attempt to address this by devising models that automatically detect factual inconsistencies in machine generated summaries.However, they focus exclusively on English, a language with abundant resources.In this work, we leverage factual consistency evaluation models to improve multilingual summarization.We explore two intuitive approaches to mitigate hallucinations based on the signal provided by a multilingual NLI model, namely data filtering and controlled generation.Experimental results in the 45 languages from the XLSum dataset show gains over strong baselines in both automatic and human evaluation.We release models and human judgements of summaries to foster progress towards more factually consistent multilingual summarization.
To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering

Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on general-purpose domains like Wikipedia.While some work has been investigating how well ODQA models perform when tested for out-of-domain (OOD) generalization, these studies have been conducted only under conservative shifts in data distribution and typically focus on a single component (i.e., retriever or reader) rather than an end-to-end system.This work proposes a more realistic endto-end domain shift evaluation setting covering five diverse domains.We not only find that endto-end models fail to generalize but that high retrieval scores often still yield poor answer prediction accuracy.To address these failures, we investigate several interventions, in the form of data augmentations, for improving model adaption and use our evaluation set to elucidate the relationship between the efficacy of an intervention scheme and the particular type of dataset shifts we consider.We propose a generalizability test that estimates the type of shift in a target dataset without training a model in the target domain and that the type of shift is predictive of which data augmentation schemes will be effective for domain adaption.Overall, we find that these interventions increase end-to-end performance by up to ∼24 points.* *This work was done while authors were at Google. Average F1 over all target datasetsAverage F1 over target datasets with specific shifts
PairSpanBERT: An Enhanced Language Model for Bridging Resolution

We present PAIRSPANBERT, a SPANBERTbased pre-trained model specialized for bridging resolution.PAIRSPANBERT is pre-trained with a novel objective that aims to learn the contexts in which two mentions are implicitly linked to each other from a large amount of data automatically generated either heuristically or via distance supervision with a knowledge graph.Despite the noise inherent in the automatically generated data, we achieve the best results reported to date on three evaluation datasets for bridging resolution when replacing SPANBERT with PAIRSPANBERT in a stateof-the-art resolver that jointly performs entity coreference resolution and bridging resolution.
PLAtE: A Large-scale Dataset for List Page Web Extraction

Recently, neural models have been leveraged to significantly improve the performance of information extraction from semi-structured websites.However, a barrier for continued progress is the small number of datasets large enough to train these models.In this work, we introduce the PLAtE (Pages of Lists Attribute Extraction) benchmark dataset as a challenging new web extraction task.PLAtE focuses on shopping data, specifically extractions from product review pages with multiple items encompassing the tasks of: (1) finding productlist segmentation boundaries and (2) extracting attributes for each product.PLAtE is composed of 52, 898 items collected from 6, 694 pages and 156, 014 attributes, making it the first largescale list page web extraction dataset.We use a multi-stage approach to collect and annotate the dataset and adapt three state-of-the-art web extraction models to the two tasks comparing their strengths and weaknesses both quantitatively and qualitatively.
f-Divergence Minimization for Sequence-Level Knowledge Distillation

Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one.It has gained increasing attention in the natural language processing community, driven by the demands of compressing evergrowing language models.In this work, we propose an f -DISTILL framework, which formulates sequence-level knowledge distillation as minimizing a generalized f -divergence function.We propose four distilling variants under our framework and show that existing SeqKD and ENGINE approaches are approximations of our f -DISTILL methods.We further derive step-wise decomposition for our f -DISTILL, reducing intractable sequence-level divergence to word-level losses that can be computed in a tractable manner.Experiments across four datasets show that our methods outperform existing KD approaches, and that our symmetric distilling losses can better force the student to learn from the teacher distribution. 1
Taxonomy of Problems in Lexical Semantics

Semantic tasks are rarely formally defined, and the exact relationship between them is an open question.We introduce a taxonomy that elucidates the connection between several problems in lexical semantics, including monolingual and cross-lingual variants.Our theoretical framework is based on the hypothesis of the equivalence of concept and meaning distinctions.Using algorithmic problem reductions, we demonstrate that all problems in the taxonomy can be reduced to word sense disambiguation (WSD), and that WSD itself can be reduced to some problems, making them theoretically equivalent.In addition, we carry out experiments that strongly support the soundness of the concept-meaning hypothesis, and the correctness of our reductions.
Consistent Text Categorization using Data Augmentation in e-Commerce

The categorization of massive e-Commerce data is a crucial, well-studied task, which is prevalent in industrial settings.In this work, we aim to improve an existing product categorization model that is already in use by a major web company, serving multiple applications.At its core, the product categorization model is a text classification model that takes a product title as an input and outputs the most suitable category out of thousands of available candidates.Upon a closer inspection, we found inconsistencies in the labeling of similar items.For example, minor modifications of the product title pertaining to colors or measurements majorly impacted the model's output.This phenomenon can negatively affect downstream recommendation or search applications, leading to a sub-optimal user experience.To address this issue, we propose a new framework for consistent text categorization.Our goal is to improve the model's consistency while maintaining its production-level performance.We use a semi-supervised approach for data augmentation and presents two different methods for utilizing unlabeled samples.One method relies directly on existing catalogs, while the other uses a generative model.We compare the pros and cons of each approach and present our experimental results.
BioNART: A Biomedical Non-AutoRegressive Transformer for Natural Language Generation

We propose a novel Biomedical domainspecific Non-AutoRegressive Transformer model for natural language generation: BioNART.Our BioNART is based on an encoder-decoder model, and both encoder and decoder are compatible with widely used BERT architecture, which allows benefiting from publicly available pre-trained biomedical language model checkpoints.We performed additional pre-training and finetuned BioNART on biomedical summarization and doctor-patient dialogue tasks in English.Experimental results show that our BioNART achieves about 94% of the ROUGE score to the pre-trained autoregressive model while realizing an 18 times faster inference speed on the iCliniq dataset.
Decker: Double Check with Heterogeneous Knowledge for Commonsense Fact Verification

Commonsense fact verification, as a challenging branch of commonsense questionanswering (QA), aims to verify through facts whether a given commonsense claim is correct or not.Answering commonsense questions necessitates a combination of knowledge from various levels.However, existing studies primarily rest on grasping either unstructured evidence or potential reasoning paths from structured knowledge bases, yet failing to exploit the benefits of heterogeneous knowledge simultaneously.In light of this, we propose DECKER, a commonsense fact verification model that is capable of bridging heterogeneous knowledge by uncovering latent relationships between structured and unstructured knowledge.Experimental results on two commonsense fact verification benchmark datasets, CSQA2.0 and CREAK demonstrate the effectiveness of our DECKER and further analysis verifies its capability to seize more precious information through reasoning.The official implementation of DECKER is available at https://github.com/Anni-Zou/Decker.
ITTC at SemEval 2023-Task 7: Document Retrieval and Sentence Similarity for Evidence Retrieval in Clinical Trial Data

This paper describes the submissions of the Natural Language Processing (NLP) team from the Australian Research Council Industrial Transformation Training Centre (ITTC) for Cognitive Computing in Medical Technologies to the SemEval 2023 Task 7, i.e., multi-evidence natural language inference for clinical trial data (NLI4CT).More specifically, we were working on subtask 2 whose objective is to identify the relevant parts of the premise from clinical trial report that justify the truth of information in the statement.We approach the evidence retrieval problem as a document retrieval and sentence similarity task.Our results show that the task poses some challenges which involve dealing with complex sentences and implicit evidences.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations

Formulating selective information needs results in queries that implicitly specify set operations, such as intersection, union, and difference.For instance, one might search for "shorebirds that are not sandpipers" or "science-fiction films shot in England".To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of entities corresponding to Wikipedia documents.The dataset challenges models to match multiple constraints mentioned in queries with corresponding evidence in documents and correctly perform various set operations.The dataset is constructed semi-automatically using Wikipedia category names.Queries are automatically composed from individual categories, then paraphrased and further validated for naturalness and fluency by crowdworkers.Crowdworkers also assess the relevance of entities based on their documents and highlight attribution of query constraints to spans of document text.We analyze several modern retrieval systems, finding that they often struggle on such queries.Queries involving negation and conjunction are particularly challenging and systems are further challenged with combinations of these operations. 1 * Work done during an internship at Google. 1 The dataset is available at https://github.
Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models

Existing dialogue models may encounter scenarios which are not well-represented in the training data, and as a result generate responses that are unnatural, inappropriate, or unhelpful.We propose the "Ask an Expert" framework in which the model is trained with access to an "expert" which it can consult at each turn.Advice is solicited via a structured dialogue with the expert, and the model is optimized to selectively utilize (or ignore) it given the context and dialogue history.In this work the expert takes the form of an LLM.We evaluate this framework in a mental health support domain, where the structure of the expert conversation is outlined by pre-specified prompts which reflect a reasoning strategy taught to practitioners in the field.Blenderbot models utilizing "Ask an Expert" show quality improvements across all expert sizes, including those with fewer parameters than the dialogue model itself.Our best model provides a ∼ 10% improvement over baselines, approaching human-level scores on "engingingness" and "helpfulness" metrics.
The Hidden Folk: Linguistic Properties Encoded in Multilingual Contextual Character Representations

To gain a better understanding of the linguistic information encoded in character-based language models, we probe the multilingual contextual CANINE model.We design a range of phonetic probing tasks in six Nordic languages, including Faroese as an additional zero-shot instance.We observe that some phonetic information is indeed encoded in the character representations, as consonants and vowels can be well distinguished using a linear classifier.Furthermore, results for the Danish and Norwegian language seem to be worse for the consonant/vowel distinction in comparison to other languages.The information encoded in these representations can also be learned in a zero-shot scenario, as Faroese shows a reasonably good performance in the same vowel/consonant distinction task.
VoteTRANS: Detecting Adversarial Text without Training by Voting on Hard Labels of Transformations

Adversarial attacks reveal serious flaws in deep learning models.More dangerously, these attacks preserve the original meaning and escape human recognition.Existing methods for detecting these attacks need to be trained using original/adversarial data.In this paper, we propose detection without training by voting on hard labels from predictions of transformations, namely, VoteTRANS.Specifically, VoteTRANS detects adversarial text by comparing the hard labels of input text and its transformation.The evaluation demonstrates that VoteTRANS effectively detects adversarial text across various state-of-the-art attacks, models, and datasets.
Transferable and Efficient: Unifying Dynamic Multi-Domain Product Categorization

As e-commerce platforms develop different business lines, a special but challenging product categorization scenario emerges, where there are multiple domain-specific category taxonomies and each of them evolves dynamically over time.In order to unify the categorization process and ensure efficiency, we propose a two-stage taxonomy-agnostic framework that relies solely on calculating the semantic relatedness between product titles and category names in the vector space.To further enhance domain transferability and better exploit cross-domain data, we design two plugin modules: a heuristic mapping scorer and a pretrained contrastive ranking module with the help of "meta concepts", which represent keyword knowledge shared across domains.Comprehensive offline experiments show that our method outperforms strong baselines on three dynamic multi-domain product categorization (DMPC) tasks, and online experiments reconfirm its efficacy with a 5% increase on seasonal purchase revenue.Related datasets are released 1 .
SkillQG: Learning to Generate Question for Reading Comprehension Assessment

We present SkillQG: a question generation framework with controllable comprehension types for assessing and improving machine reading comprehension models. Existing question generation systems widely differentiate questions by literal information such as question words and answer types to generate semantically relevant questions for a given context.However, they rarely consider the comprehension nature of questions, i.e., the different comprehension capabilities embodied by different questions.In comparison, our SkillQG is able to tailor a fine-grained assessment and improvement to the capabilities of questions answering models built on it.Specifically, we first frame the comprehension type of questions based on a hierarchical skill-based schema.We then formulate SkillQG as a skill-conditioned question generator.Furthermore, to improve the controllability of generation, we augment the input text with skill-specific question focus and knowledge, which are constructed by iteratively prompting the pre-trained language models.Empirical results demonstrate that SkillQG outperforms baselines in terms of quality, relevance, and skill-controllability while showing a promising performance boost in downstream question answering task.
Domain Aligned Prefix Averaging for Domain Generalization in Abstractive Summarization

Domain generalization is hitherto an underexplored area applied in abstractive summarization.Moreover, most existing works on domain generalization have sophisticated training algorithms.In this paper, we propose a lightweight, weight averaging based, Domain Aligned Prefix Averaging approach to domain generalization for abstractive summarization.Given a number of source domains, our method first trains a prefix for each one of them.These source prefixes generate summaries for a small number of target domain documents.The similarity of the generated summaries to their corresponding documents is used for calculating weights required to average source prefixes.In DAPA, prefix tuning allows for lightweight finetuning, and weight averaging allows for the computationally efficient addition of new source domains.When evaluated on four diverse summarization domains, DAPA shows comparable or better performance against the baselines, demonstrating the effectiveness of its prefix averaging scheme 1 .
SConE: Simplified Cone Embeddings with Symbolic Operators for Complex Logical Queries

Geometric representation of query embeddings (using points, particles, rectangles and cones) can effectively achieve the task of answering complex logical queries expressed in first-order logic (FOL) form over knowledge graphs, allowing intuitive encodings. However, current geometric-based methods depend on the neural approach to model FOL operators (conjunction, disjunction and negation), which are not easily explainable with considerable computation cost. We overcome this challenge by introducing a symbolic modeling approach for the FOL operators, emphasizing the direct calculation of the intersection between geometric shapes, particularly sector-cones in the embedding space, to model the conjunction operator. This approach reduces the computation cost as a non-neural approach is involved in the core logic operators. Moreover, we propose to accelerate the learning in the relation projection operator using the neural approach to emphasize the essential role of this operator in all query structures. Although empirical evidence for explainability is challenging, our approach demonstrates a significant improvement in answering complex logical queries (both non-negative and negative FOL forms) over previous geometric-based models.
Causality-aware Concept Extraction based on Knowledge-guided Prompting

Concepts benefit natural language understanding but are far from complete in existing knowledge graphs (KGs).Recently, pre-trained language models (PLMs) have been widely used in text-based concept extraction (CE).However, PLMs tend to mine the co-occurrence associations from massive corpus as pre-trained knowledge rather than the real causal effect between tokens.As a result, the pre-trained knowledge confounds PLMs to extract biased concepts based on spurious co-occurrence correlations, inevitably resulting in low precision.In this paper, through the lens of a Structural Causal Model (SCM), we propose equipping the PLM-based extractor with a knowledgeguided prompt as an intervention to alleviate concept bias.The prompt adopts the topic of the given entity from the existing knowledge in KGs to mitigate the spurious co-occurrence correlations between entities and biased concepts.Our extensive experiments on representative multilingual KG datasets justify that our proposed prompt can effectively alleviate concept bias and improve the performance of PLM-based CE models.The code has been released on https://github.com/siyuyuan/KPCE.