Reviewerly/exHarmony huggingface dataset:
When to Read Documents or QA History: On Unified and Selective Open-domain QA

This paper studies the problem of open-domain question answering, with the aim of answering a diverse range of questions leveraging knowledge resources.Two types of sources, QApair and document corpora, have been actively leveraged with the following complementary strength.The former is highly precise when the paraphrase of given question q was seen and answered during training, often posed as a retrieval problem, while the latter generalizes better for unseen questions.A natural follow-up is thus leveraging both models, while a naive pipelining or integration approaches have failed to bring additional gains over either model alone.Our distinction is interpreting the problem as calibration, which estimates the confidence of predicted answers as an indicator to decide when to use a document or QA-pair corpus.The effectiveness of our method was validated on widely adopted benchmarks such as Natural Questions and TriviaQA.
Referring to Screen Texts with Voice Assistants

Voice assistants help users make phone calls, send messages, create events, navigate and do a lot more.However assistants have limited capacity to understand their users' context.In this work, we aim to take a step in this direction.Our work dives into a new experience for users to refer to phone numbers, addresses, email addresses, urls, and dates on their phone screens.Our focus lies in reference understanding, which becomes particularly interesting when multiple similar texts are present on screen, similar to visual grounding.We collect a dataset and propose a lightweight general purpose model for this novel experience.Due to the high cost of consuming pixels directly, our system is designed to rely on the extracted text from the UI.Our model is modular, thus offering flexibility, improved interpretability, and efficient runtime memory utilization.
Efficient and Interpretable Compressive Text Summarisation with Unsupervised Dual-Agent Reinforcement Learning

Recently, compressive text summarisation offers a balance between the conciseness issue of extractive summarisation and the factual hallucination issue of abstractive summarisation.However, most existing compressive summarisation methods are supervised, relying on the expensive effort of creating a new training dataset with corresponding compressive summaries.In this paper, we propose an efficient and interpretable compressive summarisation method that utilises unsupervised dual-agent reinforcement learning to optimise a summary's semantic coverage and fluency by simulating human judgment on summarisation quality.Our model consists of an extractor agent and a compressor agent, and both agents have a multi-head attentional pointerbased structure.The extractor agent first chooses salient sentences from a document, and then the compressor agent compresses these extracted sentences by selecting salient words to form a summary without using reference summaries to compute the summary reward.To our best knowledge, this is the first work on unsupervised compressive summarisation.Experimental results on three widely used datasets (e.g., Newsroom, CNN/DM, and XSum) show that our model achieves promising performance and a significant improvement on Newsroom in terms of the ROUGE metric, as well as interpretability of semantic coverage of summarisation results. 1
AdamR at SemEval-2023 Task 10: Solving the Class Imbalance Problem in Sexism Detection with Ensemble Learning

The Explainable Detection of Online Sexism task presents the problem of explainable sexism detection through fine-grained categorisation of sexist cases with three subtasks.Our team experimented with different ways to combat class imbalance throughout the tasks using data augmentation and loss alteration techniques.We tackled the challenge by utilising ensembles of Transformer models trained on different datasets, which are tested to find the balance between performance and interpretability.This solution ranked us in the top 40% of teams for each of the tracks.
Stephen Colbert at SemEval-2023 Task 5: Using Markup for Classifying Clickbait

For SemEval-2023 Task 5, we have submitted three DeBERTaV3 LARGE models to tackle the first subtask, classifying spoiler types (passage, phrase, multi) of clickbait web articles.The choice of basic parameters like sequence length with BERT BASE uncased and further approaches were then tested with DeBERTaV3 BASE only moving the most promising ones to DeBERTaV3 LARGE .Our research showed that information-placement on webpages is often optimized regarding e.g.adplacement.Those informations are usually described within the webpages markup which is why we conducted an approach that takes this into account.Overall we could not manage to beat the baseline, which we lead down to three reasons: First we only crawled markup for Huffington Post articles, extracting only -and <a>-tags which will not cover enough aspects of a webpages design.Second Huffington Post articles are overrepresented in the given dataset, which, third, shows an imbalance towards the spoiler tags.We highly suggest re-annotating the given dataset to use markupoptimized models like MarkupLM or TIE and to clear it from embedded articles like "Yahoo" or archives like "archive.is"or "web.archive" to avoid noise.Also, the imbalance should be tackled by adding articles from sources other than Huffington Post, considering that also multi-tagged entries should be balanced towards passage-and phrase-tagged ones.
Towards Reference-free Text Simplification Evaluation with a BERT Siamese Network Architecture

Text simplification (TS) aims to modify sentences to make their both content and structure easier to understand.Traditional n-gram matching-based TS evaluation metrics heavily rely on the exact token match and humanannotated simplified sentences.In this paper, we present a novel neural-network-based reference-free TS metric BETS that leverages pre-trained contextualized language representation models and large-scale paraphrasing datasets to evaluate simplicity and meaning preservation.We show that our metric, without collecting any costly human simplification reference, correlates better than existing metrics with human judgments for the quality of both overall simplification (+7.7%) and its key aspects, i.e., comparative simplicity (+11.2%) and meaning preservation (+9.2%).
Label-Aware Hyperbolic Embeddings for Fine-grained Emotion Classification

Fine-grained emotion classification (FEC) is a challenging task.Specifically, FEC needs to handle subtle nuance between labels, which can be complex and confusing.Most existing models only address text classification problem in the euclidean space, which we believe may not be the optimal solution as labels of close semantic (e.g., afraid and terrified) may not be differentiated in such space, which harms the performance.In this paper, we propose Hy-pEmo, a novel framework that can integrate hyperbolic embeddings to improve the FEC task.First, we learn label embeddings in the hyperbolic space to better capture their hierarchical structure, and then our model projects contextualized representations to the hyperbolic space to compute the distance between samples and labels.Experimental results show that incorporating such distance to weight cross entropy loss substantially improves the performance with significantly higher efficiency.We evaluate our proposed model on two benchmark datasets and found 4.8% relative improvement compared to the previous state of the art with 43.2% fewer parameters and 76.9% less training time.Code is available at https: //github.com/dinobby/HypEmo.
Enhancing Few-shot Cross-lingual Transfer with Target Language Peculiar Examples

Few-shot cross-lingual transfer, fine-tuning Multilingual Masked Language Model (MMLM) with source language labeled data and a small amount of target language labeled data, provides excellent performance in the target language.However, if no labeled data in the target language are available, they need to be created through human annotations.In this study, we devise a metric to select annotation candidates from an unlabeled data pool that efficiently enhance accuracy for few-shot cross-lingual transfer.It is known that training a model with hard examples is important to improve the model’s performance.Therefore, we first identify examples that MMLM cannot solve in a zero-shot cross-lingual transfer setting and demonstrate that it is hard to predict peculiar examples in the target language, i.e., the examples distant from the source language examples in cross-lingual semantic space of the MMLM.We then choose high peculiarity examples as annotation candidates and perform few-shot cross-lingual transfer.In comprehensive experiments with 20 languages and 6 tasks, we demonstrate that the high peculiarity examples improve the target language accuracy compared to other candidate selection methods proposed in previous studies.
KNSE: A Knowledge-aware Natural Language Inference Framework for Dialogue Symptom Status Recognition

Symptom diagnosis in medical conversations aims to correctly extract both symptom entities and their status from the doctor-patient dialogue.In this paper, we propose a novel framework called KNSE for symptom status recognition (SSR), where the SSR is formulated as a natural language inference (NLI) task.For each mentioned symptom in a dialogue window, we first generate knowledge about the symptom and hypothesis about status of the symptom, to form a (premise, knowledge, hypothesis) triplet.The BERT model is then used to encode the triplet, which is further processed by modules including utterance aggregation, self-attention, cross-attention, and GRU to predict the symptom status.Benefiting from the NLI formalization, the proposed framework can encode more informative prior knowledge to better localize and track symptom status, which can effectively improve the performance of symptom status recognition.Preliminary experiments on Chinese medical dialogue datasets show that KNSE outperforms previous competitive baselines and has advantages in cross-disease and cross-symptom scenarios.
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks

Societal biases are reflected in large pre-trained language models and their fine-tuned versions on downstream tasks.Common in-processing bias mitigation approaches, such as adversarial training and mutual information removal, introduce additional optimization criteria, and update the model to reach a new debiased state.However, in practice, end-users and practitioners might prefer to switch back to the original model, or apply debiasing only on a specific subset of protected attributes.To enable this, we propose a novel modular bias mitigation approach, consisting of stand-alone highly sparse debiasing subnetworks, where each debiasing module can be integrated into the core model on-demand at inference time.Our approach draws from the concept of diff pruning, and proposes a novel training regime adaptable to various representation disentanglement optimizations.We conduct experiments on three classification tasks with gender, race, and age as protected attributes.The results show that our modular approach, while maintaining task performance, improves (or at least remains onpar with) the effectiveness of bias mitigation in comparison with baseline finetuning.Particularly on a two-attribute dataset, our approach with separately learned debiasing subnetworks shows effective utilization of either or both the subnetworks for selective bias mitigation. … …
Bi-level Finetuning with Task-dependent Similarity Structure for Low-resource Training

Training a large language model in lowresource settings is challenging since they are susceptible to overfitting with limited generalization abilities.Previous work addresses this issue by approaches such as tunable parameters reduction or data augmentation.However, they either limit the trained models' expressiveness or rely on task-independent knowledge.In this paper, we propose the Bilevel Finetuning with Task-dependent Similarity Structure framework where all parameters, including the embeddings for unseen tokens, are finetuned with task-dependent information from the training data only.In this framework, a task-dependent similarity structure is learned in a data-driven fashion, which in turn is used to compose soft embeddings from conventional embeddings to be used in training to update all parameters.In order to learn the similarity structure and model parameters, we propose a bi-level optimization algorithm with two stages-search and finetuneto ensure successful learning.Results of experiments on several classification datasets in low-resource scenarios demonstrate that models trained with our method outperform strong baselines.Ablation experiments further support the effectiveness of different components in our framework.Code is available at https://github.com/Sai-Ashish/BFTSS.* Equal Contribution 1 Our work is conducted in low resource scenarios that comprise a few hundred instances of data.
Scaling Laws for BERT in Low-Resource Settings

Large language models are very resource intensive, both financially and environmentally, and require an amount of training data which is simply unobtainable for the majority of NLP practitioners.Previous work has researched the scaling laws of such models, but optimal ratios of model parameters, dataset size, and computation costs focused on the large scale.In contrast, we analyze the effect those variables have on the performance of language models in constrained settings, by building three lightweight BERT models (16M/51M/124M parameters) trained over a set of small corpora (5M/25M/125M words).We experiment on four languages of different linguistic characteristics (Basque, Spanish, Swahili and Finnish), and evaluate the models on MLM and several NLU tasks.We conclude that the power laws for parameters, data and compute for lowresource settings differ from the optimal scaling laws previously inferred, and data requirements should be higher.Our insights are consistent across all the languages we study, as well as across the MLM and downstream tasks.Furthermore, we experimentally establish when the cost of using a Transformer-based approach is worth taking, instead of favouring other computationally lighter solutions.
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises

For many real-world applications, the usergenerated inputs usually contain various noises due to speech recognition errors caused by linguistic variations 1 or typographical errors (typos).Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness.However, little study has been done to construct such benchmarks for Chinese, where various languagespecific input noises happen in the real world.In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises.READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input.We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises.We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation.As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks.The source code and dataset can be obtained from https://github.com/thunlp/READIN.
Extracting Text Representations for Terms and Phrases in Technical Domains

Extracting dense representations for terms and phrases is a task of great importance for knowledge discovery platforms targeting highlytechnical fields.Dense representations are used as features for downstream components and have multiple applications ranging from ranking results in search to summarization.Common approaches to create dense representations include training domain-specific embeddings with self-supervised setups or using sentence encoder models trained over similarity tasks.In contrast to static embeddings, sentence encoders do not suffer from the out-ofvocabulary (OOV) problem, but impose significant computational costs.In this paper, we propose a fully unsupervised approach to text encoding that consists of training small character-based models with the objective of reconstructing large pre-trained embedding matrices.Models trained with this approach can not only match the quality of sentence encoders in technical domains, but are 5 times smaller and up to 10 times faster, even on high-end GPUs.
Evaluation of Question Generation Needs More References

Question generation (QG) is the task of generating a valid and fluent question based on a given context and the target answer.According to various purposes, even given the same context, instructors can ask questions about different concepts, and even the same concept can be written in different ways.However, the evaluation for QG usually depends on single reference-based similarity metrics, such as ngram-based metric or learned metric, which is not sufficient to fully evaluate the potential of QG methods.To this end, we propose to paraphrase the reference question for a more robust QG evaluation.Using large language models such as GPT-3, we created semantically and syntactically diverse questions, then adopt the simple aggregation of the popular evaluation metrics as the final scores.Through our experiments, we found that using multiple (pseudo) references is more effective for QG evaluation while showing a higher correlation with human evaluations than evaluation with a single reference.
Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast

Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages.However, there is a clear gap between the performance of the source language and that of the non-source languages.This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most.Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting.Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues.Experimental results show the proposed method outperforms baselines by a clear margin.
Boosting Radiology Report Generation by Infusing Comparison Prior

Recent transformer-based models have made significant strides in generating radiology reports from chest X-ray images.However, a prominent challenge remains: these models often lack prior knowledge, resulting in the generation of synthetic reports that mistakenly reference non-existent prior exams.This discrepancy can be attributed to a knowledge gap between radiologists and the generation models.While radiologists possess patientspecific prior information, the models solely receive X-ray images at a specific time point.To tackle this issue, we propose a novel approach that leverages a rule-based labeler to extract comparison prior information from radiology reports.This extracted comparison prior is then seamlessly integrated into stateof-the-art transformer-based models, enabling them to produce more realistic and comprehensive reports.Our method is evaluated on English report datasets, such as IU X-ray and MIMIC-CXR.The results demonstrate that our approach surpasses baseline models in terms of natural language generation metrics.Notably, our model generates reports that are free from false references to non-existent prior exams, setting it apart from previous models.By addressing this limitation, our approach represents a significant step towards bridging the gap between radiologists and generation models in the domain of medical report generation.
Faithful Low-Resource Data-to-Text Generation through Cycle Training

Methods to generate text from structured data have advanced significantly in recent years, primarily due to fine-tuning of pre-trained language models on large datasets.However, such models can fail to produce output faithful to the input data, particularly on out-of-domain data.Sufficient annotated data is often not available for specific domains, leading us to seek an unsupervised approach to improve the faithfulness of output text.Since the problem is fundamentally one of consistency between the representations of the structured data and text, we evaluate the effectiveness of cycle training in this work.Cycle training uses two models which are inverses of each other: one that generates text from structured data, and one which generates the structured data from natural language text.We show that cycle training, when initialized with a small amount of supervised data (100 samples in our case), achieves nearly the same performance as fully supervised approaches for the data-to-text generation task on the WebNLG, E2E, WTQ, and WSQL datasets.We perform extensive empirical analysis with automated evaluation metrics and a newly designed human evaluation schema to reveal different cycle training strategies' effectiveness of reducing various types of generation errors.
Chain-of-Skills: A Configurable Model for Open-Domain Question Answering

The retrieval model is an indispensable component for real-world knowledge-intensive tasks, e.g., open-domain question answering (ODQA).As separate retrieval skills are annotated for different datasets, recent work focuses on customized methods, limiting the model transferability and scalability.In this work, we propose a modular retriever where individual modules correspond to key skills that can be reused across datasets.Our approach supports flexible skill configurations based on the target domain to boost performance.To mitigate task interference, we design a novel modularization parameterization inspired by sparse Transformer.We demonstrate that our model can benefit from self-supervised pretraining on Wikipedia and fine-tuning using multiple ODQA datasets, both in a multi-task fashion.Our approach outperforms recent self-supervised retrievers in zero-shot evaluations and achieves state-ofthe-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA.