Reviewerly/exHarmony huggingface dataset:
Answering Ambiguous Questions via Iterative Prompting

In open-domain question answering, due to the ambiguity of questions, multiple plausible answers may exist.To provide feasible answers to an ambiguous question, one approach is to directly predict all valid answers, but this can struggle with balancing relevance and diversity.An alternative is to gather candidate answers and aggregate them, but this method can be computationally costly and may neglect dependencies among answers.In this paper, we present AmbigPrompt to address the imperfections of existing approaches to answering ambiguous questions.Specifically, we integrate an answering model with a prompting model in an iterative manner.The prompting model adaptively tracks the reading process and progressively triggers the answering model to compose distinct and relevant answers.Additionally, we develop a task-specific post-pretraining approach for both the answering model and the prompting model, which greatly improves the performance of our framework.Empirical studies on two commonlyused open benchmarks show that Ambig-Prompt achieves state-of-the-art or competitive results while using less memory and having a lower inference latency than competing approaches.Additionally, AmbigPrompt also performs well in low-resource settings.
Visual Coherence Loss for Coherent and Visually Grounded Story Generation

Local coherence is essential for long-form text generation models.We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters.In this paper, we propose a loss function inspired by a linguistic theory of coherence for selfsupervised learning for image sequence representations.We further propose combining features from an object and a face detector to construct stronger character features.To evaluate input-output relevance that current referencebased metrics don't measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences.Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories.
From Key Points to Key Point Hierarchy: Structured and Expressive Opinion Summarization

Key Point Analysis (KPA) has been recently proposed for deriving fine-grained insights from collections of textual comments.KPA extracts the main points in the data as a list of concise sentences or phrases, termed key points, and quantifies their prevalence.While key points are more expressive than word clouds and key phrases, making sense of a long, flat list of key points, which often express related ideas in varying levels of granularity, may still be challenging.To address this limitation of KPA, we introduce the task of organizing a given set of key points into a hierarchy, according to their specificity.Such hierarchies may be viewed as a novel type of Textual Entailment Graph.We develop THINKP, a high quality benchmark dataset of key point hierarchies for business and product reviews, obtained by consolidating multiple annotations.We compare different methods for predicting pairwise relations between key points, and for inferring a hierarchy from these pairwise predictions.In particular, for the task of computing pairwise key point relations, we achieve significant gains over existing strong baselines by applying directional distributional similarity methods to a novel distributional representation of key points, and further boost performance via weak supervision.https://github.com/IBM/kpa-hierarchy IntroductionMany organizations face the challenge of extracting insights from large collections of textual comments, such as user reviews, survey responses, and feedback from customers or employees.Current text analytics tools summarize such datasets via word clouds (Heimerl et al., 2014) or key phrases (Hasan and Ng, 2014;Merrouni et al., 2019), which are often too crude to capture fine-grained insights.
Generating Labeled Data for Relation Extraction: A Meta Learning Approach with Joint GPT-2 Training

Relation Extraction (RE) is the task of identifying semantic relation between real-world entities mentioned in text.Despite significant progress in RE research, a remaining challenge for RE concerns the lack of training data for data-hungry deep learning models.Cost of annotation and difficulty of the task are among hindrance to collect a large-scale RE dataset in different domains.To address this limitation, we propose a novel framework to automatically generate labeled data for RE.Our framework presents the pre-trained language model GPT-2 for data generation.In addition, to optimize the generated samples for an RE model, we introduce a meta learning approach to allow the GPT-2 model to be updated during the training process for RE.In particular, to leverage the feedback from the RE model to improve the data generation from GPT-2, we propose a novel reward function to update the GPT-2 model with REINFORCE, seeking to promote the similarity of the RE loss function's gradients computed for generated data and a meta development set.We conduct extensive experiments on two benchmark datasets to produce state-of-the-art performance for RE.
Multilingual Event Extraction from Historical Newspaper Adverts

NLP methods can aid historians in analyzing textual materials in greater volumes than manually feasible.Developing such methods poses substantial challenges though.First, acquiring large, annotated historical datasets is difficult, as only domain experts can reliably label them.Second, most available off-the-shelf NLP models are trained on modern language texts, rendering them significantly less effective when applied to historical corpora.This is particularly problematic for less well studied tasks, and for languages other than English.This paper addresses these challenges while focusing on the under-explored task of event extraction from a novel domain of historical texts.We introduce a new multilingual dataset in English, French, and Dutch composed of newspaper ads from the early modern colonial period reporting on enslaved people who liberated themselves from enslavement.We find that: 1) even with scarce annotated data, it is possible to achieve surprisingly good results by formulating the problem as an extractive QA task and leveraging existing datasets and models for modern languages; and 2) cross-lingual low-resource learning for historical languages is highly challenging, and machine translation of the historical datasets to the considered target languages is, in practice, often the best-performing solution.RUn away from a Ship in the Port of Weymouth, belonging to Dorchester, Mr William Ward Commander, bound for Newfoundland, Philip Mardery a Negro, Aged 22 years middle Sized Man, with a Close Bodied Frize Coat Lined, Buttons of the same, a small Cape, a great Coat not Lined, with Blew Shirts fitting for the Sea; All Gentlemen Captains, or Masters of Ships, that shall happen to have any such offered to them, as a Servant, are desired to give notice to Mr. Walters Grocer in Westminster; or to Mr. Killman Apothecary in Sarum; or to Mr. Turner, at his Coffee House in Dorchester, and they shall be well rewarded with
Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages

Sentiment analysis (SA) systems are used in many products and hundreds of languages.Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies.To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages.We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training?Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies.We release our code and evaluation corpora to facilitate future research.
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models

Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions.Knowledge distillation has been shown to be a practical solution to reduce inference costs, but the distillation process itself requires significant computational resources.Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manually label additional fine-tuning data.In this paper, we investigate how to most efficiently use a fixed budget to build a compact model.Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small).We further investigate how the optimal budget allocated towards computation varies across scenarios.We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.
Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) has made great progress that benefits from extraordinary fusion scheme.However, there is a lack of labeled data, resulting in severe overfitting and poor generalization for supervised models applied in this field.In this paper, we propose Sentiment Knowledge Enhanced Selfsupervised Learning (SKESL) to capture common sentimental patterns in unlabeled videos, which facilitates further learning on limited labeled data.Specifically, with the help of sentiment knowledge and non-verbal behavior, SKESL conducts sentiment word masking and predicts fine-grained word sentiment intensity, so as to embed sentiment information at the word level into pre-trained multimodal representation.In addition, a non-verbal injection method is also proposed to integrate non-verbal information into the word semantics.Experiments on two standard benchmarks of MSA clearly show that SKESL significantly outperforms the baseline, and achieves new State-Of-The-Art (SOTA) results.
Comparing and combining some popular NER approaches on Biomedical tasks

We compare three simple and popular approaches for NER: 1) SEQ (sequencelabeling with a linear token classifier) 2) SeqCRF (sequence-labeling with Conditional Random Fields), and 3) SpanPred (spanprediction with boundary token embeddings).We compare the approaches on 4 biomedical NER tasks:GENIA, NCBI-Disease, LivingNER (Spanish), SocialDisNER (Spanish).The SpanPred model demonstrates state-of-the-art performance on LivingNER and SocialDisNER, improving F1 by 1.3 and 0.6 F1 respectively.The SeqCRF model also demonstrates state-of-the-art performance on LivingNER and SocialDisNER, improving F1 by 0.2 F1 and 0.7 respectively.The SEQ model is competitive with the state-of-the-art on the LivingNER dataset.We explore some simple ways of combining the three approaches.We find that majority voting consistently gives high precision and high F1 across all 4 datasets.Lastly, we implement a system that learns to combine the predictions of SEQ and SpanPred, generating systems that consistently give high recall and high F1 across all 4 datasets.On the GENIA dataset, we find that our learned combiner system significantly boosts F1(+1.2) and recall(+2.1)over the systems being combined.We release all the well-documented code necessary to reproduce all systems at this Github repository.
VTCC-NLP at SemEval-2023 Task 6:Long-Text Representation Based on Graph Neural Network for Rhetorical Roles Prediction

Rhetorical Roles (RR) prediction is to predict the label of each sentence in legal documents, which is regarded as an emergent task for legal document understanding.In this study, we present a novel method for the RR task by exploiting the long context representation.Specifically, legal documents are known as long texts, in which previous works have no ability to consider the inherent dependencies among sentences.In this paper, we propose GNNRR (Graph Neural Network for Rhetorical Roles Prediction), which is able to model the crossinformation for long texts.Furthermore, we develop multitask learning by incorporating label shift prediction (LSP) for segmenting a legal document.The proposed model is evaluated on the SemEval 2023 Task 6 -Legal Eval Understanding Legal Texts for RR sub-task.Accordingly, our method achieves the top 4 in the public leaderboard of the sub-task.Our source code is available for further investigation 1 .
Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models

Language models have been shown to exhibit positive scaling, where performance improves as models are scaled up in terms of size, compute, or data.In this work, we introduce NeQA, a dataset consisting of questions with negation in which language models do not exhibit straightforward positive scaling.We show that this task can exhibit inverse scaling, U-shaped scaling, or positive scaling, and the three scaling trends shift in this order as we use more powerful prompting methods or model families.We hypothesize that solving NeQA depends on two subtasks: question answering (task 1) and negation understanding (task 2).We find that task 1 has linear scaling, while task 2 has sigmoid-shaped scaling with an emergent transition point, and composing these two scaling trends yields the final scaling trend of NeQA.Our work reveals and provides a way to analyze the complex scaling trends of language models.
A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks

By focusing the pre-training process on domain-specific corpora, some domain-specific pre-trained language models (PLMs) have achieved state-of-the-art results.However, it is under-investigated to design a unified paradigm to inject domain knowledge in the PLM finetuning stage.We propose KnowledgeDA, a unified domain language model development service to enhance the task-specific training procedure with domain knowledge graphs.Given domain-specific task texts input, KnowledgeDA can automatically generate a domain-specific language model following three steps: (i) localize domain knowledge entities in texts via an embedding-similarity approach; (ii) generate augmented samples by retrieving replaceable domain entity pairs from two views of both knowledge graph and training data; (iii) select high-quality augmented samples for fine-tuning via confidence-based assessment.We implement a prototype of KnowledgeDA to learn language models for two domains, healthcare and software development.Experiments on domainspecific text classification and QA tasks verify the effectiveness and generalizability of KnowledgeDA.
A Gradient Control Method for Backdoor Attacks on Parameter-Efficient Tuning

Parameter-Efficient Tuning (PET) has shown remarkable performance by fine-tuning only a small number of parameters of the pre-trained language models (PLMs) for the downstream tasks, while it is also possible to construct backdoor attacks due to the vulnerability of pretrained weights.However, a large reduction in the number of attackable parameters in PET will cause the user's fine-tuning to greatly affect the effectiveness of backdoor attacks, resulting in backdoor forgetting.We find that the backdoor injection process can be regarded as multitask learning, which has a convergence imbalance problem between the training of clean and poisoned data.And this problem might result in forgetting the backdoor.Based on this finding, we propose a gradient control method to consolidate the attack effect, comprising two strategies.One controls the gradient magnitude distribution cross layers within one task and the other prevents the conflict of gradient directions between tasks.Compared with previous backdoor attack methods in the scenario of PET, our method improves the effect of the attack on sentiment classification and spam detection respectively, which shows that our method is widely applicable to different tasks.
IITR at BioLaySumm Task 1:Lay Summarization of BioMedical articles using Transformers

This paper presents our submission to the shared task-Lay Summarization of Biomedical Research Articles at BioNLP-2023 workshop(Goldsack et al., 2023).The purpose of this task is the summarize biomedical articles in a concise and less technical way increasing their readability and their reach to lay audiences.In this paper, we use BART-based summarization techniques.We used labels of sentences to improve the performance of our model.Our model achieved a rouge-1 score value of 42.89% and an FKGL score of 10.7901 in relevance and readability parameters respectively.
Detecting Contradictory COVID-19 Drug Efficacy Claims from Biomedical Literature

The COVID-19 pandemic created a deluge of questionable and contradictory scientific claims about drug efficacy -an "infodemic" with lasting consequences for science and society.In this work, we argue that NLP models can help domain experts distill and understand the literature in this complex, high-stakes area.Our task is to automatically identify contradictory claims about COVID-19 drug efficacy.We frame this as a natural language inference problem and offer a new NLI dataset created by domain experts.The NLI framing allows us to create curricula combining existing datasets and our own.The resulting models are useful investigative tools.We provide a case study of how these models help a domain expert summarize and assess evidence concerning remdisivir and hydroxychloroquine. 1
The Kyoto Speech-to-Speech Translation System for IWSLT 2023

This paper describes the Kyoto speech-tospeech translation system for IWSLT 2023.Our system is a combination of speech-to-text translation and text-to-speech synthesis.For the speech-to-text translation model, we used the dual-decoder Transformer model.For the text-to-speech synthesis model, we took a cascade approach of an acoustic model and a vocoder.
COGEN: Abductive Commonsense Language Generation

Reasoning is one of the most important elements in achieving Artificial General Intelligence (AGI), specifically when it comes to Abductive and counterfactual reasoning.In order to introduce these capabilities of reasoning in Natural Language Processing (NLP) models, there have been recent advances toward training NLP models to better perform on two main tasks -Abductive Natural Language Inference (αNLI) and Abductive Natural Language Generation Task (αNLG).This paper proposes CO-GEN, a model for both αNLI and αNLG tasks that employs a novel approach of combining the temporal commonsense reasoning for each observation (before and after a real hypothesis) from pre-trained models with entailment-based filtering for training.Additionally, we use stateof-the-art semantic entailment to filter out the contradictory hypothesis during the inference.Our experimental results show that COGEN outperforms current models and set a new state of the art in regards to αNLI and αNLG tasks.We make the source code of the COGEN model publicly available for reproducibility and to facilitate relevant future research.
Learning Nearest Neighbour Informed Latent Word Embeddings to Improve Zero-Shot Machine Translation

Multilingual neural translation models exploit cross-lingual transfer to perform zero-shot translation between unseen language pairs.Past efforts to improve cross-lingual transfer have focused on aligning contextual sentence-level representations.This paper introduces three novel contributions to allow exploiting nearest neighbours at the token level during training, including: (i) an efficient, gradient-friendly way to share representations between neighboring tokens; (ii) an attentional semantic layer which extracts latent features from shared embeddings; and (iii) an agreement loss to harmonize predictions across different sentence representations.Experiments on two multilingual datasets demonstrate consistent gains in zero shot translation over strong baselines.
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning

Large pre-trained models are capable of fewshot in-context learning (ICL), i.e., performing a new task by prepending a few demonstrations before the test input.However, the concatenated demonstrations are often excessively long and induce additional computation.Inspired by fusion-in-decoder (FiD) models which efficiently aggregate more passages and thus outperforms concatenation-based models in opendomain QA, we hypothesize that similar techniques can be applied to improve the efficiency and end-task performance of ICL.To verify this, we present a comprehensive study on applying three fusion methods-concatenationbased (early fusion), FiD (intermediate), and ensemble-based (late)-to ICL.We adopt a meta-learning setup where a model is first trained to perform ICL on a mixture of tasks using one selected fusion method, then evaluated on held-out tasks for ICL.Results on 11 heldout tasks show that FiD-ICL matches or outperforms the other two fusion methods.Additionally, we show that FiD-ICL (1) is 10x faster at inference time compared to concat-based and ensemble-based ICL, as we can easily precompute the representations of in-context examples and reuse them; (2) enables scaling up to meta-training 3B-sized models, which would fail for concat-based ICL.1
Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers

Without any explicit cross-lingual training data, multilingual language models can achieve cross-lingual transfer.One common way to improve this transfer is to perform realignment steps before fine-tuning, i.e., to train the model to build similar representations for pairs of words from translated sentences.But such realignment methods were found to not always improve results across languages and tasks, which raises the question of whether aligned representations are truly beneficial for crosslingual transfer.We provide evidence that alignment is actually significantly correlated with cross-lingual transfer across languages, models and random seeds.We show that fine-tuning can have a significant impact on alignment, depending mainly on the downstream task and the model.Finally, we show that realignment can, in some instances, improve cross-lingual transfer, and we identify conditions in which realignment methods provide significant improvements.Namely, we find that realignment works better on tasks for which alignment is correlated with cross-lingual transfer when generalizing to a distant language and with smaller models, as well as when using a bilingual dictionary rather than FastAlign to extract realignment pairs.For example, for POS-tagging, between English and Arabic, realignment can bring a +15.8 accuracy improvement on distilm-BERT, even outperforming XLM-R Large by 1.7.We thus advocate for further research on realignment methods for smaller multilingual models as an alternative to scaling.