Reviewerly/exHarmony huggingface dataset:
Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning

Thanks to the recent success of Pre-trained Language Models (PLMs), it has become a promising research direction to develop a universal model (UIE) that can solve all typical information extraction tasks within one generative framework. Nonetheless, in real-world scenarios of UIE applications, new data of different IE tasks and domains usually come in a stream over time. A desirable UIE system should be capable of continually learning new tasks without forgetting old ones, thereby allowing knowledge and functionalities expansion without re-training the whole system. In this paper, we study the UIE system under a more challenging yet practical scenario, i.e., “lifelong learning” settings, to evaluate its abilities in three aspects, including knowledge sharing and expansion, catastrophic forgetting prevention, and rapid generalization on few-shot and unseen tasks.To achieve these three goals, we present a novel parameter- and deployment-efficient prompt tuning method namely Lottery Prompt Tuning (LPT).LPT freezes the PLM’s parameters and sequentially learns compact pruned prompt vectors for each task leveraging a binary prompt mask, while keeping the prompt parameters selected by the previous tasks insusceptible.Furthermore, we use a simple yet effective method to perform mask selection and show the powerful transferability of Lottery Prompts to novel tasks.Extensive experiments demonstrate that LPT consistently sets state-of-the-art performance on multiple lifelong learning settings of UIE, including task-incremental setting on seen tasks, few-shot adaptation, and zero-shot generalization on novel tasks.
Jack-Ryder at SemEval-2023 Task 5: Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions

In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023.The core idea behind our system is to leverage pretrained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any taskspecific training.Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QAcapable models.Also, to fit as much relevant context into the model's limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model.Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text.Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results.The ideas described in this paper can easily be applied in fine-tuning settings.
HyHTM: Hyperbolic Geometry-based Hierarchical Topic Model

Hierarchical Topic Models (HTMs) are useful for discovering topic hierarchies in a collection of documents.However, traditional HTMs often produce hierarchies where lower-level topics are unrelated and not specific enough to their higher-level topics.Additionally, these methods can be computationally expensive.We present HyHTM -a Hyperbolic geometry based Hierarchical Topic Models -that addresses these limitations by incorporating hierarchical information from hyperbolic geometry to explicitly model hierarchies in topic models.Experimental results with four baselines show that HyHTM can better attend to parent-child relationships among topics.Hy-HTM produces coherent topic hierarchies that specialise in granularity from generic higherlevel topics to specific lower-level topics.Further, our model is significantly faster and leaves a much smaller memory footprint than our bestperforming baseline.We have made the source code for our algorithm publicly accessible.
Query Enhanced Knowledge-Intensive Conversation via Unsupervised Joint Modeling

In this paper, we propose an unsupervised query enhanced approach for knowledgeintensive conversations, namely QKConv.There are three modules in QKConv: a query generator, an off-the-shelf knowledge selector, and a response generator.QKConv is optimized through joint training, which produces the response by exploring multiple candidate queries and leveraging corresponding selected knowledge.The joint training solely relies on the dialogue context and target response, getting exempt from extra query annotations or knowledge provenances.To evaluate the effectiveness of the proposed QKConv, we conduct experiments on three representative knowledgeintensive conversation datasets: conversational question-answering, task-oriented dialogue, and knowledge-grounded conversation.Experimental results reveal that QKConv performs better than all unsupervised methods across three datasets and achieves competitive performance compared to supervised methods.. . .
TextObfuscator: Making Pre-trained Language Model a Privacy Protector via Obfuscating Word Representations

In real-world applications, pre-trained language models are typically deployed on the cloud, allowing clients to upload data and perform compute-intensive inference remotely.To avoid sharing sensitive data directly with service providers, clients can upload numerical representations rather than plain text to the cloud.However, recent text reconstruction techniques have demonstrated that it is possible to transform representations into original words, suggesting that privacy risk remains.In this paper, we propose TextObfuscator, a novel framework for preserving inference privacy by applying random perturbations to clustered representations.The random perturbations make each word representation indistinguishable from surrounding functionally similar representations, thus obscuring word information while retaining the original word functionality.To achieve this, we utilize prototypes to learn clustered representations, where words of similar functionality are encouraged to be closer to the same prototype during training.Additionally, we design different methods to find prototypes for token-level and sentencelevel tasks, which can improve performance by incorporating semantic and task information.Experimental results on token and sentence classification tasks show that TextObfuscator achieves improvement over compared methods without increasing inference cost.* Equal contribution. Microsoft plansto invest in OpenAI.
mOKB6: A Multilingual Open Knowledge Base Completion Benchmark

Automated completion of open knowledge bases (Open KBs), which are constructed from triples of the form (subject phrase, relation phrase, object phrase), obtained via open information extraction (Open IE) system, are useful for discovering novel facts that may not be directly present in the text.However, research in Open KB completion (Open KBC) has so far been limited to resource-rich languages like English.Using the latest advances in multilingual Open IE, we construct the first multilingual Open KBC dataset, called mOKB6, containing facts from Wikipedia in six languages (including English).Improving the previous Open KB construction pipeline by doing multilingual coreference resolution and keeping only entity-linked triples, we create a dense Open KB.We experiment with several models for the task and observe a consistent benefit of combining languages with the help of shared embedding space as well as translations of facts.We also observe that current multilingual models struggle to remember facts seen in languages of different scripts. 1
Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation

Goal-oriented Script Generation is a new task of generating a list of steps that can fulfill the given goal.In this paper, we propose to extend the task from the perspective of cognitive theory.Instead of a simple flat structure, the steps are typically organized hierarchically -Human often decompose a complex task into subgoals, where each subgoal can be further decomposed into steps.To establish the benchmark, we contribute a new dataset, propose several baseline methods, and set up evaluation metrics.Both automatic and human evaluation verify the high-quality of dataset, as well as the effectiveness of incorporating subgoals into hierarchical script generation.Furthermore, We also design and evaluate the model to discover subgoal, and find that it is a bit more difficult to decompose the goals than summarizing from segmented steps.
iREL at SemEval-2023 Task 10: Multi-level Training for Explainable Detection of Online Sexism

This paper describes our approach for SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS).The task deals with identification and categorization of sexist content into fine-grained categories for explainability in sexism classification.The explainable categorization is proposed through a set of three hierarchical tasks that constitute a taxonomy of sexist content, each task being more granular than the former for categorization of the content.Our team (iREL) participated in all three hierarchical subtasks.Considering the inter-connected task structure, we study multilevel training to study the transfer learning from coarser to finer tasks.Our experiments based on pretrained transformer architectures also make use of additional strategies such as domain-adaptive pretraining to adapt our models to the nature of the content dealt with, and use of the focal loss objective for handling class imbalances.Our best-performing systems on the three tasks achieve macro-F1 scores of 85.93, 69.96 and 54.62 on their respective validation sets.
RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue

Evaluating open-domain dialogue systems is challenging for reasons such as the one-tomany problem, i.e., many appropriate responses other than just the golden response.As of now, automatic evaluation methods need better consistency with humans, while reliable human evaluation can be time-and cost-intensive.To this end, we propose the Reference-Assisted Dialogue Evaluation (RADE) approach under the multi-task learning framework, which leverages the pre-created utterance as reference other than the gold response to relief the one-tomany problem.Specifically, RADE explicitly compares reference and the candidate response to predict their overall scores.Moreover, an auxiliary response generation task enhances prediction via a shared encoder.To support RADE, we extend three datasets with additional rated responses other than just a golden response by human annotation.Experiments on our three datasets and two existing benchmarks demonstrate the effectiveness of our method, where Pearson, Spearman, and Kendall correlations with human evaluation outperform stateof-the-art baselines.
IDRISI-RA: The First Arabic Location Mention Recognition Dataset of Disaster Tweets

Extracting geolocation information from social media data enables effective disaster management, as it helps response authorities; for example, in locating incidents for planning rescue activities, and affected people for evacuation.Nevertheless, geolocation extraction is greatly understudied for the low resource languages such as Arabic.To fill this gap, we introduce IDRISI-RA, the first publicly-available Arabic Location Mention Recognition (LMR) dataset that provides human-and automatically-labeled versions in order of thousands and millions of tweets, respectively.It contains both location mentions and their types (e.g., district, city).Our extensive analysis shows the decent geographical, domain, location granularity, temporal, and dialectical coverage of IDRISI-RA.Furthermore, we establish baselines using the standard Arabic NER models and build two simple, yet effective, LMR models.Our rigorous experiments confirm the need for developing specific models for Arabic LMR in the disaster domain.Moreover, experiments show the promising domain and geographical generalizability of IDRISI-RA under zero-shot learning.
Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers (BORT)

Speech language pathologists rely on information spanning the layers of language, often drawing from multiple layers (e.g.phonology & semantics) at once.Recent innovations in large language models (LLMs) have been shown to build powerful representations for many complex language structures, especially syntax and semantics, unlocking the potential of large datasets through self-supervised learning techniques.However, these datasets are overwhelmingly orthographic, favoring writing systems like the English alphabet, a natural but phonetically imprecise choice.Meanwhile, LLM support for the international phonetic alphabet (IPA) ranges from poor to absent.Further, LLMs encode text at a word-or near-word level, and pre-training tasks have little to gain from phonetic/phonemic representations.In this paper, we introduce BORT, an LLM for mixed orthography/IPA meant to overcome these limitations.To this end, we extend the pre-training of an existing LLM with our own self-supervised pronunciation tasks.We then finetune for a clinical task that requires simultaneous phonological and semantic analysis.For an "easy" and "hard" version of these tasks, we show that fine-tuning from our models is more accurate by a relative 24% and 29%, and improves on character error rates by a relative 75% and 31%, respectively, than those starting from the original model.
Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projection

Natural language processing models tend to learn and encode social biases present in the data.One popular approach for addressing such biases is to eliminate encoded information from the model's representations.However, current methods are restricted to removing only linearly encoded information.In this work, we propose Iterative Gradient-Based Projection (IGBP), a novel method for removing non-linear encoded concepts from neural representations.Our method consists of iteratively training neural classifiers to predict a particular attribute we seek to eliminate, followed by a projection of the representation on a hypersurface, such that the classifiers become oblivious to the target attribute.We evaluate the effectiveness of our method on the task of removing gender and race information as sensitive attributes.Our results demonstrate that IGBP is effective in mitigating bias through intrinsic and extrinsic evaluations, with minimal impact on downstream task accuracy. 1
Abstractive Summarizers are Excellent Extractive Summarizers

In this paper, we explore the efficacy of modeling extractive summarization with an abstractive summarization system.We propose three novel inference algorithms for sequence-tosequence models, evaluate them on established summarization benchmarks, and show that recent advancements in abstractive designs have enabled them to compete directly with extractive systems with custom extractive architectures.We show for the first time that a single model can simultaneously produce both stateof-the-art abstractive and extractive summaries, introducing a unified paradigm for summarization systems.Our results question fundamental concepts of extractive systems and pave the way for a new paradigm -generative modeling for extractive summarization. 1
VBD_NLP at SemEval-2023 Task 2: Named Entity Recognition Systems Enhanced by BabelNet and Wikipedia

We describe our systems participated in the SemEval-2023 shared task for Named Entity Recognition (NER) in English and Bangla.In order to address the challenges of the task, where a large number of fine-grained named entity types need to be detected with only a small amount of training data, we use a method to augment the training data based on Babel-Net concepts and Wikipedia redirections to automatically annotate named entities from Wikipedia articles.We build our NER systems based on the powerful mDeBERTa pretrained language model and trained on the augmented data.Our approach significantly enhances the performance of the fine-grained NER task in both English and Bangla subtracks, outperforming the baseline models.Specifically, our augmented systems achieve macro-f1 scores of 52.64% and 64.31%, representing improvements of 2.38% and 11.33% over the English and Bangla baselines, respectively.
Coherent or Not? Stressing a Neural Language Model for Discourse Coherence in Multiple Languages

In this study, we investigate the capability of a Neural Language Model (NLM) to distinguish between coherent and incoherent text, where the latter has been artificially created to gradually undermine local coherence within text.While previous research on coherence assessment using NLMs has primarily focused on English, we extend our investigation to multiple languages.We employ a consistent evaluation framework to compare the performance of monolingual and multilingual models in both in-domain and out-domain settings.Additionally, we explore the model's performance in a cross-language scenario.
PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

Large-scale vision language (VL) models use Transformers to perform cross-modal interactions between the input text and image.These cross-modal interactions are computationally expensive and memory-intensive due to the quadratic complexity of processing the input image and text.We present PuMer 1 : a token reduction framework that uses text-informed Pruning and modality-aware Merging strategies to progressively reduce the tokens of input image and text, improving model inference speed and reducing memory footprint.PuMer learns to keep salient image tokens related to the input text and merges similar textual and visual tokens by adding lightweight token reducer modules at several cross-modal layers in the VL model.Training PuMer is mostly the same as finetuning the original VL model but faster.Our evaluation for two vision language models on four downstream VL tasks shows PuMer increases inference throughput by up to 2x and reduces memory footprint by over 50% while incurring less than a 1% accuracy drop. 2
Promoting Fairness in Classification of Quality of Medical Evidence

Automatically rating the quality of published research is a critical step in medical evidence synthesis.While several methods have been proposed, their algorithmic fairness has been overlooked, even though significant risks may result when such systems are deployed in biomedical contexts.In this work, we study the fairness of two systems with respect to two sensitive attributes: participant sex and medical area.In some cases, we find important inequalities, leading us to apply various debiasing methods.Upon examining the interplay of predictive performance and fairness, as well as medically-critical selective classification capabilities and calibration performance, we find that it is possible to improve fairness through debiasing, but often at a cost to other performance measures.
Conjunct Resolution in the Face of Verbal Omissions

Verbal omissions are complex syntactic phenomena in VP coordination structures.They occur when verbs and (some of) their arguments are omitted from subsequent clauses after being explicitly stated in an initial clause.Recovering these omitted elements is necessary for accurate interpretation of the sentence, and while humans easily and intuitively fill in the missing information, state-of-the-art models continue to struggle with this task.Previous work is limited to small-scale datasets, synthetic data creation methods, and to resolution methods in the dependency-graph level.In this work we propose a conjunct resolution task that operates directly on the text and makes use of a split-andrephrase paradigm in order to recover the missing elements in the coordination structure.To this end, we first formulate a pragmatic framework of verbal omissions which describes the different types of omissions, and develop an automatic scalable collection method.Based on this method, we curate a large dataset, containing over 10K examples of naturally-occurring verbal omissions with crowd-sourced annotations of the resolved conjuncts.We train various neural baselines for this task, and show that while our best method obtains decent performance, it leaves ample space for improvement.We propose our dataset, metrics and models as a starting point for future research on this topic.
“When Words Fail, Emojis Prevail”: A Novel Architecture for Generating Sarcastic Sentences With Emoji Using Valence Reversal and Semantic Incongruity

Sarcasm is a form of figurative language that serves as a humorous tool for mockery and ridicule.We present a novel architecture for sarcasm generation with emoji from a nonsarcastic input sentence in English.We divide the generation task into two sub tasks: one for generating textual sarcasm and another for collecting emojis associated with those sarcastic sentences.Two key elements of sarcasm are incorporated into the textual sarcasm generation task: valence reversal and semantic incongruity with context, where the context may involve shared commonsense or general knowledge between the speaker and their audience.The majority of existing sarcasm generation works have focused on this textual form.However, in the real world, when written texts fall short of effectively capturing the emotional cues of spoken and face-to-face communication, people often opt for emojis to accurately express their emotions.Due to the wide range of applications of emojis, incorporating appropriate emojis to generate textual sarcastic sentences helps advance sarcasm generation.We conclude our study by evaluating the generated sarcastic sentences using human judgement.
Recyclable Tuning for Continual Pre-training

Continual pre-training is the paradigm where pre-trained language models (PLMs) continually acquire fresh knowledge from growing data and gradually get upgraded.Before an upgraded PLM is released, we may have tuned the original PLM for various tasks and stored the adapted weights.However, when tuning the upgraded PLM, these outdated adapted weights will typically be ignored and discarded, causing a potential waste of resources.We bring this issue to the forefront and contend that proper algorithms for recycling outdated adapted weights should be developed.To this end, we formulate the task of recyclable tuning for continual pre-training.In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent.Motivated by this finding, we analyze the connection between continually pre-trained PLMs from two novel aspects, i.e., mode connectivity, and functional similarity.Based on the corresponding findings, we propose both an initializationbased method and a distillation-based method for our task.We demonstrate their feasibility in improving the convergence and performance for tuning the upgraded PLM.We also show that both methods can be combined to achieve better performance.The source codes are publicly available at https://github.com/thunlp/RecyclableTuning.