Reviewerly/exHarmony huggingface dataset:
Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models

Recent studies have revealed that the widelyused Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora.Existing solutions require debiasing training processes and datasets for debiasing, which are resourceintensive and costly.Furthermore, these methods hurt the PLMs' performance on downstream tasks.In this study, we propose Gendertuning, which debiases the PLMs through finetuning on downstream tasks' datasets.For this aim, Gender-tuning integrates Masked Language Modeling (MLM) training objectives into fine-tuning's training process.Comprehensive experiments show that Gender-tuning outperforms the state-of-the-art baselines in terms of average gender bias scores in PLMs while improving PLMs' performance on downstream tasks solely using the downstream tasks' dataset.Also, Gender-tuning is a deployable debiasing tool for any PLM that works with original fine-tuning.
DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition

The MultiCoNER II shared task aims to tackle multilingual named entity recognition (NER) in fine-grained and noisy scenarios, and it inherits the semantic ambiguity and low-context setting of the MultiCoNER I task.To cope with these problems, the previous top systems in the MultiCoNER I either incorporate the knowledge bases or gazetteers.However, they still suffer from insufficient knowledge, limited context length, single retrieval strategy.In this paper, our team DAMO-NLP proposes a unified retrieval-augmented system (U-RaNER) for fine-grained multilingual NER.We perform error analysis on the previous top systems and reveal that their performance bottleneck lies in insufficient knowledge.Also, we discover that the limited context length causes the retrieval knowledge to be invisible to the model.To enhance the retrieval context, we incorporate the entity-centric Wikidata knowledge base, while utilizing the infusion approach to broaden the contextual scope of the model.Also, we explore various search strategies and refine the quality of retrieval knowledge.Our system 1 wins 9 out of 13 tracks in the MultiCoNER II shared task.Additionally, we compared our system with ChatGPT, one of the large language models which have unlocked strong capabilities on many tasks.The results show that there is still much room for improvement for ChatGPT on the extraction task.
NetEase.AI at SemEval-2023 Task 2: Enhancing Complex Named Entities Recognition in Noisy Scenarios via Text Error Correction and External Knowledge

Complex named entities (NE), like the titles of creative works, are not simple nouns and pose challenges for NER systems.In the Se-mEval 2023, Task 2: MultiCoNER II was proposed, whose goal is to recognize complex entities against out of knowledge-base entities and noisy scenarios.To address the challenges posed by MultiCoNER II, our team NetEase.AI proposed an entity recognition system that integrates text error correction system and external knowledge, which can recognize entities in scenes that contain entities out of knowledge base and text with noise.Upon receiving an input sentence, our systems will correct the sentence, extract the entities in the sentence as candidate set using the entity recognition model that incorporates the gazetteer information, and then use the external knowledge to classify the candidate entities to obtain entity type features.Finally, our system fused the multi-dimensional features of the candidate entities into a stacking model, which was used to select the correct entities from the candidate set as the final output.Our system exhibited good noise resistance and excellent entity recognition performance, resulting in our team's first place victory in the Chinese track of MultiCoNER II.
HealthMavericks@MEDIQA-Chat 2023: Benchmarking different Transformer based models for Clinical Dialogue Summarization

In recent years, we have seen many Transformer based models being created to address Dialog Summarization problem. While there has been a lot of work on understanding how these models stack against each other in summarizing regular conversations such as the ones found in DialogSum dataset, there haven’t been many analysis of these models on Clinical Dialog Summarization. In this article, we describe our solution to MEDIQA-Chat 2023 Shared Tasks as part of ACL-ClinicalNLP 2023 workshop which benchmarks some of the popular Transformer Architectures such as BioBart, Flan-T5, DialogLED, and OpenAI GPT3 on the problem of Clinical Dialog Summarization. We analyse their performance on two tasks - summarizing short conversations and long conversations. In addition to this, we also benchmark two popular summarization ensemble methods and report their performance.
NLP-Titan at SemEval-2023 Task 6: Identification of Rhetorical Roles Using Sequential Sentence Classification

The analysis of legal cases poses a considerable challenge for researchers, practitioners, and academicians due to the lengthy and intricate nature of these documents.Developing countries such as India are experiencing a significant increase in the number of pending legal cases, which are often unstructured and difficult to process using conventional methods.To address this issue, the authors have implemented a sequential sentence classification process, which categorizes legal documents into 13 segments, known as Rhetorical Roles.This approach enables the extraction of valuable insights from the various classes of the structured document.The performance of this approach was evaluated using the F1 score, which measures the model's precision and recall.The authors' approach achieved an F1 score of 0.83, which surpasses the baseline score of 0.79 established by the task organizers.The authors have combined sequential sentence classification and the SetFit method in a hierarchical manner by combining similar classes to achieve this score.
Togedemaru at SemEval-2023 Task 8: Causal Medical Claim Identification and Extraction from Social Media Posts

The "Causal Medical Claim Identification and 2 Extraction from Social Media Posts" task at SemEval 3 2023 competition focuses on identifying and validating 4 medical claims in English, by posing two subtasks on 5 causal claim identification and PIO (Population, 6 Intervention, Outcome) frame extraction.In the context 7 of SemEval, we present a method for sentence 8 classification in four categories (claim, experience, 9 experience_based_claim or a question) based on 10 BioBERT model with a MLP layer.The website from 11 which the dataset was gathered, Reddit, is a social news 12 and content discussion site.The evaluation results show 13 the effectiveness of the solution of this study (83.68%).14 This topic has attracted significant attention in 46 recent years, evidenced by increasing number of 47 workshops (e.g., Workshop on Curative Power of 48
IUTEAM1 at MEDIQA-Chat 2023: Is simple fine tuning effective for multi layer summarization of clinical conversations?

Clinical conversation summarization has become an important application of Natural language Processing.In this work, we intend to analyze summarization model ensembling approaches, that can be utilized to improve the overall accuracy of the generated medical report called chart note.The work starts with a single summarization model creating the baseline.Then leads to an ensemble of summarization models trained on a separate section of the chart note.This leads to the final approach of passing the generated results to another summarization model in a multi-layer/stage fashion for better coherency of the generated text.Our results indicate that although an ensemble of models specialized in each section produces better results, the multi-layer/stage approach does not improve accuracy.The code for the above paper is available at https://github.
LRL_NC at SemEval-2023 Task 4: The Touche23-George-boole Approach for Multi-Label Classification of Human-Values behind Arguments

The task ValueEval aims at assigning a subset of possible human value categories underlying a given argument.Values behind arguments are often determinants to evaluate the relevance and importance of decisions in ethical sense, thereby making them essential for argument mining.The work presented here proposes two systems for the same.Both systems use RoBERTa to encode sentences in each document.System1 makes use of features obtained from training models for two auxiliary tasks, whereas System2 combines RoBERTa with topic modeling to get sentence representation.These features are used by a classification head to generate predictions.System1 secured the rank 22 in the official task ranking, achieving the macro F1-score 0.46 on the main dataset.System2 was not a part of official evaluation.Subsequent experiments achieved highest (among the proposed systems) macro F1-scores of 0.48 (System2), 0.31 (ablation on System1) and 0.33 (ablation on System1) on the main dataset, the Nahj al-Balagha dataset, and the New York Times dataset.
NewAgeHealthWarriors at MEDIQA-Chat 2023 Task A: Summarizing Short Medical Conversation with Transformers

This paper presents the MEDIQA-Chat 2023 shared task organized at the ACL-Clinical NLP workshop.The shared task is motivated by the need to develop methods to automatically generate clinical notes from doctor-patient conversations.In this paper, we present our submission for MEDIQA-Chat 2023 Task A: Short Di-alogue2Note Summarization.Manual creation of these clinical notes requires extensive human efforts, thus making it a time-consuming and expensive process.To address this, we propose an ensemble-based method over GPT-3, BART, BERT variants, and Rule-based systems to automatically generate clinical notes from these conversations.The proposed system achieves a score of 0.730 and 0.544 for both the sub-tasks on the test set (ranking 8th on the leaderboard for both tasks) and shows better performance compared to a baseline system using BART variants.
LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum Learning

A hallmark of human intelligence is the ability to learn new concepts purely from language.Several recent approaches have explored training machine learning models via natural language supervision.However, these approaches fall short in leveraging linguistic quantifiers (such as 'always' or 'rarely') and mimicking humans in compositionally learning complex tasks.Here, we present LaSQuE, a method that can learn zero-shot classifiers from language explanations by using three new strategies -(1) modeling the semantics of linguistic quantifiers in explanations (including exploiting ordinal strength relationships, such as 'always' > 'likely'), (2) aggregating information from multiple explanations using an attention-based mechanism, and (3) model training via curriculum learning.With these strategies, LaSQuE outperforms prior work, showing an absolute gain of up to 7% in generalizing to unseen realworld classification tasks.
How Well Do Large Language Models Perform on Faux Pas Tests?

Motivated by the question of the extent to which large language models "understand" social intelligence, we investigate the ability of such models to generate correct responses to questions involving descriptions of faux pas situations.The faux pas test is a test used in clinical psychology, which is known to be more challenging for children than individual tests of theory-of-mind or social intelligence.Our results demonstrate that, while the models seem to sometimes offer correct responses, they in fact struggle with this task, and that many of the seemingly correct responses can be attributed to over-interpretation by the human reader ("the ELIZA effect").An additional phenomenon observed is the failure of most models to generate a correct response to presupposition questions.Finally, in an experiment in which the models are tasked with generating original faux pas stories, we find that while some models are capable of generating novel faux pas stories, the stories are all explicit, as the models are limited in their abilities to describe situations in an implicit manner.
Exploiting Pseudo Image Captions for Multimodal Summarization

Multimodal summarization with multimodal output (MSMO) faces a challenging semantic gap between visual and textual modalities due to the lack of reference images for training.Our pilot investigation indicates that image captions, which naturally connect texts and images, can significantly benefit MSMO.However, exposure of image captions during training is inconsistent with MSMO's task settings, where prior cross-modal alignment information is excluded to guarantee the generalization of cross-modal semantic modeling.To this end, we propose a novel coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge and bridging the crossmodal semantic gap.Equipped with this alignment mechanism, our method easily yet impressively sets up state-of-the-art performances on all intermodality and intramodality metrics (e.g., more than 10% relative improvement on image recommendation precision).Further experiments reveal the correlation between image captions and text summaries, and prove that the pseudo image captions we generated are even better than the original ones in terms of promoting multimodal summarization.
Frustratingly Easy Label Projection for Cross-lingual Transfer

Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer.For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts.Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence (Lewis et al., 2020;Hu et al., 2020).However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment.In this paper, we present an extensive empirical study across 57 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature.Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods.We analyze several key factors that affect the end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. 1Iraqis protesting the conference said it did not represent their interests .抗议 会议 的 伊拉克⼈ 表示 , 它 不 代表 他们的 利益 。
Resolving Elliptical Compounds in German Medical Text

Elliptical coordinated compound noun phrases (ECCNPs), a special kind of coordination ellipsis, are a common phenomenon in German medical texts.As their presence is known to affect the performance in downstream tasks such as entity extraction and disambiguation, their resolution can be a useful preprocessing step in information extraction pipelines.In this work, we present a new comprehensive dataset of more than 4,000 manually annotated ECCNPs in German medical text, along with the respective ground truth resolutions.Based on this data, we propose a generative encoder-decoder Transformer model, allowing for a simple end-to-end resolution of ECCNPs from raw input strings with very high accuracy (90.5 % exact match score).We compare our approach to an elaborate rule-based baseline, which the generative model outperforms by a large margin.We further investigate different scenarios for prompting large language models (LLM) to resolve ECCNPs.In a zeroshot setting, performance is remarkably poor (21.6 % exact matches), as the LLM tends to apply complex changes to the inputs unrelated to our specific task.We also find no improvement over the generative model when using the LLM for post-filtering of generated candidate resolutions.
Abstractive Text Summarization Using the BRIO Training Paradigm

sentences produced by abstractive summarization models may be coherent and comprehensive, but they lack control and rely heavily on reference summaries.The BRIO training paradigm assumes a non-deterministic distribution to reduce the model's dependence on reference summaries, and improve model performance during inference.This paper presents a straightforward but effective technique to improve abstractive summaries by finetuning pre-trained language models, and training them with the BRIO paradigm.We build a text summarization dataset for Vietnamese, called VieSum.We perform experiments with abstractive summarization models trained with the BRIO paradigm on the CNNDM and the VieSum datasets.The results show that the models, trained on basic hardware, outperform all existing abstractive summarization models, especially for Vietnamese.
Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions

While there is much recent interest in studying why Transformer-based large language models make predictions the way they do, the complex computations performed within each layer have made their behavior somewhat opaque.To mitigate this opacity, this work presents a linear decomposition of final hidden states from autoregressive language models based on each initial input token, which is exact for virtually all contemporary Transformer architectures.This decomposition allows the definition of probability distributions that ablate the contribution of specific input tokens, which can be used to analyze their influence on model probabilities over a sequence of upcoming words with only one forward pass from the model.Using the change in next-word probability as a measure of importance, this work first examines which context words make the biggest contribution to language model predictions.Regression experiments suggest that Transformer-based language models rely primarily on collocational associations, followed by linguistic factors such as syntactic dependencies and coreference relationships in making next-word predictions.Additionally, analyses using these measures to predict syntactic dependencies and coreferent mention spans show that collocational association and repetitions of the same token largely explain the language models' predictions on these tasks.
Towards Fewer Hallucinations in Knowledge-Grounded Dialogue Generation via Augmentative and Contrastive Knowledge-Dialogue

Existing knowledge-grounded open-domain dialogue generation models often face the hallucination problem, i.e. the dialogue generative model will persist in an inappropriate knowledge and generate responses that inconsistent with the facts.We argue that this problem mainly stems from the polarized optimization objectives and weak knowledge generation ability.To mitigate the hallucination, we take inspiration from human communicating that people will replay euphemistic responses for the unclear or unrecognizable knowledge, and propose an Augmentative and Contrastive Knowledge Dialogue Expansion Framework (ACK-DEF).ACK-DEF constructs the augmentative and contrastive knowledge dialogue samples, which consist of the knowledge of different degrees of errors and the response of manual design, to expand the original training set and smooth the polarized optimization objective that enables models to generate ground-truth with or without gold knowledge.Not only the knowledge, ACK-DEF also provides the tactful responses of manual design corresponding to the incomplete correct knowledge.Experimental results on the Wikipedia of Wizard dataset show that employing the ACK-DEF is effective to alleviate the hallucination problem.
Learning Language-Specific Layers for Multilingual Machine Translation

Multilingual Machine Translation promises to improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (e.g., avoiding losing gender and formality information when translating through English).On the downside, adding more languages reduces model capacity per language, which is usually countered by increasing the overall model size, making training harder and inference slower.In this work, we introduce Language-Specific Transformer Layers (LSLs), which allow us to increase model capacity, while keeping the amount of computation and the number of parameters used in the forward pass constant. The key idea is to have some layers of the encoder be source or target language-specific, while keeping the remaining layers shared. We study the best way to place these layers using a neural architecture search inspired approach, and achieve an improvement of 1.3 chrF (1.5 spBLEU) points over not using LSLs on a separate decoder architecture, and 1.9 chrF (2.2 spBLEU) on a shared decoder one.
Domain Incremental Lifelong Learning in an Open World

Lifelong learning (LL) is an important ability for NLP models to learn new tasks continuously.Architecture-based approaches are reported to be effective implementations for LL models.However, it is non-trivial to extend previous approaches to domain incremental LL scenarios since they either require access to task identities in the testing phase or cannot handle samples from unseen tasks.In this paper, we propose Diana: a dynamic architecture-based lifelong learning model that tries to learn a sequence of tasks with a promptenhanced language model.Four types of hierarchically organized prompts are used in Diana to capture knowledge from different granularities.Specifically, we dedicate tasklevel prompts to capture task-specific knowledge to retain high LL performances and maintain instance-level prompts to learn knowledge shared across input samples to improve the model's generalization performance.Moreover, we dedicate separate prompts to explicitly model unseen tasks and introduce a set of prompt key vectors to facilitate knowledge sharing between tasks.Extensive experiments demonstrate that Diana outperforms state-ofthe-art LL models, especially in handling unseen tasks.We release the code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/diana.
Enhancing Out-of-Vocabulary Estimation with Subword Attention

Word embedding methods like word2vec and GloVe have been shown to learn strong representations of words.However, these methods only learn representations for words in the training corpus and therefore struggle to handle unknown and new words, known as out-of-vocabulary (OOV) words.As a result, there have been multiple attempts to learn OOV word representations in a similar fashion to how humans learn new words, using word roots/subwords and/or surrounding words.However, while most of these approaches use advanced architectures like attention on the context of the OOV word, they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN) to handle processing subword information.In response to this, we propose SubAtt, a transformer based OOV estimation model that uses attention mechanisms on both the context and the subwords.In addition to attention, we also show that pretraining subword representations also leads to improvement in OOV estimation.We show SubAtt outperforms current state-ofthe-art OOV estimation models.