Reviewerly/exHarmony huggingface dataset:
Improving Gradient Trade-offs between Tasks in Multi-task Text Classification

Multi-task learning (MTL) has emerged as a promising approach for sharing inductive bias across multiple tasks to enable more efficient learning in text classification.However, training all tasks simultaneously often yields degraded performance of each task than learning them independently, since different tasks might conflict with each other.Existing MTL methods for alleviating this issue is to leverage heuristics or gradient-based algorithm to achieve an arbitrary Pareto optimal trade-off among different tasks.In this paper, we present a novel gradient trade-off approach to mitigate the task conflict problem, dubbed GetMTL, which can achieve a specific tradeoff among different tasks nearby the main objective of multi-task text classification (MTC), so as to improve the performance of each task simultaneously.The results of extensive experiments on two benchmark datasets back up our theoretical analysis and validate the superiority of our proposed GetMTL.
Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints

Multilingual Knowledge Graph Completion (mKGC) aim at solving queries like (h, r, ?) in different languages by reasoning a tail entity t thus improving multilingual knowledge graphs.Previous studies leverage multilingual pretrained language models (PLMs) and the generative paradigm to achieve mKGC.Although multilingual pretrained language models contain extensive knowledge of different languages, its pretraining tasks cannot be directly aligned with the mKGC tasks.Moreover, the majority of KGs and PLMs currently available exhibit a pronounced English-centric bias.This makes it difficult for mKGC to achieve good results, particularly in the context of low-resource languages.To overcome previous problems, this paper introduces global and local knowledge constraints for mKGC.The former is used to constrain the reasoning of answer entities, while the latter is used to enhance the representation of query contexts.The proposed method makes the pretrained model better adapt to the mKGC task.Experimental results on public datasets demonstrate that our method outperforms the previous SOTA on Hits@1 and Hits@10 by an average of 12.32% and 16.03%, which indicates that our proposed method has significant enhancement on mKGC.
WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models

Content Warning: This paper contains examples of homophobic and transphobic stereotypes.We present WinoQueer: a benchmark specifically designed to measure whether large language models (LLMs) encode biases that are harmful to the LGBTQ+ community.The benchmark is community-sourced, via application of a novel method that generates a bias benchmark from a community survey.We apply our benchmark to several popular LLMs and find that off-the-shelf models generally do exhibit considerable anti-queer bias.Finally, we show that LLM bias against a marginalized community can be somewhat mitigated by finetuning on data written about or by members of that community, and that social media text written by community members is more effective than news text written about the community by non-members.Our method for community-in-the-loop benchmark development provides a blueprint for future researchers to develop community-driven, harms-grounded LLM benchmarks for other marginalized communities.
More than Classification: A Unified Framework for Event Temporal Relation Extraction

Event temporal relation extraction (ETRE) is usually formulated as a multi-label classification task, where each type of relation is simply treated as a one-hot label.This formulation ignores the meaning of relations and wipes out their intrinsic dependency.After examining the relation definitions in various ETRE tasks, we observe that all relations can be interpreted using the start and end time points of events.For example, relation Includes could be interpreted as event 1 starting no later than event 2 and ending no earlier than event 2. In this paper, we propose a unified event temporal relation extraction framework, which transforms temporal relations into logical expressions of time points and completes the ETRE by predicting the relations between certain time point pairs.Experiments on TB-Dense and MATRES show significant improvements over a strong baseline and outperform the state-of-the-art model by 0.3% on both datasets.By representing all relations in a unified framework, we can leverage the relations with sufficient data to assist the learning of other relations, thus achieving stable improvement in low-data scenarios.When the relation definitions are changed, our method can quickly adapt to the new ones by simply modifying the logic expressions that map time points to new event relations.
RE-Matching: A Fine-Grained Semantic Matching Method for Zero-Shot Relation Extraction

Semantic matching is a mainstream paradigm of zero-shot relation extraction, which matches a given input with a corresponding label description.The entities in the input should exactly match their hypernyms in the description, while the irrelevant contexts should be ignored when matching.However, general matching methods lack explicit modeling of the above matching pattern.In this work, we propose a fine-grained semantic matching method tailored for zero-shot relation extraction.Following the above matching pattern, we decompose the sentence-level similarity score into entity and context matching scores.Due to the lack of explicit annotations of the redundant components, we design a feature distillation module to adaptively identify the relationirrelevant features and reduce their negative impact on context matching.Experimental results show that our method achieves higher matching F 1 score and has an inference speed 10 times faster, when compared with the stateof-the-art methods.
Joint Generator-Ranker Learning for Natural Language Generation

Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates.However, existing methods usually train the generator and the ranker individually, neglecting the mutual feedback that could further enhance the generation quality.To tackle this limitation, we propose JGR, a novel joint training algorithm that integrates the generator and the ranker in a single framework.JGR optimizes the generator with a hybrid objective that combines data likelihood and ranker reward, and trains the ranker with a contrastive loss that compares the generator outputs.By iteratively updating the generator and the ranker, JGR can effectively harmonize their learning and enhance their quality jointly.We evaluate JGR on various text generation tasks and demonstrate that it surpasses existing methods on four public datasets across three common generation scenarios.Our code and models are publicly available at https:
GDA: Generative Data Augmentation Techniques for Relation Extraction Tasks

Relation extraction (RE) tasks show promising performance in extracting relations from two entities mentioned in sentences, given sufficient annotations available during training.Such annotations would be labor-intensive to obtain in practice.Existing work adopts data augmentation techniques to generate pseudo-annotated sentences beyond limited annotations.These techniques neither preserve the semantic consistency of the original sentences when rulebased augmentations are adopted, nor preserve the syntax structure of sentences when expressing relations using seq2seq models, resulting in less diverse augmentations.In this work, we propose a dedicated augmentation technique for relational texts, named GDA, which uses two complementary modules to preserve both semantic consistency and syntax structures.We adopt a generative formulation and design a multi-tasking solution to achieve synergies.Furthermore, GDA adopts entity hints as the prior knowledge of the generative model to augment diverse sentences.Experimental results in three datasets under a low-resource setting showed that GDA could bring 2.0% F1 improvements compared with no augmentation technique.Source code and data are available 1 .
PTCSpell: Pre-trained Corrector Based on Character Shape and Pinyin for Chinese Spelling Correction

Chinese spelling correction (CSC) is a challenging task with the goal of correcting each wrong character in Chinese texts.Incorrect characters in a Chinese text are mainly due to the similar shape and similar pronunciation of Chinese characters.Recently, the paradigm of pre-training and fine-tuning has achieved remarkable success in natural language processing.However, the pre-training objectives in existing methods are not tailored for the CSC task since they neglect the visual and phonetic properties of characters, resulting in suboptimal spelling correction.In this work, we propose to pre-train a new corrector named PTCSpell for the CSC task under the detector-corrector architecture.The corrector we propose has the following two improvements.First, we design two novel pre-training objectives to capture pronunciation and shape information in Chinese characters.Second, we propose a new strategy to tackle the issue that the detector's prediction results mislead the corrector by balancing the loss of wrong characters and correct characters.Experiments on three benchmarks (i.e., SIGHAN 2013SIGHAN , 2014SIGHAN , and 2015) ) show that our model achieves an average of 5.8% F1 improvements at the correction level over state-of-theart methods, verifying its effectiveness.
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA).They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters.While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA.Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved.To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT.Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC.We observe similar substantial gains in out-ofdistribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training.IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning. 1.
Bidirectional Transformer Reranker for Grammatical Error Correction

Pre-trained seq2seq models have achieved stateof-the-art results in the grammatical error correction task.However, these models still suffer from a prediction bias due to their unidirectional decoding.Thus, we propose a bidirectional Transformer reranker (BTR), that reestimates the probability of each candidate sentence generated by the pre-trained seq2seq model.The BTR preserves the seq2seq-style Transformer architecture but utilizes a BERTstyle self-attention mechanism in the decoder to compute the probability of each target token by using masked language modeling to capture bidirectional representations from the target context.For guiding the reranking, the BTR adopts negative sampling in the objective function to minimize the unlikelihood.During inference, the BTR gives final results after comparing the reranked top-1 results with the original ones by an acceptance threshold λ.Experimental results show that, in reranking candidates from a pre-trained seq2seq model, T5-base, the BTR on top of T5-base could yield 65.47 and 71.27 F 0.5 scores on the CoNLL-14 and BEA test sets, respectively, and yield 59.52 GLEU score on the JFLEG corpus, with improvements of 0.36, 0.76 and 0.48 points compared with the original T5-base.Furthermore, when reranking candidates from T5-large, the BTR on top of T5-base improved the original T5-large by 0.26 points on the BEA test set. 1 Source Speed camera can be placed in many locations along a highway .Gold 1 Speed cameras can be placed in many locations along a highway .Candidate 1 (RoBERTa, T5GEC) A speed camera can be placed in many locations along a highway .Candidate 2 (BTR, R2L) Speed cameras can be placed in many locations along a highway .Candidate 3A Speed camera can be placed in many locations along a highway .
Generating Deep Questions with Commonsense Reasoning Ability from the Text by Disentangled Adversarial Inference

This paper proposes a new task of commonsense question generation, which aims to yield deep-level and to-the-point questions from the text.Their answers need to reason over disjoint relevant contexts and external commonsense knowledge, such as encyclopedic facts and causality.The knowledge may not be explicitly mentioned in the text but is used by most humans for problem-shooting.Such complex reasoning with hidden contexts involves deep semantic understanding.Thus, this task has great application value, such as making high-quality quizzes in advanced exams.Due to the lack of modeling complexity, existing methods may produce shallow questions that can be answered by simple word matching.To address these challenges, we propose a new QG model by simultaneously considering asking contents, expressive ways, and answering complexity.We first retrieve text-related commonsense context.Then we disentangle the key factors that control questions in terms of reasoning content and verbalized way.Independence priors and constraints are imposed to facilitate disentanglement.We further develop a discriminator to promote the deep results by considering their answering complexity.Through adversarial inference, we learn the latent factors from data.By sampling the expressive factor from the data distributions, diverse questions can be yielded.Evaluations of two typical data sets show the effectiveness of our approach.
Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark

Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario.This paper introduces a new dataset SURE (Multimodal Recommendation Dialog with SUbjective PREference), which contains 12K shopping dialogs in complex store scenes.The data is built in two phases with human annotations to ensure quality and diversity.SURE is well-annotated with subjective preferences and recommendation acts proposed by sales experts.A comprehensive analysis is given to reveal the distinguishing features of SURE.Three benchmark tasks are then proposed on the data to evaluate the capability of multimodal recommendation agents.Based on the SURE, we propose a baseline model † , powered by a stateof-the-art multimodal model, for these tasks.
Reducing Knowledge Noise for Improved Semantic Analysis in Biomedical Natural Language Processing Applications

Graph-based techniques have gained traction for representing and analyzing data in various natural language processing (NLP) tasks.Knowledge graph-based language representation models have shown promising results in leveraging domain-specific knowledge for NLP tasks, particularly in the biomedical NLP field.However, such models have limitations, including knowledge noise and neglect of contextual relationships, leading to potential semantic errors and reduced accuracy.To address these issues, this paper proposes two novel methods.The first method combines knowledge graphbased language model with nearest-neighbor models to incorporate semantic and category information from neighboring instances.The second method involves integrating knowledge graph-based language model with graph neural networks (GNNs) to leverage feature information from neighboring nodes in the graph.Experiments on relation extraction (RE) and classification tasks in English and Chinese language datasets demonstrate significant performance improvements with both methods, highlighting their potential for enhancing the performance of language models and improving NLP applications in the biomedical domain.
Uncertainty-Aware Bootstrap Learning for Joint Extraction on Distantly-Supervised Data

Jointly extracting entity pairs and their relations is challenging when working on distantlysupervised data with ambiguous or noisy labels.To mitigate such impact, we propose uncertainty-aware bootstrap learning, which is motivated by the intuition that the higher uncertainty of an instance, the more likely the model confidence is inconsistent with the ground truths.Specifically, we first explore instance-level data uncertainty to create an initial high-confident examples.Such subset serves as filtering noisy instances and facilitating the model to converge fast at the early stage.During bootstrap learning, we propose self-ensembling as a regularizer to alleviate inter-model uncertainty produced by noisy labels.We further define probability variance of joint tagging probabilities to estimate innermodel parametric uncertainty, which is used to select and build up new reliable training instances for the next iteration.Experimental results on two large datasets reveal that our approach outperforms existing strong baselines and related methods.
Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation

Visual temporal-aligned translation aims to transform the visual sequence into natural words, including important applicable tasks such as lipreading and fingerspelling recognition.However, various performance habits of specific words by different speakers or signers can lead to visual ambiguity, which has become a major obstacle to the development of current methods.Considering the constraints above, the generalization ability of the translation system is supposed to be further explored through the evaluation results on unseen performers.In this paper, we develop a novel generalizable framework named Contrastive Token-Wise Meta-learning (CtoML), which strives to transfer recognition skills to unseen performers.To the best of our knowledge, employing meta-learning methods directly in the image domain poses two main challenges, and we propose corresponding strategies.First, sequence prediction in visual temporal-aligned translation, which aims to generate multiple words autoregressively, is different from the vanilla classification.Thus, we devise the token-wise diversity-aware weights for the meta-train stage, which encourages the model to make efforts on those ambiguously recognized tokens.Second, considering the consistency of word-visual prototypes across different domains, we develop two complementary global and local contrastive losses to maintain inter-class relationships and promote domainindependence.We conduct extensive experiments on the widely-used lipreading dataset GRID and the fingerspelling dataset ChicagoF-SWild, and the experimental results show the effectiveness of our proposed CtoML over existing state-of-the-art methods.
DICE: Data-Efficient Clinical Event Extraction with Generative Models

Event extraction for the clinical domain is an under-explored research area.The lack of training data along with the high volume of domainspecific terminologies with vague entity boundaries makes the task especially challenging.In this paper, we introduce DICE, a robust and data-efficient generative model for clinical event extraction.DICE frames event extraction as a conditional generation problem and introduces a contrastive learning objective to accurately decide the boundaries of biomedical mentions.DICE also trains an auxiliary mention identification task jointly with event extraction tasks to better identify entity mention boundaries, and further introduces special markers to incorporate identified entity mentions as trigger and argument candidates for their respective tasks.To benchmark clinical event extraction, we compose MACCROBAT-EE, the first clinical event extraction dataset with argument annotation, based on an existing clinical information extraction dataset, MACCROBAT (Caufield et al., 2019).Our experiments demonstrate state-of-the-art performances of DICE for clinical and news domain event extraction, especially under low data settings.* Equal contribution.A 45 -year -old lady sought dermatology consultation for severely tender erythematous vesicles and bullae over back , chest and arms .
Towards Boosting the Open-Domain Chatbot with Human Feedback

Many open-domain dialogue models pretrained with social media comments can generate coherent replies but have difficulties producing engaging responses.This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference.In this paper, we propose a novel and efficient framework Diamante to boost the open-domain chatbot, where two kinds of human feedback (including explicit demonstration and implicit preference) are collected and leveraged.By asking annotators to select or amend the modelgenerated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs a Chinese chit-chat dataset.To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the generation-evaluation joint training.Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of pre-trained dialogue models.The overall engagingness of the previous state-ofthe-art model has been improved remarkably by 50% in Chinese open-domain conversations.
Semantic-conditioned Dual Adaptation for Cross-domain Query-based Visual Segmentation

Visual segmentation from language queries has attracted significant research interest.Despite the effectiveness, existing works require expensive labeling and suffer severe degradation when deployed to an unseen domain.In this paper, we investigate a novel task Cross-domain Query-based Visual Segmentation (CQVS), aiming to adapt the segmentation model from a labeled domain to a new unlabeled domain.The challenges of CQVS stem from three domain discrepancies: (1) multi-modal content shift, (2) uni-modal feature gap and (3) crossmodal relation bias.Existing domain adaptation methods fail to address them comprehensively and precisely (e.g. at pixel level), thus being suboptimal for CQVS.To overcome this limitation, we propose Semantic-conditioned Dual Adaptation (SDA), a novel framework to achieve precise feature-and relation-invariant across domains via a universal semantic structure.The SDA consists of two key components: Content-aware Semantic Modeling (CSM) and Dual Adaptive Branches (DAB).First, CSM introduces a common semantic space across domains to provide uniform guidance.Then, DAB seamlessly leverages this semantic information to develop a contrastive feature branch for category-wise pixel alignment, and design a reciprocal relation branch for relation enhancement via two complementary masks.Extensive experiments on three video benchmarks and three image benchmarks evidence the superiority of our approach over the state-of-the-arts.