Reviewerly/exHarmony huggingface dataset:
MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset

Relation extraction (RE) is a fundamental task in information extraction, whose extension to multilingual settings has been hindered by the lack of supervised resources comparable in size to large English datasets such as TACRED (Zhang et al., 2017).To address this gap, we introduce the MultiTACRED dataset, covering 12 typologically diverse languages from 9 language families, which is created by machine-translating TACRED instances and automatically projecting their entity annotations.We analyze translation and annotation projection quality, identify error categories, and experimentally evaluate fine-tuned pretrained mono-and multilingual language models in common transfer learning scenarios.Our analyses show that machine translation is a viable strategy to transfer RE instances, with native speakers judging more than 83% of the translated instances to be linguistically and semantically acceptable.We find monolingual RE model performance to be comparable to the English original for many of the target languages, and that multilingual models trained on a combination of English and target language data can outperform their monolingual counterparts.However, we also observe a variety of translation and annotation projection errors, both due to the MT systems and linguistic features of the target languages, such as pronoun-dropping, compounding and inflection, that degrade dataset quality and RE model performance.
Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation

In the real-world scenario, a longstanding goal of multilingual neural machine translation (MNMT) is that a single model can incrementally adapt to new language pairs without accessing previous training data.In this scenario, previous studies concentrate on overcoming catastrophic forgetting while lacking encouragement to learn new knowledge from incremental language pairs, especially when the incremental language is not related to the set of original languages.To better acquire new knowledge, we propose a knowledge transfer method that can efficiently adapt original MNMT models to diverse incremental language pairs.The method flexibly introduces the knowledge from an external model into original models, which encourages the models to learn new language pairs, completing the procedure of knowledge transfer.Moreover, all original parameters are frozen to ensure that translation qualities on original language pairs are not degraded.Experimental results show that our method can learn new knowledge from diverse language pairs incrementally meanwhile maintaining performance on original language pairs, outperforming various strong baselines in incremental learning for MNMT.
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models

Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pretrained network.Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS).We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning.We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.
Multi-Agent Language Learning: Symbolic Mapping

The study of emergent communication has long been devoted to coax neural network agents to learn a language sharing similar properties with human language.In this paper, we try to find a 'natural' way to help agents learn a compositional and symmetric language in complex settings like dialog games.Inspired by the theory that human language was originated from simple interactions, we hypothesize that language may evolve from simple tasks to difficult tasks.We propose a curriculum learning method called task transfer, and propose a novel architecture called symbolic mapping.We find that task transfer distinctly helps language learning in difficult tasks, and symbolic mapping promotes the effect.Further, we explore vocabulary expansion, and show that with the help of symbolic mapping, agents can easily learn to use new symbols when the environment becomes more complex.All in all, we find that a process from simplicity to complexity can serve as a natural way to help multi-agent language learning, and the proposed symbolic mapping is effective for this process.
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting

Existing studies addressing gender bias of pretrained language models, usually build a small gender-neutral data set and conduct a second phase pre-training on the model with such data.However, given the limited size and concentrated focus of the gender-neutral data, catastrophic forgetting would occur during secondphase pre-training.Forgetting information in the original training data may damage the model's downstream performance by a large margin.In this work, we empirically show that catastrophic forgetting occurs in such methods by evaluating them with general NLP tasks in GLUE.Then, we propose a new method, GEnder Equality Prompt (GEEP), to improve gender fairness of pre-trained models with less forgetting.GEEP freezes the pre-trained model and learns gender-related prompts with genderneutral data.Empirical results show that GEEP not only achieves SOTA performances on gender fairness tasks, but also forgets less and performs better on GLUE by a large margin.
On the Difference of BERT-style and CLIP-style Text Encoders

Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g., BERT, one of the representative models.Recently, contrastive language-image pretraining (CLIP) has also attracted attention, especially its vision models that achieve excellent performance on a broad range of vision tasks.However, few studies are dedicated to studying the text encoders learned by CLIP.In this paper, we analyze the difference between BERT-style and CLIP-style text encoders from three experiments: (i) general text understanding, (ii) vision-centric text understanding, and (iii) text-to-image generation.Experimental analyses show that although CLIP-style text encoders underperform BERTstyle ones for general text understanding tasks, they are equipped with a unique ability, i.e., synesthesia, for the cross-modal association, which is more similar to the senses of humans.Our code is released at https://github.com/zhjohnchan/probing-clip-dev.
Team:PULSAR at ProbSum 2023:PULSAR: Pre-training with Extracted Healthcare Terms for Summarising Patients’ Problems and Data Augmentation with Black-box Large Language Models

Medical progress notes play a crucial role in documenting a patient's hospital journey, including his or her condition, treatment plan, and any updates for healthcare providers.Automatic summarisation of a patient's problems in the form of a "problem list" can aid stakeholders in understanding a patient's condition, reducing workload and cognitive bias.BioNLP 2023 Shared Task 1A focuses on generating a list of diagnoses and problems from the provider's progress notes during hospitalisation.In this paper, we introduce our proposed approach to this task, which integrates two complementary components ‡ .One component employs large language models (LLMs) for data augmentation; the other is an abstractive summarisation LLM with a novel pre-training objective for generating the patients' problems summarised as a list.Our approach was ranked second among all submissions to the shared task.The performance of our model on the development and test datasets shows that our approach is more robust on unknown data, with an improvement of up to 3.1 points over the same size of the larger model.
Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond

Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability?Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays?In this study, we discover that the answers to both questions are no.This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently.Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-byword rerankers.In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative.In summarization experiments, without significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30% in BookSum paragraph-level dataset.
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation

In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation.Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient.Instead, our approach adopts a "coarse-to-fine" process, in which the video can be generated in parallel at the same granularity.A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames.This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap and makes it possible to generate all segments in parallel.To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation.Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26%) at the same hardware setting when generating 1024 frames.
Open-ended Long Text Generation via Masked Language Modeling

Pre-trained autoregressive (AR) language models such as BART and GPTs have dominated Open-ended Long Text Generation (Open-LTG).However, the AR nature will decrease the inference efficiency along with the increase of generation length, which hinder their application in Open-LTG.To improve inference efficiency, we alternatively explore the potential of the pre-trained masked language models (MLMs) along with a representative iterative non-autoregressive (NAR) decoding strategy for Open-LTG.Our preliminary study shows that pre-trained MLMs can merely generate short text and will collapse for long text modeling.To enhance the long text generation capability of MLMs, we introduce two simple yet effective strategies for the iterative NAR model: dynamic sliding window attention (DSWA) and linear temperature decay (LTD).It can alleviate long-distance collapse problems and achieve longer text generation with a flexible trade-off between performance and inference speedup.Experiments on the storytelling and multi-paragraph opinionated article writing tasks show that pre-trained MLMs can achieve more than 3 × → 13 × speedup with better performance than strong AR models.Our code is available at GitHub * .
Early Discovery of Disappearing Entities in Microblogs

We make decisions by reacting to changes in the real world, particularly the emergence and disappearance of impermanent entities such as restaurants, services, and events.Because we want to avoid missing out on opportunities or making fruitless actions after those entities have disappeared, it is important to know when entities disappear as early as possible.We thus tackle the task of detecting disappearing entities from microblogs where various information is shared timely.The major challenge is detecting uncertain contexts of disappearing entities from noisy microblog posts.To collect such disappearing contexts, we design timesensitive distant supervision, which utilizes entities from the knowledge base and time-series posts.Using this method, we actually build large-scale Twitter datasets of disappearing entities 1 for English and Japanese.To ensure robust detection in noisy environments, we refine pretrained word embeddings for the detection model on microblog streams in a timely manner.Experimental results on the Twitter datasets confirmed the effectiveness of the collected labeled data and refined word embeddings; the proposed method outperformed a baseline in terms of accuracy, and more than 70% of the detected disappearing entities in Wikipedia are discovered earlier than the update on Wikipedia, with the average lead-time is over one month.* Part of this work was carried out during the period the author was at the University of Tokyo.1 We release the datasets (tweet IDs) used in the experiments to promote reproducibility.URL
End-to-End Simultaneous Speech Translation with Differentiable Segmentation

End-to-end simultaneous speech translation (SimulST) outputs translation while receiving the streaming speech inputs (a.k.a.streaming speech translation), and hence needs to segment the speech inputs and then translate based on the current received speech.However, segmenting the speech inputs at unfavorable moments can disrupt the acoustic integrity and adversely affect the performance of the translation model.Therefore, learning to segment the speech inputs at those moments that are beneficial for the translation model to produce highquality translation is the key to SimulST.Existing SimulST methods, either using the fixedlength segmentation or external segmentation model, always separate segmentation from the underlying translation model, where the gap results in segmentation outcomes that are not necessarily beneficial for the translation process.In this paper, we propose Differentiable Segmentation (DiSeg) for SimulST to directly learn segmentation from the underlying translation model.DiSeg turns hard segmentation into differentiable through the proposed expectation training, enabling it to be jointly trained with the translation model and thereby learn translation-beneficial segmentation.Experimental results demonstrate that DiSeg achieves state-of-the-art performance and exhibits superior segmentation capability 1 .
Speech-to-Speech Translation for a Real-world Unwritten Language

We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems.We use English↔Taiwanese Hokkien as a case study, and present an endto-end solution from training data collection, modeling choices to benchmark dataset release.First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data.On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training.Finally, we release an S2ST benchmark set to facilitate future research in this field. 1 2
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics.Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human.Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective.However, their effectiveness at discerning system-level performance and quality of system outputs remain unclear.We present metric preference checklist as a framework to assess the effectiveness of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation.Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation.We show that automatic metrics provide a better guidance than human on discriminating system-level performance in Text Summarization and Controlled Generation tasks.We also show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly in Controlled Generation tasks. 1
Enhancing Hierarchical Text Classification through Knowledge Graph Integration

Hierarchical Text Classification (HTC) is an essential and challenging subtask of multi-label text classification with a taxonomic hierarchy.Recent advances in deep learning and pretrained language models have led to significant breakthroughs in the HTC problem.However, despite their effectiveness, these methods are often restricted by a lack of domain knowledge, which leads them to make mistakes in a variety of situations.Generally, when manually classifying a specific document to the taxonomic hierarchy, experts make inference based on their prior knowledge and experience.For machines to achieve this capability, we propose a novel Knowledge-enabled Hierarchical Text Classification model (K-HTC), which incorporates knowledge graphs into HTC.Specifically, K-HTC innovatively integrates knowledge into both the text representation and hierarchical label learning process, addressing the knowledge limitations of traditional methods.Additionally, a novel knowledge-aware contrastive learning strategy is proposed to further exploit the information inherent in the data.Extensive experiments on two publicly available HTC datasets show the efficacy of our proposed method, and indicate the necessity of incorporating knowledge graphs in HTC tasks.
BIC: Twitter Bot Detection with Text-Graph Interaction and Semantic Consistency

Twitter bots are automatic programs operated by malicious actors to manipulate public opinion and spread misinformation.Research efforts have been made to automatically identify bots based on texts and networks on social media.Existing methods only leverage texts or networks alone, and while few works explored the shallow combination of the two modalities, we hypothesize that the interaction and information exchange between texts and graphs could be crucial for holistically evaluating bot activities on social media.In addition, according to a recent survey (Cresci, 2020), Twitter bots are constantly evolving while advanced bots steal genuine users' tweets and dilute their malicious content to evade detection.This results in greater inconsistency across the timeline of novel Twitter bots, which warrants more attention.In light of these challenges, we propose BIC, a Twitter Bot detection framework with text-graph Interaction and semantic Consistency.In particular, BIC utilizes a textgraph focused approach to facilitate the two communication styles of social media that may trade useful data throughout the training cycle.In addition, given the stealing behavior of novel Twitter bots, BIC proposes to model semantic consistency in tweets based on attention weights while using it to augment the decision process.Extensive experiments demonstrate that BIC consistently outperforms state-of-theart baselines on two widely adopted datasets.Further analyses reveal that text-graph interactions and modeling semantic consistency are essential improvements and help combat bot evolution.
A Critical Evaluation of Evaluations for Long-form Question Answering

Long-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation.We perform the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices.We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices.We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer.Next, we examine automatic text generation metrics, finding that no existing metrics are predictive of human preference judgments.However, some metrics correlate with fine-grained aspects of answers (e.g., coherence).We encourage future work to move away from a single "overall score" of the answer and adopt a multi-faceted evaluation, targeting aspects such as factuality and completeness.We publicly release all of our annotations and code to spur future work into LFQA evaluation.
Limitations of Language Models in Arithmetic and Symbolic Induction

Recent work has shown that large pretrained Language Models (LMs) can not only perform remarkably well on a range of Natural Language Processing (NLP) tasks but also start improving on reasoning tasks such as arithmetic induction, symbolic manipulation, and commonsense reasoning with increasing size of models (Wei et al., 2022;Chowdhery et al., 2022).However, it is still unclear what the underlying capabilities of these LMs are.Surprisingly, we find that these models have limitations on certain basic symbolic manipulation tasks such as copy, reverse, and addition.When the total number of symbols or repeating symbols increases, the model performance drops quickly.We investigate the potential causes behind this phenomenon and examine a set of possible methods, including explicit positional markers, fine-grained computation steps, and LMs with callable programs.Experimental results show that none of these techniques can solve the simplest addition induction problem completely.In the end, we introduce LMs with tutor, which demonstrates every single step of teaching.LMs with tutor is able to deliver 100% accuracy in situations of OOD and repeating symbols, shedding new insights on the boundary of large LMs in induction.* The first two authors (Jing and Hong) contributed equally to this work.
Measuring and Mitigating Local Instability in Deep Neural Networks

Deep Neural Networks (DNNs) are becoming integral components of real world services relied upon by millions of users.Unfortunately, architects of these systems can find it difficult to ensure reliable performance as irrelevant details like random initialization can unexpectedly change the outputs of a trained system with potentially disastrous consequences.We formulate the model stability problem by studying how the predictions of a model change, even when it is retrained on the same data, as a consequence of stochasticity in the training process.For Natural Language Understanding (NLU) tasks, we find instability in predictions for a significant fraction of queries.We formulate principled metrics, like per-sample "label entropy" across training runs or within a single training run, to quantify this phenomenon.Intriguingly, we find that unstable predictions do not appear at random, but rather appear to be clustered in data-specific ways.We study dataagnostic regularization methods to improve stability and propose new data-centric methods that exploit our local stability estimates.We find that our localized data-specific mitigation strategy dramatically outperforms dataagnostic methods, and comes within 90% of the gold standard, achieved by ensembling, at a fraction of the computational cost.