Reviewerly/exHarmony huggingface dataset:
Transformer-based Entity Typing in Knowledge Graphs

We investigate the knowledge graph entity typing task which aims at inferring plausible entity types.In this paper, we propose a novel Transformer-based Entity Typing (TET) approach, effectively encoding the content of neighbors of an entity.More precisely, TET is composed of three different mechanisms: a local transformer allowing to infer missing types of an entity by independently encoding the information provided by each of its neighbors; a global transformer aggregating the information of all neighbors of an entity into a single long sequence to reason about more complex entity types; and a context transformer integrating neighbors content based on their contribution to the type inference through information exchange between neighbor pairs.Furthermore, TET uses information about class membership of types to semantically strengthen the representation of an entity.Experiments on two real-world datasets demonstrate the superior performance of TET compared to the state-of-the-art.
Benchmarking transfer learning approaches for sentiment analysis of Arabic dialect

Arabic has a widely varying collection of dialects.With the explosion of the use of social networks, the volume of written texts has remarkably increased.Most users express themselves using their own dialect.Unfortunately, many of these dialects remain under-studied due to the scarcity of resources.Researchers and industry practitioners are increasingly interested in analyzing users' sentiments.In this context, several approaches have been proposed, namely: traditional machine learning, deep learning transfer learning and more recently few-shot learning approaches.In this work, we compare their efficiency as part of the NADI competition to develop a country-level sentiment analysis model.Three models were beneficial for this sub-task: The first based on Sentence Transformer (ST) and achieve 43.23% on DEV set and 42.33% on TEST set, the second based on CAMeLBERT and achieve 54.00% on DEV set and 43.11% on TEST set and the third based on multi-dialect BERT model and achieve 66.72% on DEV set and 39.69% on TEST set.
Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction

Text error correction aims to correct the errors in text sequences such as those typed by humans or generated by speech recognition models.Previous error correction methods usually take the source (incorrect) sentence as encoder input and generate the target (correct) sentence through the decoder.Since the error rate of the incorrect sentence is usually low (e.g., 10%), the correction model can only learn to correct on limited error tokens but trivially copy on most tokens (correct tokens), which harms the effective training of error correction.In this paper, we argue that the correct tokens should be better utilized to facilitate effective training and then propose a simple yet effective masking strategy to achieve this goal.Specifically, we randomly mask out a part of the correct tokens in the source sentence and let the model learn to not only correct the original error tokens but also predict the masked tokens based on their context information.Our method enjoys several advantages: 1) it alleviates trivial copy; 2) it leverages effective training signals from correct tokens; 3) it is a plug-and-play module and can be applied to different models and tasks.Experiments on spelling error correction and speech recognition error correction on Mandarin datasets and grammar error correction on English datasets with both autoregressive and non-autoregressive generation models show that our method improves the correction accuracy consistently 1 .
A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification

We present a multilingual bag-of-entities model that effectively boosts the performance of zeroshot cross-lingual text classification by extending a multilingual pre-trained language model (e.g., M-BERT).It leverages the multilingual nature of Wikidata: entities in multiple languages representing the same concept are defined with a unique identifier.This enables entities described in multiple languages to be represented using shared embeddings.A model trained on entity features in a resource-rich language can thus be directly applied to other languages.Our experimental results on crosslingual topic classification (using the MLDoc and TED-CLDC datasets) and entity typing (using the SHINRA2020-ML dataset) show that the proposed model consistently outperforms state-of-the-art models.
On the Limitations of Reference-Free Evaluations of Generated Text

There is significant interest in developing evaluation metrics which accurately estimate the quality of generated text without the aid of a human-written reference text, which can be time consuming and expensive to collect or entirely unavailable in online applications.However, in this work, we demonstrate that these reference-free metrics are inherently biased and limited in their ability to evaluate generated text, and we argue that they should not be used to measure progress on tasks like machine translation or summarization.We show how reference-free metrics are equivalent to using one generation model to evaluate another, which has several limitations: (1) the metrics can be optimized at test time to find the approximate best-possible output, (2) they are inherently biased toward models which are more similar to their own, and (3) they can be biased against higher-quality outputs, including those written by humans.Therefore, we recommend that reference-free metrics should be used as diagnostic tools for analyzing and understanding model behavior instead of measures of how well models perform a task, in which the goal is to achieve as high of a score as possible. 1
Polyglot Prompt: Multilingual Multitask Prompt Training

This paper aims for a potential architectural improvement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. without any task/language-specific module?The benefit of achieving this could open new doors for future multilingual research, including allowing systems trained on low resources to be further assisted by other languages as well as other tasks.We approach this goal by developing a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages and tasks with multilingual prompt engineering.We performed a comprehensive evaluation of 6 tasks, namely topic classification, sentiment classification, named entity recognition, question answering, natural language inference, and summarization, covering 24 datasets and 49 languages.The experimental results demonstrated the efficacy of multilingual multitask prompt-based learning and led to inspiring observations.We also present an interpretable multilingual evaluation methodology and show how the proposed framework, multilingual multitask prompt training, works.We release all datasets prompted in the best setting and code.
Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario

People can acquire knowledge in an unsupervised manner by reading, and compose the knowledge to make novel combinations.In this paper, we investigate whether pretrained language models can perform compositional generalization in a realistic setting: recipe generation.We design the counterfactual recipe generation task, which asks models to modify a base recipe according to the change of an ingredient.This task requires compositional generalization at two levels: the surface level of incorporating the new ingredient into the base recipe, and the deeper level of adjusting actions related to the changing ingredient.We collect a large-scale recipe dataset in Chinese for models to learn culinary knowledge, and a subset of action-level fine-grained annotations for evaluation.We finetune pretrained language models on the recipe corpus, and use unsupervised counterfactual generation methods to generate modified recipes.Results show that existing models have difficulties in modifying the ingredients while preserving the original text style, and often miss actions that need to be adjusted.Although pretrained language models can generate fluent recipe texts, they fail to truly learn and use the culinary knowledge in a compositional way.Code and data are available at https://github.com/xxxiaol/counterfactual-recipe-generation.
Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese Hokkien

In natural language processing (NLP), codemixing (CM) is a challenging task, especially when the mixed languages include dialects.In Southeast Asian countries such as Singapore, Indonesia, and Malaysia, Hokkien-Mandarin is the most widespread code-mixed language pair among Chinese immigrants, and it is also common in Taiwan.However, dialects such as Hokkien often have a scarcity of resources and the lack of an official writing system, limiting the development of dialect CM research.In this paper, we propose a method to construct a Hokkien-Mandarin CM dataset to mitigate the limitation, overcome the morphological issue under the Sino-Tibetan language family, and offer an efficient Hokkien word segmentation method through a linguistics-based toolkit.Furthermore, we use our proposed dataset and employ transfer learning to train the XLM (cross-lingual language model) for translation tasks.To fit the code-mixing scenario, we adapt XLM slightly.We found that by using linguistic knowledge, rules, and language tags, the model produces good results on CM data translation while maintaining monolingual translation quality.
Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production

Mixture of Experts (MoE) models with conditional execution of sparsely activated layers has enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cut down the memory consumption significantly. While we achieve up to 26x speed-up in terms of throughput, we also reduce the model size almost to one eighth of the original 32-bit float model by quantizing expert weights into 4-bit integers. As a result, we are able to deploy 136x larger models with 27% less cost and significantly better quality with large scale MoE model deployment compared to the existing solutions. This enables a paradigm shift in deploying large scale multilingual MoE transformers models instead of distilling into dozens of smaller models per language or task.
Finding Skill Neurons in Pre-trained Transformer-based Language Models

Transformer-based pre-trained language models have demonstrated superior performance on various natural language processing tasks.However, it remains unclear how the skills required to handle these tasks distribute among model parameters.In this paper, we find that after prompt tuning for specific tasks, the activations of some neurons within pre-trained Transformers 1 are highly predictive of the task labels.We dub these neurons skill neurons and confirm they encode task-specific skills by finding that:(1) Skill neurons are crucial for handling tasks.Performances of pre-trained Transformers on a task significantly drop when corresponding skill neurons are perturbed.(2) Skill neurons are task-specific.Similar tasks tend to have similar distributions of skill neurons.Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than finetuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit.We also explore the applications of skill neurons, including accelerating Transformers with network pruning and building better transferability indicators.These findings may promote further research on understanding Transformers.
Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation

Prompt tuning, or the conditioning of a frozen pretrained language model (PLM) with soft prompts learned from data, has demonstrated impressive performance on a wide range of NLP tasks.However, prompt tuning requires a large training dataset to be effective and is outperformed by finetuning the entire PLM in data-scarce regimes.Previous work (Gu et al., 2022;Vu et al., 2022) proposed to transfer soft prompts pretrained on the source domain to the target domain.In this paper, we explore domain adaptation for prompt tuning, a problem setting where unlabeled data from the target domain are available during pretraining.We propose bOosting Prompt TunIng with doMain Adaptation (OPTIMA), which regularizes the decision boundary to be smooth around regions where source and target data distributions are similar.Extensive experiments demonstrate that OPTIMA significantly enhances the transferability and sample-efficiency of prompt tuning compared to strong baselines.Moreover, in few-shot settings, OPTIMA exceeds full-model tuning by a large margin.
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts

This work introduces a new multi-task, parameter-efficient language model (LM) tuning method that learns to transfer knowledge across different tasks via a mixture of soft prompts-small prefix embedding vectors pretrained for different tasks.Our method, called ATTEMPT (ATTEntional Mixtures of Prompt Tuning), obtains source prompts as encodings of large-scale source tasks into a small number of parameters and trains an attention module to interpolate the source prompts and a newly initialized target prompt for every instance in the target task.During training, only the target task prompt and the attention weights, which are shared between tasks in multi-task training, are updated, while the original LM and source prompts are intact.ATTEMPT is highly parameter-efficient (e.g., updates 2,300 times fewer parameters than full fine-tuning), while achieving high task performance using knowledge from high-resource tasks.Moreover, it is modular using pre-trained soft prompts and can flexibly add or remove source prompts for effective knowledge transfer.Our experimental results across 21 diverse NLP datasets show that ATTEMPT significantly outperforms prompt tuning and outperforms or matches fully finetuned or other parameter-efficient tuning approaches that use over ten times more parameters.Finally, ATTEMPT outperforms previous work in few-shot learning settings. 1
Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt

Prompt-based tuning has been proven effective for pretrained language models (PLMs).While most of the existing work focuses on the monolingual prompts, we study the multilingual prompts for multilingual PLMs, especially in the zero-shot cross-lingual setting.To alleviate the effort of designing different prompts for multiple languages, we propose a novel model that uses a unified prompt for all languages, called UniPrompt.Different from the discrete prompts and soft prompts, the unified prompt is model-based and languageagnostic.Specifically, the unified prompt is initialized by a multilingual PLM to produce language-independent representation, after which is fused with the text input.During inference, the prompts can be pre-computed so that no extra computation cost is needed.To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model's transferability across languages.Extensive experiments show that our proposed methods can significantly outperform the strong baselines across different languages.We release data and code to facilitate future research 1 .
Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering

We propose a simple refactoring of multichoice question answering (MCQA) tasks as a series of binary classifications.The MCQA task is generally performed by scoring each (question, answer) pair normalized over all the pairs, and then selecting the answer from the pair that yield the highest score.For n answer choices, this is equivalent to an n-class classification setup where only one class (true answer) is correct.We instead show that classifying (question, true answer) as positive instances and (question, false answer) as negative instances is significantly more effective across various models and datasets.We show the efficacy of our proposed approach in different tasks -abductive reasoning, commonsense question answering, science question answering, and sentence completion.Our DeBERTa binary classification model reaches the top or close to the top performance on public leaderboards for these tasks.
ScienceWorld: Is your Agent Smarter than a 5th Grader?

We present SCIENCEWORLD, a benchmark to test agents' scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curriculum.Despite the transformer-based progress seen in question-answering and scientific text processing, we find that current models cannot reason about or explain learned science concepts in novel contexts.For instance, models can easily answer what the conductivity of a known material is but struggle when asked how they would conduct an experiment in a grounded environment to find the conductivity of an unknown material.This begs the question of whether current models are simply retrieving answers by way of seeing a large number of similar examples or if they have learned to reason about concepts in a reusable manner.We hypothesize that agents need to be grounded in interactive environments to achieve such reasoning capabilities.Our experiments provide empirical evidence supporting this hypothesisshowing that a 1.5 million parameter agent trained interactively for 100k steps outperforms a 11 billion parameter model statically trained for scientific question-answering and reasoning from millions of expert demonstrations.12
Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarization

Sentence summarization shortens given texts while maintaining core contents of the texts.Unsupervised approaches have been studied to summarize texts without human-written summaries.However, recent unsupervised models are extractive, which remove words from texts and thus they are less flexible than abstractive summarization.In this work, we devise an abstractive model based on reinforcement learning without ground-truth summaries.We formulate the unsupervised summarization based on the Markov decision process with rewards representing the summary quality.To further enhance the summary quality, we develop a multi-summary learning mechanism that generates multiple summaries with varying lengths for a given text, while making the summaries mutually enhance each other.Experimental results show that the proposed model substantially outperforms both abstractive and extractive models, yet frequently generating new words not contained in input texts.
Automatic Comment Generation for Chinese Student Narrative Essays

Automatic essay evaluation can help reduce teachers' workload and enable students to refine their works rapidly.Previous studies focus mainly on giving discrete scores for either the holistic quality or several distinct traits.However, real-world teachers usually provide detailed comments in natural language, which are more informative than single scores.In this paper, we present the comment generation task, which aims to generate comments for specified segments from given student narrative essays.To tackle this task, we propose a planningbased generation model, which first plans a sequence of keywords, and then expands these keywords into a complete comment.To improve the correctness and informativeness of generated comments, we adopt two following techniques: (1) training an error correction module to filter out incorrect keywords, and (2) recognizing fine-grained structured features from source essays to enrich the keywords.To support the evaluation of the task, we collect a human-written Chinese dataset, which contains 22,399 essay-comment pairs.Extensive experiments show that our model outperforms strong baselines significantly.Moreover, we exert explicit control on our model to generate comments to describe the strengths or weaknesses of inputs with a 91% success rate.We deploy the model at http: //coai.cs.tsinghua.
Combining WordNet and Word Embeddings in Data Augmentation for Legal Texts

Creating balanced labeled textual corpora for complex tasks, like legal analysis, is a challenging and expensive process that often requires the collaboration of domain experts.To address this problem, we propose a data augmentation method based on the combination of GloVe word embeddings and the WordNet ontology.We present an example of application in the legal domain, specifically on decisions of the Court of Justice of the European Union.Our evaluation with human experts confirms that our method is more robust than the alternatives.
SHARE: a System for Hierarchical Assistive Recipe Editing

The large population of home cooks with dietary restrictions is under-served by existing cooking resources and recipe generation models.To help them, we propose the task of controllable recipe editing: adapt a base recipe to satisfy a user-specified dietary constraint.This task is challenging, and cannot be adequately solved with human-written ingredient substitution rules or existing end-to-end recipe generation models.We tackle this problem with SHARE: a System for Hierarchical Assistive Recipe Editing, which performs simultaneous ingredient substitution before generating natural-language steps using the edited ingredients.By decoupling ingredient and step editing, our step generator can explicitly integrate the available ingredients.Experiments on the novel RecipePairs dataset-83K pairs of similar recipes where each recipe satisfies one of seven dietary constraints-demonstrate that SHARE produces convincing, coherent recipes that are appropriate for a target dietary constraint.We further show through human evaluations and real-world cooking trials that recipes edited by SHARE can be easily followed by home cooks to create appealing dishes. Alfredo SauceButter, Flour, Cream, Garlic, Parmesan, Parsley 1) Melt butter in saucepan and add flour.2) Add garlic and saute.3) Add cream, simmer.Add cheese and melt... Dairy-Free "Alfredo Sauce"Olive Oil, Cashews, Nutritional Yeast, Flour, Garlic, Parsley 1) Soak cashews overnight.2) Heat olive oil in pan.Whisk in flour to make roux.3) Add roux, cashews, garlic, and yeast to blender... Butter, Flour, Cream, Garlic, Parmesan, Parsley +Olive Oil +Cashews +Nutritional Yeast User provides dietary constraint (Dairy-Free) and base recipe: System makes ingredient substitutions System re-writes recipe directions Base (Tofu Tikka Masala) E2E-P (Dairy-Free) MT (Dairy-Free) SHARE (Dairy-Free) Chili powder, Coriander, Garlic, Ginger, Jalapeno chile, Lemon juice, Onion, Salt, Soymilk, Turmeric, Vegetable oil, Yogurt Coriander, Garlic, Ginger, Onion, Salt, Turmeric, Vegetable oil, Bean, Cinnamon, Cumin, Pepper, Tomato, Water Chili powder, Jalapeno chile, Lemon juice, Soymilk, Yogurt Chili powder, Coriander, Garlic, Ginger, Lemon juice, Onion, Salt, Turmeric, Vegetable oil, Cumin, Pepper, Tofu, Water Jalapeno chile, Soymilk, Yogurt Coriander, Garlic, Ginger, Onion, Salt, Turmeric, Vegetable oil, Coconut milk, Cumin, Curry, Masala, Pea, Pepper, Rice, Shrimp, Tomato, Water Chili powder, Jalapeno chile, Lemon juice, Soymilk, Yogurt