Reviewerly/exHarmony huggingface dataset:
SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning

Multi-hop knowledge graph (KG) reasoning has been widely studied in recent years to provide interpretable predictions on missing links with evidential paths. Most previous works use reinforcement learning (RL) based methods that learn to navigate the path towards the target entity. However, these methods suffer from slow and poor convergence, and they may fail to infer a certain path when there is a missing edge along the path. Here we present SQUIRE, the first Sequence-to-sequence based multi-hop reasoning framework, which utilizes an encoder-decoder Transformer structure to translate the query to a path. Our framework brings about two benefits: (1) It can learn and predict in an end-to-end fashion, which gives better and faster convergence; (2) Our transformer model does not rely on existing edges to generate the path, and has the flexibility to complete missing edges along the path, especially in sparse KGs. Experiments on standard and sparse KGs show that our approach yields significant improvement over prior methods, while converging 4x-7x faster.
Abstract Visual Reasoning with Tangram Shapes

We introduce KILOGRAM, a resource for studying abstract visual reasoning in humans and machines.Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources.It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels.We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models.We observe that pre-trained weights demonstrate limited abstract reasoning, which dramatically improves with fine-tuning.We also observe that explicitly describing parts aids abstract reasoning for both humans and models, especially when jointly encoding the linguistic and visual inputs.
BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation

WARNING: This paper contains examples that are offensive in nature.Automatic evaluation metrics are crucial to the development of generative systems.In recent years, pre-trained language model (PLM) based metrics, such as BERTScore (Zhang et al., 2020), have been commonly adopted in various generation tasks.However, it has been demonstrated that PLMs encode a range of stereotypical societal biases, leading to a concern on the fairness of PLMs as metrics.To that end, this work presents the first systematic study on the social bias in PLM-based metrics.We demonstrate that popular PLM-based metrics exhibit significantly higher social bias than traditional metrics on 6 sensitive attributes, namely race, gender, religion, physical appearance, age, and socioeconomic status.In-depth analysis suggests that choosing paradigms (matching, regression, or generation) of the metric has a greater impact on fairness than choosing PLMs.In addition, we develop debiasing adapters that are injected into PLM layers, mitigating bias in PLM-based metrics while retaining high performance for evaluating text generation.
Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking

In information retrieval (IR), candidate set pruning has been commonly used to speed up twostage relevance ranking.However, such an approach lacks accurate error control and often trades accuracy against computational efficiency in an empirical fashion, missing theoretical guarantees.In this paper, we propose the concept of certified error control of candidate set pruning for relevance ranking, which means that the test error after pruning is guaranteed to be controlled under a user-specified threshold with high probability.Both in-domain and outof-domain experiments show that our method successfully prunes the first-stage retrieved candidate sets to improve the second-stage reranking speed while satisfying the pre-specified accuracy constraints in both settings.For example, on MS MARCO Passage v1, our method reduces the average candidate set size from 1000 to 27, increasing reranking speed by about 37 times, while keeping MRR@10 greater than a pre-specified value of 0.38 with about 90% empirical coverage.In contrast, empirical baselines fail to meet such requirements.Code and
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment

The instruction learning paradigm-where a model learns to perform new tasks from task descriptions alone-has become popular in research on general-purpose models.The capabilities of large transformer models as instruction learners, however, remain poorly understood.We use a controlled synthetic environment to characterize such capabilities.Specifically, we use the task of deciding whether a given string matches a regular expression (viewed as an instruction) to identify properties of tasks, instructions, and instances that make instruction learning challenging.For instance, we find that our model, a fine-tuned T5-based text2text transformer, struggles with large regular languages, suggesting that less precise instructions are challenging for models.Instruction executions that require tracking longer contexts of prior steps are also difficult.We use our findings to systematically construct a challenging instruction learning dataset, which we call Hard RegSet.Fine-tuning on Hard RegSet, our large transformer learns to correctly interpret (with at least 90% accuracy) only 65.6% of test instructions, and 11%-24% of the instructions in out-of-distribution generalization settings.We thus propose Hard RegSet as a challenging instruction learning dataset, and a controlled environment for studying instruction learning. 1
Garden Path Traversal in GPT-2

In recent years, large-scale transformer decoders such as the GPT-x family of models have become increasingly popular.Studies examining the behavior of these models tend to focus only on the output of the language modeling head and avoid analysis of the internal states of the transformer decoder.In this study, we present a collection of methods to analyze the hidden states of GPT-2 and use the model's navigation of garden path sentences as a case study.To enable this, we compile the largest currently available dataset of garden path sentences.We show that Manhattan distances and cosine similarities provide more reliable insights compared to established surprisal methods that analyze next-token probabilities computed by a language modeling head.Using these methods, we find that negating tokens have minimal impacts on the model's representations for unambiguous forms of sentences with ambiguity solely over what the object of a verb is, but have a more substantial impact of representations for unambiguous sentences whose ambiguity would stem from the voice of a verb.Further, we find that analyzing the decoder model's hidden states reveals periods of ambiguity that might conclude in a garden path effect but happen not to, whereas surprisal analyses routinely miss this detail.
How Large Language Models are Transforming Machine-Paraphrase Plagiarism

The recent success of large language models for text generation poses a severe threat to academic integrity, as plagiarists can generate realistic paraphrases indistinguishable from original work.However, the role of large autoregressive transformers in generating machineparaphrased plagiarism and their detection is still developing in the literature.This work explores T5 and GPT-3 for machine-paraphrase generation on scientific articles from arXiv, student theses, and Wikipedia.We evaluate the detection performance of six automated solutions and one commercial plagiarism detection software and perform a human study with 105 participants regarding their detection performance and the quality of generated examples.Our results suggest that large models can rewrite text humans have difficulty identifying as machine-paraphrased (53% mean acc.).Human experts rate the quality of paraphrases generated by GPT-3 as high as original texts (clarity 4.0/5, fluency 4.2/5, coherence 3.8/5).The best-performing detection model (GPT-3) achieves a 66% F1-score in detecting paraphrases.We make our code, data, and findings publicly available for research purposes.
POTATO: The Portable Text Annotation Tool

We present POTATO, the Portable text annotation tool, a free, fully open-sourced annotation system that 1) supports labeling many types of text and multimodal data; 2) offers easy-to-configure features to maximize the productivity of both deployers and annotators (convenient templates for common ML/NLP tasks, active learning, keypress shortcuts, keyword highlights, tooltips); and 3) supports a high degree of customization (editable UI, inserting pre-screening questions, attention and qualification tests).Experiments over two annotation tasks suggest that POTATO improves labeling speed through its specially-designed productivity features, especially for long documents and complex tasks.POTATO is available at https://github.com/davidjurgens/potato and will continue to be updated.
A Unified Encoder-Decoder Framework with Entity Memory

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks.We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation.Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead.In this work, we propose an Encoder-Decoder framework with an entity Memory, namely EDMem.The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters.To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory.EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks.Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.
Efficient Nearest Neighbor Emotion Classification with BERT-whitening

Retrieval-based methods have been proven effective in many NLP tasks.Previous methods use representations from the pre-trained model for similarity search directly.However, the sentence representations from the pre-trained model like BERT perform poorly in retrieving semantically similar sentences, resulting in poor performance of the retrieval-based methods.In this paper, we propose KNN-EC, a simple and efficient non-parametric emotion classification (EC) method using nearest neighbor retrieval.We use BERT-whitening to get better sentence semantics, ensuring that nearest neighbor retrieval works.Meanwhile, BERTwhitening can also reduce memory storage of datastore and accelerate retrieval speed, solving the efficiency problem of the previous methods.KNN-EC average improves the pre-trained model by 1.17 F1-macro on two emotion classification datasets.
FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering

Generative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passages and combining their information.However, generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoder beam search, and their generated output often suffers from hallucinations.We propose to extend transformer encoders with the ability to fuse information from multiple passages, using global representation to provide cross-sample attention over all tokens across samples.Furthermore, we propose an alternative answer span probability calculation to better aggregate answer scores in the global space of all samples.Using our proposed method, we outperform the current stateof-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference, and 4.4 Exact Match on We-bQuestions dataset.When coupled with synthetic data augmentation, we outperform larger models on the TriviaQA dataset as well.The latency and parameter savings of our method make it particularly attractive for open-domain question answering, as these models are often compute-intensive.
Is a Question Decomposition Unit All We Need?

Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks.With the growing number of new benchmarks, we build bigger and more complex LMs.However, building new LMs may not be an ideal option owing to the cost, time and environmental impact associated with it.We explore an alternative route: can we modify data by expressing it in terms of the model's strengths, so that a question becomes easier for models to answer?We investigate if humans can decompose a hard question into a set of simpler questions that are relatively easier for models to solve.We analyze a range of datasets involving various forms of reasoning and find that it is indeed possible to significantly improve model performance (24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) via decomposition.Our approach provides a viable option to involve people in NLP research in a meaningful way.Our findings indicate that Human-in-the-loop Question Decomposition (HQD) can potentially provide an alternate path to building large LMs 1 .
SNaC: Coherence Error Detection for Narrative Summarization

Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks.A long summary that appropriately covers the facets of that text must also present a coherent narrative, but current automatic and human evaluation methods fail to identify gaps in coherence.In this work, we introduce SNAC, a narrative coherence evaluation framework for fine-grained annotations of long summaries.We develop a taxonomy of coherence errors in generated narrative summaries and collect spanlevel annotations for 6.6k sentences across 150 book and movie summaries.Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowdworkers.Furthermore, we show that the collected annotations allow us to benchmark past work in coherence modeling and train a strong classifier for automatically localizing coherence errors in generated summaries.Finally, our SNAC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and posthoc summary correction.
Abstractive Summarization Guided by Latent Hierarchical Document Structure

Sequential abstractive neural summarizers often do not use the underlying structure in the input article or dependencies between the input sentences.This structure is essential to integrate and consolidate information from different parts of the text.To address this shortcoming, we propose a hierarchy-aware graph neural network (HierGNN) which captures such dependencies through three main steps: 1) learning a hierarchical document structure through a latent structure tree learned by a sparse matrixtree computation; 2) propagating sentence information over this structure using a novel message-passing node propagation mechanism to identify salient information; 3) using graphlevel attention to concentrate the decoder on salient information.Experiments confirm Hi-erGNN improves strong sequence models such as BART, with a 0.55 and 0.75 margin in average ROUGE-1/2/L for CNN/DM and XSum.Further human evaluation demonstrates that summaries produced by our model are more relevant and less redundant than the baselines, into which HierGNN is incorporated.We also find HierGNN synthesizes summaries by fusing multiple source sentences more, rather than compressing a single source sentence, and that it processes long inputs more effectively. 1
Towards Arabic Sentence Simplification via Classification and Generative Approaches

This paper presents an attempt to build a Modern Standard Arabic (MSA) sentence-level simplification system.We experimented with sentence simplification using two approaches: (i) a classification approach leading to lexical simplification pipelines which use Arabic-BERT, a pre-trained contextualised model, as well as a model of fastText word embeddings; and (ii) a generative approach, a Seq2Seq technique by applying a multilingual Text-to-Text Transfer Transformer mT5.We developed our training corpus by aligning the original and simplified sentences from the internationally acclaimed Arabic novel "Saaq al-Bambuu".We evaluate effectiveness of these methods by comparing the generated simple sentences to the target simple sentences using the BERTScore evaluation metric.The simple sentences produced by the mT5 model achieve P 0.72, R 0.68 and F-1 0.70 via BERTScore, while, combining Arabic-BERT and fastText achieves P 0.97, R 0.97 and F-1 0.97.In addition, we report a manual error analysis for these experiments.
ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts

Systems that automatically define unfamiliar terms hold the promise of improving the accessibility of scientific texts, especially for readers who may lack prerequisite background knowledge.However, current systems assume a single "best" description per concept, which fails to account for the many ways a concept can be described.We present ACCoRD, an end-to-end system tackling the novel task of generating sets of descriptions of scientific concepts.Our system takes advantage of the myriad ways a concept is mentioned across the scientific literature to produce distinct, diverse descriptions of target concepts in terms of different reference concepts.In a user study, we find that users prefer (1) descriptions produced by our end-to-end system, and (2) multiple descriptions to a single "best" description.We release the ACCoRD corpus which includes 1,275 labeled contexts and 1,787 expert-authored concept descriptions to support research on our task.
Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning

Providing Emotional Support (ES) to soothe people in emotional distress is an essential capability in social interactions.Most existing research on building ES conversation systems only considers single-turn interactions with users, which is over-simplified.In comparison, multi-turn ES conversation systems can provide ES more effectively, but face several new technical challenges, including: i) how to conduct support strategy planning that could lead to the best supporting effects; ii) how to dynamically model the user's state.In this paper, we propose a novel system named MultiESC to address these issues.For strategy planning, drawing inspiration from the A* search algorithm, we propose lookahead heuristics to estimate the future user feedback after using particular strategies, which helps to select strategies that can lead to the best long-term effects.For user state modeling, MultiESC focuses on capturing users' subtle emotional expressions and understanding their emotion causes.Extensive experiments show that MultiESC significantly outperforms competitive baselines in both strategy planning and dialogue generation.Our codes are available at https: //github.com/lwgkzl/MultiESC.
Unsupervised Tokenization Learning

In the presented study, we discover that the socalled "transition freedom" metric appears superior for unsupervised tokenization purposes in comparison to statistical metrics such as mutual information and conditional probability, providing F-measure scores in range from 0.71 to 1.0 across explored multilingual corpora.We find that different languages require different offshoots of that metric (such as derivative, variance, and "peak values") for successful tokenization.Larger training corpora do not necessarily result in better tokenization quality, while compressing the models by eliminating statistically weak evidence tends to improve performance.The proposed unsupervised tokenization technique provides quality better than or comparable to lexicon-based ones, depending on the language.
Backdoor Attacks in Federated Learning by Rare Embeddings and Gradient Ensembling

Recent advances in federated learning have demonstrated its promising capability to learn on decentralized datasets.However, a considerable amount of work has raised concerns due to the potential risks of adversaries participating in the framework to poison the global model for an adversarial purpose.This paper investigates the feasibility of model poisoning for backdoor attacks through rare word embeddings of NLP models.In text classification, less than 1% of adversary clients suffices to manipulate the model output without any drop in the performance on clean sentences.For a less complex dataset, a mere 0.1% of adversary clients is enough to poison the global model effectively.We also propose a technique specialized in the federated learning scheme called Gradient Ensemble, which enhances the backdoor performance in all our experimental settings.
CAraNER: The COVID-19 Arabic Named Entity Corpus

Named Entity Recognition (NER) is a wellknown problem for the natural language processing (NLP) community.It is a key component of different NLP applications, including information extraction, question answering, and information retrieval.In the literature, there are several Arabic NER datasets with different named entity tags; however, due to data and concept drift, we are always in need of new data for NER and other NLP applications.In this paper, first, we introduce Wassem, a web-based annotation platform for Arabic NLP applications.Wassem can be used to manually annotate textual data for a variety of NLP tasks: text classification, sequence classification, and word segmentation.Second, we introduce the COVID-19 Arabic Named Entities Recognition (CAraNER) dataset extracted from the Arabic Newspaper COVID-19 Corpus (AraNPCC).CAraNER has 55,389 tokens distributed over 1,278 sentences randomly extracted from Saudi Arabian newspaper articles published during 2019, 2020, and 2021.The dataset is labeled by five annotators with five named-entity tags, namely: Person, Title, Location, Organization, and Miscellaneous.The CAraNER corpus is available for download for free.We evaluate the corpus by finetuning four BERT-based Arabic language models on the CAraNER corpus.The best model was AraBERTv0.2-largewith 0.86 for the F1 macro measure.
Self-supervised Graph Masking Pre-training for Graph-to-Text Generation

Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text (G2T) generation by processing the linearised version of a graph.However, the linearisation is known to ignore the structural information.Additionally, PLMs are typically pre-trained on free text which introduces domain mismatch between pre-training and downstream G2T generation tasks.To address these shortcomings, we propose graph masking pre-training strategies that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model.When used with a pre-trained T5, our approach achieves new state-of-the-art results on WebNLG+2020 and EventNarrative G2T generation datasets.Our method also shows to be very effective in the low-resource setting. 1