Reviewerly/exHarmony huggingface dataset:
Uncovering Surprising Event Boundaries in Narratives

When reading stories, people can naturally identify sentences in which a new event starts, i.e., event boundaries, using their knowledge of how events typically unfold, but a computational model to detect event boundaries is not yet available.We characterize and detect sentences with expected or surprising event boundaries in an annotated corpus of short diary-like stories, using a model that combines commonsense knowledge and narrative flow features with a RoBERTa classifier.Our results show that, while commonsense and narrative features can help improve performance overall, detecting event boundaries that are more subjective remains challenging for our model.We also find that sentences marking surprising event boundaries are less likely to be causally related to the preceding sentence, but are more likely to express emotional reactions of story characters, compared to sentences with no event boundary.
Sentence-Level Resampling for Named Entity Recognition

As a fundamental task in natural language processing, named entity recognition (NER) aims to locate and classify named entities in unstructured text.However, named entities are always the minority among all tokens in the text.This data imbalance problem presents a challenge to machine learning models as their learning objective is usually dominated by the majority of non-entity tokens.To alleviate data imbalance, we propose a set of sentence-level resampling methods where the importance of each training sentence is computed based on its tokens and entities.We study the generalizability of these resampling methods on a wide variety of NER models (CRF, Bi-LSTM, and BERT) across corpora from diverse domains (general, social, and medical texts).Extensive experiments show that the proposed methods improve performance of the evaluated NER models especially on small corpora, frequently outperforming sub-sentence-level resampling, data augmentation, and special loss functions such as focal and Dice loss. 1
Learning to Selectively Learn for Weakly Supervised Paraphrase Generation with Model-based Reinforcement Learning

Paraphrase generation is an important language generation task attempting to interpret user intents and systematically generate new phrases of identical meanings to the given ones.However, the effectiveness of paraphrase generation is constrained by the access to the golden labeled data pairs where both the amount and the quality of the training data pairs are restricted.In this paper, we propose a new weakly supervised paraphrase generation approach that extends the success of a recent work that leverages reinforcement learning (RL) for effective model training with data selection.While data selection is privileged for the target task which has noisy data, developing a reinforced selective learning regime faces several unresolved challenges.In this paper, we carry on important discussions about the above problem and present a new model that could partially overcome the discussed issues with a model-based planning feature and a reward normalization feature.We perform extensive evaluation on four weakly supervised paraphrase generation tasks where the results show that our method could significantly improve the state-of-the-art performance on the evaluation datasets.
Methods for Estimating and Improving Robustness of Language Models

Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surfacelevel textual relations over full semantic complexity of the problem.This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain.We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models.Based on these findings, we present future research directions towards enhancing the robustness of LLMs.
SU-NLP at SemEval-2022 Task 11: Complex Named Entity Recognition with Entity Linking

This paper describes the system proposed by Sabancı University Natural Language Processing Group in the SemEval-2022 MultiCoNER task.We developed an unsupervised entity linking pipeline that detects potential entity mentions with the help of Wikipedia and also uses the corresponding Wikipedia context to help the classifier in finding the named entity type of that mention.The proposed pipeline significantly improved the performance, especially for complex entities in low-context settings.
TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation

Many scientific papers such as those in arXiv and PubMed data collections have abstracts with varying lengths of 50-1000 words and average length of approximately 200 words, where longer abstracts typically convey more information about the source paper.Up to recently, scientific summarization research has typically focused on generating short, abstractlike summaries following the existing datasets used for scientific summarization.In domains where the source text is relatively long-form, such as in scientific documents, such summary is not able to go beyond the general and coarse overview and provide salient information from the source document.The recent interest to tackle this problem motivated curation of scientific datasets, arXiv-Long and PubMed-Long, containing human-written summaries of 400-600 words, hence, providing a venue for research in generating long/extended summaries.Extended summaries facilitate a faster read while providing details beyond coarse information.In this paper, we propose TSTR, an extractive summarizer that utilizes the introductory information of documents as pointers to their salient information.The evaluations on two existing large-scale extended summarization datasets indicate statistically significant improvement in terms of ROUGE and average ROUGE (F1) scores (except in one case) as compared to strong baselines and state-of-theart.Comprehensive human evaluations favor our generated extended summaries in terms of cohesion and completeness.
ANTS: A Framework for Retrieval of Text Segments in Unstructured Documents

Text segmentation and extraction from unstructured documents can provide business researchers with a wealth of new information on firms and their behaviors.However, the most valuable text is often difficult to extract consistently due to substantial variations in how content can appear from document to document.Thus, the most successful way to extract this content has been through costly crowdsourcing and training of manual workers.We propose the Assisted Neural Text Segmentation (ANTS) framework to identify pertinent text in unstructured documents from a small set of labeled examples.ANTS leverages deep learning and transfer learning architectures to empower researchers to identify relevant text with minimal manual coding.Using a real world sample of accounting documents, we identify targeted sections 96% of the time using only 5 training examples.
OSU at SigMorphon 2022: Analogical Inflection With Rule Features

OSU's inflection system is a transformer whose input is augmented with an analogical exemplar showing how to inflect a different word into the target cell.In addition, alignment-based heuristic features indicate how well the exemplar is likely to match the output.OSU's scores substantially improve over the baseline transformer for instances where an exemplar is available, though not quite matching the challenge winner.In Part 2, the system shows a tendency to over-apply the majority pattern in English, but not Arabic.
A Versatile Adaptive Curriculum Learning Framework for Task-oriented Dialogue Policy Learning

Training a deep reinforcement learning-based dialogue policy with brute-force random sampling is costly.A new training paradigm was proposed to improve learning performance and efficiency by combining curriculum learning.However, attempts in the field of dialogue policy are very limited due to the lack of reliable evaluation of difficulty scores of dialogue tasks and the high sensitivity to the mode of progression through dialogue tasks.In this paper, we present a novel versatile adaptive curriculum learning (VACL) framework, which presents a substantial step toward applying automatic curriculum learning on dialogue policy tasks.It supports evaluating the difficulty of dialogue tasks only using the learning experiences of dialogue policy and skip-level selection according to their learning needs to maximize the learning efficiency.Moreover, an attractive feature of VACL is the construction of a generic, elastic global curriculum while training a good dialogue policy that could guide different dialogue policy learning without extra effort on re-training.The superiority and versatility of VACL are validated on three public dialogue datasets.
akaBERT at SemEval-2022 Task 6: An Ensemble Transformer-based Model for Arabic Sarcasm Detection

Due to the widespread usage of social media sites and the enormous number of users who utilize irony implicit words in most of their tweets and posts, it has become necessary to detect sarcasm, which strongly influences understanding and analyzing the crowd's opinions.Detecting sarcasm is difficult due to the nature of sarcastic tweets, which vary based on the topic, region, the user's attitude, culture, terminologies, and other criteria.In addition to these difficulties, detecting sarcasm in Arabic has its challenges due to its complexities, such as being morphologically rich, having many different dialects, and having low resources.In this research, we present our submission of (iSarcasmEval) sub-task A of the shared task on SemEval 2022.In Sub-task A; we determine whether the tweets are sarcastic or non-sarcastic.We implemented different approaches based on Transformers.First, we fine-tuned the AraBERT, MARABERT, and AraELECTRA.One of the challenges that faced us was that the data was not balanced.Non-sarcastic data is much more than sarcastic.We used data augmentation techniques to balance the two classes, significantly affecting the performance.The performance F1 score of the three models was 87%, 90%, and 91%, respectively.Then we boosted the three models by developing an ensemble model based on hard voting.The final performance F1 Score was 93%.
Grounding in social media: An approach to building a chit-chat dialogue model

Building open-domain dialogue systems capable of rich human-like conversational ability is one of the fundamental challenges in language generation.However, even with recent advancements in the field, existing open-domain generative models fail to capture and utilize external knowledge, leading to repetitive or generic responses to unseen utterances.Current work on knowledge-grounded dialogue generation primarily focuses on persona incorporation or searching a fact-based structured knowledge source such as Wikipedia.Our method takes a broader and simpler approach, which aims to improve the raw conversation ability of the system by mimicking the human response behavior through casual interactions found on social media.Utilizing a joint retriever-generator setup, the model queries a large set of filtered comment data from Reddit to act as additional context for the seq2seq generator.Automatic and human evaluations on open-domain dialogue datasets demonstrate the effectiveness of our approach.
Using Paraphrases to Study Properties of Contextual Embeddings

We use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT.Because paraphrases naturally encode consistent word and phrase semantics, they provide a unique lens for investigating properties of embeddings.Using the Paraphrase Database's alignments, we study words within paraphrases as well as phrase representations.We find that contextual embeddings effectively handle polysemous words, but give synonyms surprisingly different representations in many cases.We confirm previous findings that BERT is sensitive to word order, but find slightly different patterns than prior work in terms of the level of contextualization across BERT's layers.
FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction

This paper presents FAMIE, a comprehensive and efficient active learning (AL) toolkit for multilingual information extraction.FAMIE is designed to address a fundamental problem in existing AL frameworks where annotators need to wait for a long time between annotation batches due to the time-consuming nature of model training and data selection at each AL iteration.This hinders the engagement, productivity, and efficiency of annotators.Based on the idea of using a small proxy network for fast data selection, we introduce a novel knowledge distillation mechanism to synchronize the proxy network with the main large model (i.e., BERT-based) to ensure the appropriateness of the selected annotation examples for the main model.Our AL framework can support multiple languages.The experiments demonstrate the advantages of FAMIE in terms of competitive performance and time efficiency for sequence labeling with AL.We publicly release our code (
Teaching Interactively to Learn Emotions in Natural Language

Motivated by prior literature, we provide a proof of concept simulation study for an understudied interactive machine learning method, machine teaching (MT), for the text-based emotion prediction task.We compare this method experimentally against a more well-studied technique, active learning (AL).Results show the strengths of both approaches over more resource-intensive offline supervised learning.Additionally, applying AL and MT to fine-tune a pre-trained model offers further efficiency gain.We end by recommending research directions which aim to empower users in the learning process.
Mining Clues from Incomplete Utterance: A Query-enhanced Network for Incomplete Utterance Rewriting

Incomplete utterance rewriting has recently raised wide attention.However, previous works do not consider the semantic structural information between incomplete utterance and rewritten utterance or model the semantic structure implicitly and insufficiently.To address this problem, we propose a QUEry-Enhanced Network (QUEEN).Firstly, our proposed query template explicitly brings guided semantic structural knowledge between the incomplete utterance and the rewritten utterance making model perceive where to refer back to or recover omitted tokens.Then, we adopt a fast and effective edit operation scoring network to model the relation between two tokens.Benefiting from extra information and the well-designed network, QUEEN achieves state-of-the-art performance on several public datasets.
Uncertainty and Inclusivity in Gender Bias Annotation: An Annotation Taxonomy and Annotated Datasets of British English Text

Mitigating harms from gender biased language in Natural Language Processing (NLP) systems remains a challenge, and the situated nature of language means bias is inescapable in NLP data.Though efforts to mitigate gender bias in NLP are numerous, they often vaguely define gender and bias, only consider two genders, and do not incorporate uncertainty into models.To address these limitations, in this paper we present a taxonomy of gender biased language and apply it to create annotated datasets.We created the taxonomy and annotated data with the aim of making gender bias in language transparent.If biases are communicated clearly, varieties of biased language can be better identified and measured.Our taxonomy contains eleven types of gender biases inclusive of people whose gender expressions do not fit into the binary conceptions of woman and man, and whose gender differs from that they were assigned at birth, while also allowing annotators to document unknown gender information.The taxonomy and annotated data will, in future work, underpin analysis and more equitable language model development.
Controllable Sentence Simplification via Operation Classification

Different types of transformations have been used to model sentence simplification ranging from mainly local operations such as phrasal or lexical rewriting, deletion and re-ordering to the more global affecting the whole input sentence such as sentence rephrasing, copying and splitting.In this paper, we propose a novel approach to sentence simplification which encompasses four global operations: whether to rephrase or copy and whether to split based on syntactic or discourse structure.We create a novel dataset that can be used to train highly accurate classification systems for these four operations.We propose a controllable-simplification model that tailors simplifications to these operations and show that it outperforms both end-to-end, noncontrollable approaches and previous controllable approaches.
Crake: Causal-Enhanced Table-Filler for Question Answering over Large Scale Knowledge Base

Semantic parsing solves knowledge base (KB) question answering (KBQA) by composing a KB query, which generally involves node extraction (NE) and graph composition (GC) to detect and connect related nodes in a query.Despite the strong causal effects between NE and GC, previous works fail to directly model such causalities in their pipeline, hindering the learning of subtask correlations.Also, the sequencegeneration process for GC in previous works induces ambiguity and exposure bias, which further harms accuracy.In this work, we formalize semantic parsing into two stages.In the first stage (graph structure generation), we propose a causal-enhanced table-filler to overcome the issues in sequence-modelling and to learn the internal causalities.In the second stage (relation extraction), an efficient beamsearch algorithm is presented to scale complex queries on large-scale KBs.Experiments on LC-QuAD 1.0 indicate that our method surpasses previous state-of-the-arts by a large margin (17%) while remaining time and space efficiency.The code and models are available at https://github.com/AOZMH/Crake.
Small-Footprint Open-Vocabulary Keyword Spotting with Quantized LSTM Networks

We explore a keyword-based spoken language understanding system, in which the intent of the user can directly be derived from the detection of a sequence of keywords in the query. In this paper, we focus on an open-vocabulary keyword spotting method, allowing the user to define their own keywords without having to retrain the whole model. We describe the different design choices leading to a fast and small-footprint system, able to run on tiny devices, for any arbitrary set of user-defined keywords, without training data specific to those keywords. The model, based on a quantized long short-term memory (LSTM) neural network, trained with connectionist temporal classification (CTC), weighs less than 500KB. Our approach takes advantage of some properties of the predictions of CTC-trained networks to calibrate the confidence scores and implement a fast detection algorithm. The proposed system outperforms a standard keyword-filler model approach.
Problems with Shapley-value-based explanations as feature importance measures

Game-theoretic formulations of feature importance have become popular as a way to "explain" machine learning models. These methods define a cooperative game between the features of a model and distribute influence among these input elements using some form of the game's unique Shapley values. Justification for these methods rests on two pillars: their desirable mathematical properties, and their applicability to specific motivations for explanations. We show that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity, such as the need for causal reasoning. We also draw on additional literature to argue that Shapley values do not provide explanations which suit human-centric goals of explainability.
Unsupervised Question Decomposition for Question Answering

We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering. Since labeling questions with decompositions is cumbersome, we take an unsupervised approach to produce sub-questions, also enabling us to leverage millions of questions from the internet. Specifically, we propose an algorithm for One-to-N Unsupervised Sequence transduction (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We answer sub-questions with an off-the-shelf QA model and give the resulting answers to a recomposition model that combines them into a final answer. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets. ONUS automatically learns to decompose different kinds of questions, while matching the utility of supervised and heuristic decomposition methods for QA and exceeding those methods in fluency. Qualitatively, we find that using sub-questions is promising for shedding light on why a QA system makes a prediction.
Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing

Serverless computing is an excellent fit for big data processing because it can scale quickly and cheaply to thousands of parallel functions. Existing serverless platforms isolate functions in ephemeral, stateless containers, preventing them from directly sharing memory. This forces users to duplicate and serialise data repeatedly, adding unnecessary performance and resource costs. We believe that a new lightweight isolation approach is needed, which supports sharing memory directly between functions and reduces resource overheads. We introduce Faaslets, a new isolation abstraction for high-performance serverless computing. Faaslets isolate the memory of executed functions using software-fault isolation (SFI), as provided by WebAssembly, while allowing memory regions to be shared between functions in the same address space. Faaslets can thus avoid expensive data movement when functions are co-located on the same machine. Our runtime for Faaslets, Faasm, isolates other resources, e.g. CPU and network, using standard Linux cgroups, and provides a low-level POSIX host interface for networking, file system access and dynamic loading. To reduce initialisation times, Faasm restores Faaslets from already-initialised snapshots. We compare Faasm to a standard container-based platform and show that, when training a machine learning model, it achieves a 2x speed-up with 10x less memory; for serving machine learning inference, Faasm doubles the throughput and reduces tail latency by 90%.
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models

In this work, we propose a new family of generative flows on an augmented data space, with an aim to improve expressivity without drastically increasing the computational cost of sampling and evaluation of a lower bound on the likelihood. Theoretically, we prove the proposed flow can approximate a Hamiltonian ODE as a universal transport map. Empirically, we demonstrate state-of-the-art performance on standard benchmarks of flow-based generative modeling.