Reviewerly/exHarmony huggingface dataset:
Systematicity in GPT-3’s Interpretation of Novel English Noun Compounds

Levin et al. (2019) show experimentally that the interpretations of novel English noun compounds (e.g., stew skillet), while not fully compositional, are highly predictable based on whether the modifier and head refer to artifacts or natural kinds.Is the large language model GPT-3 governed by the same interpretive principles?To address this question, we first compare Levin et al.'s experimental data with GPT-3 generations, finding a high degree of similarity.However, this evidence is consistent with GPT-3 reasoning only about specific lexical items rather than the more abstract conceptual categories of Levin et al.'s theory.To probe more deeply, we construct prompts that require the relevant kind of conceptual reasoning.Here, we fail to find convincing evidence that GPT-3 is reasoning about more than just individual lexical items.These results highlight the importance of controlling for low-level distributional regularities when assessing whether a large language model latently encodes a deeper theory.
Probing for Incremental Parse States in Autoregressive Language Models

Next-word predictions from autoregressive neural language models show remarkable sensitivity to syntax.This work evaluates the extent to which this behavior arises as a result of a learned ability to maintain implicit representations of incremental syntactic structures.We extend work in syntactic probing to the incremental setting and present several probes for extracting incomplete syntactic structure (operationalized through parse states from a stack-based parser) from autoregressive language models.We find that our probes can be used to predict model preferences on ambiguous sentence prefixes and causally intervene on model representations and steer model behavior.This suggests implicit incremental syntactic inferences underlie next-word predictions in autoregressive neural language models.
Towards Inter-character Relationship-driven Story Generation

In this paper, we introduce the task of modeling interpersonal relationships for story generation.For addressing this task, we propose Relationships as Latent Variables for Story Generation, (RELIST).RELIST generates stories sentence by sentence and has two major components -a relationship selector and a story continuer.The relationship selector specifies a latent variable to pick the relationship to exhibit in the next sentence and the story continuer generates the next sentence while expressing the selected relationship in a coherent way.Our automatic and human evaluations demonstrate that RELIST is able to generate stories with relationships that are more faithful to desired relationships while maintaining the content quality.The relationship assignments to sentences during inference brings interpretability to RELIST.
20Q: Overlap-Free World Knowledge Benchmark for Language Models

What do language models know about our world?This question is hard to answer but important to get right.To this end, we introduce 20Q, a novel benchmark using the Twenty Questions game to evaluate world knowledge and common sense of language models.Thanks to our overlap-free benchmark, language models learn the game of Twenty Questions without learning relevant knowledge for the test set.We uncover two intuitive factors influencing the world knowledge of language models: the size of the model and the topic frequency in the pre-training data.Moreover, we show that in-context learning is inefficient for evaluating language models' world knowledge -fine-tuning is necessary to show their true capabilities.Lastly, our results show room for improvement to enhance the world knowledge and common sense of large language models.A potential solution would be to up-sample unfrequent topics in the pre-training of language models.
What’s in a (dataset’s) name? The case of BigPatent

Sharing datasets and benchmarks has been crucial for rapidly improving Natural Language Processing models and systems.Documenting datasets' characteristics (and any modification introduced over time) is equally important to avoid confusion and make comparisons reliable.Here, we describe the case of BigPatent, a dataset for patent summarization that exists in at least two rather different versions under the same name.While previous literature has not clearly distinguished among versions, their differences not only lay on a surface level but also modify the dataset's core nature and, thus, the complexity of the summarization task.While this paper describes a specific case, we aim to shed light on new challenges that might emerge in resource sharing and advocate for comprehensive documentation of datasets and models.
What Was Your Name Again? Interrogating Generative Conversational Models For Factual Consistency Evaluation

Generative conversational agents are known to suffer from problems like inconsistency and hallucination, and a big challenge in studying these issues remains evaluation: they are not properly reflected in common text generation metrics like perplexity or BLEU, and alternative implicit methods like semantic similarity or NLI labels can be misguided when few specific tokens are decisive.In this work we propose ConsisTest; a factual consistency benchmark including both WH and Y/N questions based on PersonaChat, along with a hybrid evaluation pipeline which aims to get the best of symbolic and sub-symbolic methods.Using these and focusing on pretrained generative models like BART, we provide detailed analysis on how the model's factual consistency is affected by variations in question and context.
Summarizing Procedural Text: Data and Approach

Procedural text is a widely used genre that contains many steps of instructions of how to cook a dish or how to conduct a chemical experiment and analyzing the procedural text has become a popular task in the NLP field.Since the procedural text can be very long and contains many details, summarizing the whole procedural text or giving an overview for each complicated procedure step can save time for readers and help the reader to capture the core action in the procedure.In this paper, we propose the procedural text summarization task with two summarization granularity: step-view and globalview, which summarizes each step in procedural text separately or gives an overall summary for all steps respectively.To tackle this task, we propose an Entity-State Graph-based Summarizer (ESGS) which is based on stateof-the-art entity state tracking methods and constructs a heterogeneous graph to aggregate contextual information for each procedure.In order to help the summarization model focus on the salient entity, we propose to use the contextualized procedure graph representation to predict the salient entity.Experiments conducted on two datasets verify the effectiveness of our proposed model, and the code and datasets will be released on https://github.com/gsh199449/procedural-summ.
On the Compositional Generalization Gap of In-Context Learning

Pretrained large generative language models have shown great performance on many tasks, but exhibit low compositional generalization abilities.Scaling such models has been shown to improve their performance on various NLP tasks even just by conditioning them on a few examples to solve the task without any finetuning (also known as in-context learning).In this work, we look at the gap between the indistribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning.In the ID settings, the demonstrations are from the same split (test or train) that the model is being evaluated on, and in the OOD settings, they are from the other split.We look at how the relative generalization gap of in-context learning evolves as models are scaled up.We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery with different number of exemplars, and observe a trend of decreasing relative generalization gap as models are scaled up.
Using NLP to Support English Teaching in Rural Schools

We present a web application for creating games and exercises for teaching English as a foreign language with the help of NLP tools.The application contains different kinds of games such as crosswords, word searches, a memory game, and a multiplayer game based on the classic battleship pen and paper game.This application was built with the aim of supporting teachers in rural schools that are teaching English lessons, so they can easily create interactive and engaging activities for their students.We present the context and history of the project, the current state of the web application, and some ideas on how we will expand it in the future.
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation

Large pretrained language models can easily produce toxic or biased content, which is prohibitive for practical use.In order to detect such toxic generations, existing methods rely on templates, real-world data extraction, crowdsourcing workers, or automatic generation to construct adversarial contexts that are likely to induce toxic generations.However, what type of context is more likely to induce unsafe responses is still under-explored.In this paper, we identify that context toxicity and context category (e.g., profanity, insult, drugs, etc.) are two important factors to cause safety issues in response generation.Hence, we propose a method called reverse generation to construct adversarial contexts conditioned on a given response, with the flexibility to control category, toxicity level, and inductivity of the generated contexts.Via reverse generation, we augment the existing BAD dataset and construct a new dataset BAD+ which contains more than 120K diverse and highly inductive contexts in 12 categories.We test three popular pretrained dialogue models (Blender, Di-aloGPT, and Plato2) and find that BAD+ can largely expose their safety problems.Furthermore, we show that BAD+ can greatly enhance the safety of generation and reveal the key factors of safety improvement.Our code and dataset is available at https://github.
MLLabs-LIG at TempoWiC 2022: A Generative Approach for Examining Temporal Meaning Shift

In this paper, we present our system for the EvoNLP 2022 shared task Temporal Meaning Shift (TempoWiC).Different from the typically used discriminative model, we propose a generative approach based on pre-trained generation models.The basic architecture of our system is a seq2seq model where the input sequence consists of two documents followed by a question asking whether the meaning of target word changed or not, the target output sequence is a declarative sentence describing the meaning of target word changed or not.The experimental results on TempoWiC test set show that our best system (with time information) obtained an accuracy and Macro-F1 score of 68.09% and 62.59% respectively, which ranked 12th among all submitted systems.The results have shown the plausibility of using generation model for WiC tasks, meanwhile also indicate there's still room for further improvement.
Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations.In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages.We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting.Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages.Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines.Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
Securely Capturing People’s Interactions with Voice Assistants at Home: A Bespoke Tool for Ethical Data Collection

Speech production is nuanced and unique to every individual, but today's Spoken Dialogue Systems (SDSs) are trained to use general speech patterns to successfully improve performance on various evaluation metrics.However, these patterns do not apply to certain user groups -often the very people that can benefit the most from SDSs.For example, people with dementia produce more disfluent speech than the general population.In order to evaluate systems with specific user groups in mind, and to guide the design of such systems to deliver maximum benefit to these users, data must be collected securely.In this short paper we present CVR-SI, a bespoke tool for ethical data collection.Designed for the healthcare domain, we argue that it should also be used in more general settings.We detail how off-the-shelf solutions fail to ensure that sensitive data remains secure and private.We then describe the ethical design and security features of our device, with a full guide on how to build both the hardware and software components of CVR-SI.Our design ensures inclusivity to all researchers in this field, particularly those who are not hardware experts.This guarantees everyone can collect appropriate data for human evaluation ethically, securely, and in a timely manner.
Generating Textual Adversaries with Minimal Perturbation

Many word-level adversarial attack approaches for textual data have been proposed in recent studies.However, due to the massive search space consisting of combinations of candidate words, the existing approaches face the problem of preserving the semantics of texts when crafting adversarial counterparts.In this paper, we develop a novel attack strategy to find adversarial texts with high similarity to the original texts while introducing minimal perturbation.The rationale is that we expect the adversarial texts with small perturbation can better preserve the semantic meaning of original texts.Experiments show that, compared with state-of-the-art attack approaches, our approach achieves higher success rates and lower perturbation rates in four benchmark datasets 1 .
MiST: a Large-Scale Annotated Resource and Neural Models for Functions of Modal Verbs in English Scientific Text

Modal verbs (e.g., can, should or must) occur highly frequently in scientific articles.Decoding their function is not straightforward: they are often used for hedging, but they may also denote abilities and restrictions.Understanding their meaning is important for various NLP tasks such as writing assistance or accurate information extraction from scientific text.To foster research on the usage of modals in this genre, we introduce the MIST (Modals In Scientific Text) dataset, which contains 3737 modal instances in five scientific domains annotated for their semantic, pragmatic, or rhetorical function.We systematically evaluate a set of competitive neural architectures on MIST.Transfer experiments reveal that leveraging non-scientific data is of limited benefit for modeling the distinctions in MIST.Our corpus analysis provides evidence that scientific communities differ in their usage of modal verbs, yet, classifiers trained on scientific data generalize to some extent to unseen scientific domains.
Textual Enhanced Contrastive Learning for Solving Math Word Problems

Solving math word problems is the task that analyses the relation of quantities and requires an accurate understanding of contextual natural language information.Recent studies show that current models rely on shallow heuristics to predict solutions and could be easily misled by small textual perturbations.To address this problem, we propose a Textual Enhanced Contrastive Learning framework, which enforces the models to distinguish semantically similar examples while holding different mathematical logic.We adopt a self-supervised manner strategy to enrich examples with subtle textual variance by textual reordering or problem re-construction.We then retrieve the hardest to differentiate samples from both equation and textual perspectives and guide the model to learn their representations.Experimental results show that our method achieves state-of-the-art on both widely used benchmark datasets and also exquisitely designed challenge datasets in English and Chinese.
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling.However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings.In this paper, we propose an alternative mining-based approach for zero-shot learning.Instead of prompting language models, we use regular expressions to mine labeled examples 1 from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model.Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates.Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.
Structural Constraints and Natural Language Inference for End-to-End Flowchart Grounded Dialog Response Generation

Flowchart grounded dialog systems converse with users by following a given flowchart and a corpus of FAQs.The existing state-of-the-art approach (Raghu et al., 2021) for learning such a dialog system, named FLONET, has two main limitations.(1) It uses a Retrieval Augmented Generation (RAG) framework which represents a flowchart as a bag of nodes.By doing so, it loses the connectivity structure between nodes which can aid in better response generation.(2) Typically dialogs progress with the agent asking polar (Y/N) questions, but users often respond indirectly without the explicit use of polar words.In such cases, it fails to understand the correct polarity of the answer.To overcome these issues, we propose Structure-Aware FLONET (SA-FLONET) which infuses structural constraints derived from the connectivity structure of flowcharts into the RAG framework.It uses natural language inference to better predict the polarity of indirect Y/N answers.We find that SA-FLONET outperforms FLONET, with a success rate improvement of 68% and 123% in flowchart grounded response generation and zero-shot flowchart grounded response generation tasks respectively.
Using Two Losses and Two Datasets Simultaneously to Improve TempoWiC Accuracy

WSD (Word Sense Disambiguation) is the task of identifying which sense of a word is meant in a sentence or other segment of text.Researchers have worked on this task (e.g.Pustejovsky, 2002) for years but it's still a challenging one even for SOTA (state-of-the-art) LMs (language models).The new dataset, Tem-poWiC introduced by Loureiro et al. (2022b) focuses on the fact that words change over time.Their best baseline achieves 70.33% macro-F1.In this work, we use two different losses simultaneously to train RoBERTa-based classification models.We also improve our model by using another similar dataset to generalize better.Our best configuration beats their best baseline by 4.23% and reaches 74.56% macro-F1.
Algorithmic Diversity and Tiny Models: Comparing Binary Networks and the Fruit Fly Algorithm on Document Representation Tasks

Neural language models have seen a dramatic increase in size in the last years.While many still advocate that 'bigger is better', work in model distillation has shown that the number of parameters used by very large networks is actually more than what is required for state-of-theart performance.This prompts an obvious question: can we build smaller models from scratch, rather than going through the inefficient process of training at scale and subsequently reducing model size?In this paper, we investigate the behaviour of a biologically inspired algorithm, based on the fruit fly's olfactory system.This algorithm has shown good performance in the past on the task of learning word embeddings.We now put it to the test on the task of semantic hashing.Specifically, we compare the fruit fly to a standard binary network on the task of generating locality-sensitive hashes for text documents, measuring both task performance and energy consumption.Our results indicate that the two algorithms have complementary strengths while showing similar electricity usage.
HashFormers: Towards Vocabulary-independent Pre-trained Transformers

Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding.This one-to-one mapping results into embedding matrices that occupy a lot of memory (i.e.millions of parameters) and grow linearly with the size of the vocabulary.Previous work on on-device transformers dynamically generate token embeddings on-the-fly without embedding matrices using locality-sensitive hashing over morphological information.These embeddings are subsequently fed into transformer layers for text classification.However, these methods are not pre-trained.Inspired by this line of work, we propose HASHFORMERS, a new family of vocabulary-independent pretrained transformers that support an unlimited vocabulary (i.e.all possible tokens in a corpus) given a substantially smaller fixed-sized embedding matrix.We achieve this by first introducing computationally cheap hashing functions that bucket together individual tokens to embeddings.We also propose three variants that do not require an embedding matrix at all, further reducing the memory requirements.We empirically demonstrate that HASHFORM-ERS are more memory efficient compared to standard pre-trained transformers while achieving comparable predictive performance when fine-tuned on multiple text classification tasks.For example, our most efficient HASHFORMER variant has a negligible performance degradation (0.4% on GLUE) using only 99.1K parameters for representing the embeddings compared to 12.3-38M parameters of state-of-the-art models. 1
Adaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Prediction

Modern Review Helpfulness Prediction systems are dependent upon multiple modalities, typically texts and images.Unfortunately, those contemporary approaches pay scarce attention to polish representations of cross-modal relations and tend to suffer from inferior optimization.This might cause harm to model's predictions in numerous cases.To overcome the aforementioned issues, we propose Multimodal Contrastive Learning for Multimodal Review Helpfulness Prediction (MRHP) problem, concentrating on mutual information between input modalities to explicitly elaborate cross-modal relations.In addition, we introduce Adaptive Weighting scheme for our contrastive learning approach in order to increase flexibility in optimization.Lastly, we propose Multimodal Interaction module to address the unalignment nature of multimodal data, thereby assisting the model in producing more reasonable multimodal representations.Experimental results show that our method outperforms prior baselines and achieves state-of-the-art results on two publicly available benchmark datasets for MRHP problem.