Reviewerly/exHarmony huggingface dataset:
No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media

News articles both shape and reflect public opinion across the political spectrum.Analyzing them for social bias can thus provide valuable insights, such as prevailing stereotypes in society and the media, which are often adopted by NLP models trained on respective data.Recent work has relied on word embedding bias measures, such as WEAT.However, several representation issues of embeddings can harm the measures' accuracy, including low-resource settings and token frequency differences.In this work, we study what kind of embedding algorithm serves best to accurately measure types of social bias known to exist in US online news articles.To cover the whole spectrum of political bias in the US, we collect 500k articles and review psychology literature with respect to expected social bias.We then quantify social bias using WEAT along with embedding algorithms that account for the aforementioned issues.We compare how models trained with the algorithms on news articles represent the expected social bias.Our results suggest that the standard way to quantify bias does not align well with knowledge from psychology.While the proposed algorithms reduce the gap, they still do not fully match the literature.
Syntactically Rich Discriminative Training: An Effective Method for Open Information Extraction

Open information extraction (OIE) is the task of extracting facts "(Subject, Relation, Object)" from natural language text.We propose several new methods for training neural OIE models in this paper.First, we propose a novel method for computing syntactically rich text embeddings using the structure of dependency trees.Second, we propose a new discriminative training approach to OIE in which tokens in the generated fact are classified as "real" or "fake", i.e., those tokens that are in both the generated and gold tuples, and those that are only in the generated tuple but not in the gold tuple.We also address the issue of repetitive tokens in generated facts and improve the models' ability to generate implicit facts.Our approach reduces repetitive tokens by a factor of 23%.Finally, we present paraphrased versions of the CaRB, OIE2016, and LSOIE datasets, and show that the models' performance substantially improves when trained on datasets augmented by such data.Our best model beats the SOTA of IMoJIE on the recent CaRB dataset, with an improvement of 39.63% in F 1 score.
Towards More Natural Artificial Languages

A number of papers have recently argued in favor of using artificially generated languages to investigate the inductive biases of linguistic models, or to develop models for low-resource languages with underrepresented typologies.But the promise of artificial languages comes with a caveat: if these artificial languages are not sufficiently reflective of natural language, then using them as a proxy may lead to inaccurate conclusions.In this paper, we take a step towards increasing the realism of artificial language by introducing a variant of indexed grammars that draw their weights from hierarchical Pitman-Yor processes.We show that this framework generates languages that emulate the statistics of natural language corpora better than the current approach of directly formulating weighted context-free grammars.
Can Language Models Serve as Temporal Knowledge Bases?

Recent progress regarding the use of language models (LMs) as knowledge bases (KBs) has shown that language models can act as structured knowledge bases for storing relational facts.However, most existing works only considered the LM-as-KB paradigm in a static setting, which ignores the analysis of temporal dynamics of world knowledge.Furthermore, a basic function of KBs, i.e., the ability to store conflicting information (i.e., 1-N, N-1 and N-M relations), is underexplored.In this paper, we formulate two practical requirements for treating LMs as temporal KBs: (i) the capacity to store temporally-scoped knowledge that contains conflicting information and (ii) the ability to use stored knowledge for temporally-scoped knowledge queries.We introduce a new dataset called LAMA-TK which is aimed at probing temporally-scoped knowledge, and investigate the two above requirements to explore the LM-as-KB paradigm in the temporal domain.On the one hand, experiments show that LMs can memorize millions of temporally-scoped facts with relatively high accuracy and transfer stored knowledge to temporal knowledge queries, thereby expanding the LM-as-KB paradigm to the temporal domain.On the other hand, we show that memorizing conflicting information, which has been neglected by previous works, is still challenging for LMs and hinders the memorization of other unrelated one-to-one relationships.
JamPatoisNLI: A Jamaican Patois Natural Language Inference Dataset

JamPatoisNLI provides the first dataset for natural language inference in a creole language, Jamaican Patois.Many of the most-spoken low-resource languages are creoles.These languages commonly have a lexicon derived from a major world language and a distinctive grammar reflecting the languages of the original speakers and the process of language birth by creolization.This gives them a distinctive place in exploring the effectiveness of transfer from large monolingual or multilingual pretrained models.While our work, along with previous work, shows that transfer from these models to low-resource languages that are unrelated to languages in their training set is not very effective, we would expect stronger results from transfer to creoles.Indeed, our experiments show considerably better results from few-shot learning of JamPatoisNLI than for such unrelated languages, and help us begin to understand how the unique relationship between creoles and their high-resource base languages affect cross-lingual transfer.JamPat-oisNLI, which consists of naturally-occurring premises and expert-written hypotheses, is a step towards steering research into a traditionally underserved language and a useful benchmark for understanding cross-lingual NLP.
CheckHARD: Checking Hard Labels for Adversarial Text Detection, Prediction Correction, and Perturbed Word Suggestion

An adversarial attack generates harmful text that fools a target model.More dangerously, this text is unrecognizable by humans.Existing work detects adversarial text and corrects a target's prediction by identifying perturbed words and changing them into their synonyms, but many benign words are also changed.In this paper, we directly detect adversarial text, correct the prediction, and suggest perturbed words by checking the change in the hard labels from the target's predictions after replacing a word with its transformation using a model that we call CHECKHARD.The experiments demonstrate that CHECKHARD outperforms existing work on various attacks, models, and datasets.
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences

Natural language inference (NLI) is a task to infer the relationship between a premise and a hypothesis (e.g., entailment, neutral, or contradiction), and transformer-based models perform well on current NLI datasets such as MNLI and SNLI.Nevertheless, given the linguistic complexity of the large-scale datasets, it remains controversial whether these models can truly infer the relationship between sentences or they simply guess the answer via shallow heuristics.Here, we introduce a controlled evaluation set called Simple Pair to test the basic sentence inference ability of NLI models using sentences with syntactically simple structures.Three popular transformer-based models, i.e., BERT, RoBERTa, and DeBERTa, are employed.We find that these models finetuned on MNLI or SNLI perform very poorly on Simple Pair (< 35.4% accuracy).Further analyses reveal event coreference and compositional binding problems in these models.To improve the model performance, we augment the training set, i.e., MNLI or SNLI, with a few examples constructed based on Simple Pair (∼ 1% of the size of the original SNLI/MNLI training sets).Models fine-tuned on the augmented training set maintain high performance on MNLI/SNLI and perform very well on Simple Pair (∼100% accuracy).Furthermore, the positive performance of the augmented training models can transfer to more complex examples constructed based on sentences from MNLI and SNLI.Taken together, the current work shows that (1) models achieving high accuracy on mainstream large-scale datasets still lack the capacity to draw accurate inferences on simple sentences, and (2) augmenting mainstream datasets with a small number of target simple sentences can effectively improve model performance.
Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval

Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query.For narrative videos, e.g., dramas or movies, the holistic understanding of temporal dynamics and multimodal reasoning is crucial.Previous works have shown promising results; however, they relied on the expensive query annotations for VCMR, i.e., the corresponding moment intervals.To overcome this problem, we propose a self-supervised learning framework: Modal-specific Pseudo Query Generation Network (MPGN).First, MPGN selects candidate temporal moments via subtitle-based moment sampling.Then, it generates pseudo queries exploiting both visual and textual information from the selected temporal moments.Through the multimodal information in the pseudo queries, we show that MPGN successfully learns to localize the video corpus moment without any explicit annotation.We validate the effectiveness of MPGN on the TVR dataset, showing competitive results compared with both supervised models and unsupervised setting models.
Benchmarking Language Models for Code Syntax Understanding

Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding, which represent the input as a token sequence without explicitly modeling its structure.Some prior works show that pretrained language models can capture the syntactic rules of natural languages without finetuning on syntax understanding tasks.However, there is limited understanding of how well pre-trained models understand the code structure so far.In this work, we perform the first thorough benchmarking of the stateof-the-art pre-trained models for identifying the syntactic structures of programs.Specifically, we introduce CodeSyntax, a large-scale dataset of programs annotated with the syntactic relationships in their corresponding abstract syntax trees.Our key observation is that existing language models pretrained on code still lack the understanding of code syntax.In fact, these pre-trained programming language models fail to match the performance of simple baselines based on positional offsets and keywords.We also present a natural language benchmark to highlight the differences between natural languages and programming languages in terms of syntactic structure understanding.Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures. 1
Enhancing and Evaluating the Grammatical Framework Approach to Logic-to-Text Generation

Logic-to-text generation is an important yet underrepresented area of natural language generation (NLG).In particular, most previous works on this topic lack sound evaluation.We address this limitation by building and evaluating a system that generates high-quality English text given a first-order logic (FOL) formula as input.We start by analyzing the performance of Ranta (2011)'s system.Based on this analysis, we develop an extended version of the system, which we name LOLA, that performs formula simplification based on logical equivalences and syntactic transformations.We carry out an extensive evaluation of LOLA using standard automatic metrics and human evaluation.We compare the results against a baseline and Ranta (2011)'s system.The results show that LOLA outperforms the other two systems in most aspects.
Composition, Attention, or Both?

In this paper, we propose a novel architecture called Composition Attention Grammars (CAGs) that recursively compose subtrees into a single vector representation with a composition function, and selectively attend to previous structural information with a self-attention mechanism.We investigate whether these components-the composition function and the self-attention mechanism-can both induce human-like syntactic generalization.Specifically, we train language models (LMs) with and without these two components with the model sizes carefully controlled, and evaluate their syntactic generalization performance against six test circuits on the SyntaxGym benchmark.The results demonstrated that the composition function and the self-attention mechanism both play an important role to make LMs more human-like, and closer inspection of grammatical phenomena implied that the composition function allowed syntactic features, but not semantic features, to percolate into subtree representations.
A POMDP Dialogue Policy with 3-way Grounding and Adaptive Sensing for Learning through Communication

Agents to assist with rescue, surgery, and similar activities could collaborate better with humans if they could learn new strategic behaviors through communication.We introduce a novel POMDP dialogue policy for learning from people.The policy has 3-way grounding of language in the shared physical context, the dialogue context, and persistent knowledge.It can learn distinct but related games, and can continue learning across dialogues for complex games.A novel sensing component supports adaptation to information-sharing differences across people.The single policy performs better than oracle policies customized to specific games and information behavior.
Wish I Can Feel What You Feel: A Neural Approach for Empathetic Response Generation

Expressing empathy is important in everyday conversations, and exploring how empathy arises is crucial in automatic response generation.Most previous approaches consider only a single factor that affects empathy.However, in practice, empathy generation and expression is a very complex and dynamic psychological process.A listener needs to find out events which cause a speaker's emotions (emotion cause extraction), project the events into some experience (knowledge extension), and express empathy in the most appropriate way (communication mechanism).To this end, we propose a novel approach, which integrates the three components -emotion cause, knowledge graph, and communication mechanism for empathetic response generation.Experimental results on the benchmark dataset demonstrate the effectiveness of our method and show that incorporating the key components generates more informative and empathetic responses.
Explicit Role Interaction Network for Event Argument Extraction

Event argument extraction is a challenging subtask of event extraction, aiming to identify and assign roles to arguments under a certain event.Existing methods extract arguments of each role independently, ignoring the relationship between different roles.Such an approach hinders the model from learning explicit interactions between different roles to improve the performance of individual argument extraction.As a solution, we design a neural model that we refer to as the Explicit Role Interaction Network (ERIN) which allows for dynamically capturing the correlations between different argument roles within an event.Extensive experiments on the benchmark dataset ACE2005 demonstrate the superiority of our proposed model to existing approaches.
Multilingual Social Media Text Generation and Evaluation with Few-Shot Prompting

This work adapts large language models to generate multilingual social media text that meets several objectives simultaneously: topic relevance, author style consistency, and reply validity. Leveraging existing online information behavior simulators, which currently only forecast activities but not content, our approach comprised of generalizable prompt formation and efficient evaluation to produce a believable, personalized, and responsive synthetic social network. According to some preliminary experiments, our multi-objective prompt formation and automatic evaluation/selection methods are able to yield a significant number of high-quality synthetic texts according to both standardized and trained metrics.
monoQA: Multi-Task Learning of Reranking and Answer Extraction for Open-Retrieval Conversational Question Answering

To address the Conversational Question Answering (ORConvQA) task, previous work has considered an effective three-stage architecture, consisting of a retriever, a reranker, and a reader to extract the answers.In order to effectively answer the users' questions, a number of existing approaches have applied multi-task learning, such that the same model is shared between the reranker and the reader.Such approaches also typically tackle reranking and reading as classification tasks.On the other hand, recent text generation models, such as monoT5 and UnifiedQA, have been shown to respectively yield impressive performances in passage reranking and reading.However, no prior work has combined monoT5 and Uni-fiedQA to share a single text generation model that directly extracts the answers for the users instead of predicting the start/end positions in a retrieved passage.In this paper, we investigate the use of Multi-Task Learning (MTL) to improve performance on the ORConvQA task by sharing the reranker and reader's learned structure in a generative model.In particular, we propose monoQA, which uses a text generation model with multi-task learning for both the reranker and reader.Our model, which is based on the T5 text generation model, is fine-tuned simultaneously for both reranking (in order to improve the precision of the top retrieved passages) and extracting the answer.Our results on the OR-QuAC and OR-CoQA datasets demonstrate the effectiveness of our proposed model, which significantly outperforms existing strong baselines with improvements ranging from +12.31% to +19.51% in MAP and from +5.70% to +23.34% in F1 on all used test sets.
Probing the representations of named entities in Transformer-based Language Models

In this work we analyze the named entity representations learned by Transformer-based language models.We investigate the role entities play in two tasks: a language modeling task, and a sequence classification task.For this purpose we collect a novel news topic classification dataset with 12 topics called RefNews-12.We perform two complementary methods of analysis.First, we use diagnostic models allowing us to quantify to what degree entity information is present in the hidden representations.Second, we perform entity mention substitution to measure how substitute-entities with different properties impact model performance.By controlling for model uncertainty we are able to show that entities are identified, and depending on the task, play a measurable role in the model's predictions.Additionally, we show that the entities' types alone are not enough to account for this.Finally, we find that the the frequency with which entities occur are important for the masked language modeling task, and that the entities' distributions over topics are important for topic classification.
On Utilizing Constituent Language Resources to Improve Downstream Tasks in Hinglish

Performance of downstream NLP tasks on code-switched Hindi-English (aka Hinglish ) continues to remain a significant challenge.Intuitively, Hindi and English corpora should aid improve task performance on Hinglish.We show that meta-learning framework can effectively utilize the the labelled resources of the downstream tasks in the constituent 1 languages.The proposed approach improves the performance on downstream tasks on codeswitched language.We experiment with Hinglish code-switching benchmark GLUE-CoS and report significant improvements.
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token

The pre-training of masked language models (MLMs) consumes massive computation to achieve good results on downstream NLP tasks, resulting in a large carbon footprint.In the vanilla MLM, the virtual tokens, [MASK]s, act as placeholders and gather the contextualized information from unmasked tokens to restore the corrupted information.It raises the question of whether we can append [MASK]s at a later layer, to reduce the sequence length for earlier layers and make the pre-training more efficient.We show: (1) [MASK]s can indeed be appended at a later layer, being disentangled from the word embedding; (2) The gathering of contextualized information from unmasked tokens can be conducted with a few layers.By further increasing the masking rate from 15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch with only 78% and 68% of the original computational budget without any degradation on the GLUE benchmark.When pre-training with the original budget, our method outperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%. 1
WordTies: Measuring Word Associations in Language Models via Constrained Sampling

Word associations are widely used in psychology to provide insights on how humans perceive and understand concepts.Comparing word associations in language models (LMs) to those generated by human subjects can serve as a proxy to uncover embedded lexical and commonsense knowledge in language models.While much helpful work has been done applying direct metrics, such as cosine similarity, to help understand latent spaces, these metrics are symmetric, while human word associativity is asymmetric.We propose WordTies, an algorithm based on constrained sampling from LMs, which allows an asymmetric measurement of associated words, given a cue word as the input.Comparing to existing methods, word associations found by this method share more overlap with associations provided by humans, and observe the asymmetric property of human associations.To examine possible reasons behind associations, we analyze the knowledge and reasoning behind the word pairings as they are linked to lexical and commonsense knowledge graphs.When the knowledge about the nature of the word pairings is combined with a probability that the LM has learned that information, we have a new way to examine what information is captured in LMs.
NLP@UIT at FigLang-EMNLP 2022: A Divide-and-Conquer System For Shared Task On Understanding Figurative Language

This paper describes our submissions to the EMNLP 2022 shared task on Understanding Figurative Language as part of the Figurative Language Workshop (FigLang 2022).Our systems based on pre-trained language model T5 are divide-and-conquer models which can address both two requirements of the task: 1) classification, and 2) generation.In this paper, we introduce different approaches in which each approach we employ a processing strategy on input model.We also emphasize the influence of the types of figurative language on our systems.