Reviewerly/exHarmony huggingface dataset:
Topical Segmentation of Spoken Narratives: A Test Case on Holocaust Survivor Testimonies

The task of topical segmentation is well studied, but previous work has mostly addressed it in the context of structured, well-defined segments, such as segmentation into paragraphs, chapters, or segmenting text that originated from multiple sources.We tackle the task of segmenting running (spoken) narratives, which poses hitherto unaddressed challenges.As a test case, we address Holocaust survivor testimonies, given in English.Other than the importance of studying these testimonies for Holocaust research, we argue that they provide an interesting test case for topical segmentation, due to their unstructured surface level, relative abundance (tens of thousands of such testimonies were collected), and the relatively confined domain that they cover.We hypothesize that boundary points between segments correspond to low mutual information between the sentences proceeding and following the boundary.Based on this hypothesis, we explore a range of algorithmic approaches to the task, building on previous work on segmentation that uses generative Bayesian modeling and state-of-the-art neural machinery.Compared to manually annotated references, we find that the developed approaches show considerable improvements over previous work. 1
Is It Smaller Than a Tennis Ball? Language Models Play the Game of Twenty Questions

Researchers often use games to analyze the abilities of Artificial Intelligence models.In this work, we use the game of Twenty Questions to study the world knowledge of language models.Despite its simplicity for humans, this game requires a broad knowledge of the world to answer yes/no questions.We evaluate several language models on this task and find that only the largest model has enough world knowledge to play it well, although it still has difficulties with the shape and size of objects.We also present a new method to improve the knowledge of smaller models by leveraging external information from the web.Finally, we release our dataset and Twentle, a website to interactively test the knowledge of language models by playing Twenty Questions.
Exploring Logographic Image for Chinese Aspect-based Sentiment Classification

In logographic languages like Chinese, word meanings are constructed using specific character formations, which can help to disambiguate word senses and are beneficial for sentiment analysis.However, such knowledge is rarely explored in previous sentiment analysis methods.In this paper, we focus on exploring logographic information for aspect-based sentiment classification in Chinese text.Specifically, we employ a logographic image to capture an internal morphological structure from the character sequence.In addition, we also employ the logographic image to learn the external relations among context and aspect words.Furthermore, we propose a multimodal language model to explicitly incorporate a logographic image with review text.Experimental results show that our method brings substantial performance improvement over strong baselines.The results also indicate that the logographic image is very important for exploring the internal structure and external relations from the character sequence.
Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking

Pairing a lexical retriever with a neural reranking model has set state-of-the-art performance on large-scale information retrieval datasets.This pipeline covers scenarios like question answering or navigational queries, however, for information-seeking scenarios, users often provide information on whether a document is relevant to their query in form of clicks or explicit feedback.Therefore, in this work, we explore how relevance feedback can be directly integrated into neural re-ranking models by adopting few-shot and parameterefficient learning techniques.Specifically, we introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant.Further, we explore Cross-Encoder models that we pre-train using meta-learning and subsequently fine-tune for each query, training only on the feedback documents.To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.Extensive experiments demonstrate that integrating relevance feedback directly in neural re-ranking models improves their performance, and fusing lexical ranking with our best performing neural reranker outperforms all other methods by 5.2%
Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models

Structural probing work has found evidence for latent syntactic information in pre-trained language models.However, much of this analysis has focused on monolingual models, and analyses of multilingual models have employed correlational methods that are confounded by the choice of probing tasks.In this study, we causally probe multilingual language models (XGLM and multilingual BERT) as well as monolingual BERT-based models across various languages; we do this by performing counterfactual perturbations on neuron activations and observing the effect on models' subjectverb agreement probabilities.We observe where in the model and to what extent syntactic agreement is encoded in each language.We find significant neuron overlap across languages in autoregressive multilingual language models, but not masked language models.We also find two distinct layer-wise effect patterns and two distinct sets of neurons used for syntactic agreement, depending on whether the subject and verb are separated by other tokens.Finally, we find that behavioral analyses of language models are likely underestimating how sensitive masked language models are to syntactic information.
QASem Parsing: Text-to-text Modeling of QA-based Semantics

Various works suggest the appeal of incorporating explicit semantic representations when addressing challenging realistic NLP scenarios.Common approaches offer either comprehensive linguistically-based formalisms, like AMR, or alternatively Open-IE, which provides a shallow and partial representation.More recently, an appealing trend introduces semi-structured natural-language structures as an intermediate meaning-capturing representation, often in the form of questions and answers.In this work, we further promote this line of research by considering three prior QA-based semantic representations.These cover verbal, nominalized and discourse-based predications, regarded here as jointly providing a comprehensive representation of textual informationtermed QASem.To facilitate this perspective, we investigate how to best utilize pre-trained sequence-to-sequence language models, which seem particularly promising for generating representations that consist of natural language expressions (questions and answers).In particular, we examine and analyze input and output linearization strategies, as well as data augmentation and multitask learning for a scarce training data setup.Consequently, we release the first unified QASem parsing tool, easily applicable for downstream tasks that can benefit from an explicit semi-structured account of information units in text.
SPOCK @ Causal News Corpus 2022: Cause-Effect-Signal Span Detection Using Span-Based and Sequence Tagging Models

Understanding causal relationship is an importance part of natural language processing.We address the causal information extraction problem with different neural models built on top of pre-trained transformer-based language models for identifying Cause, Effect and Signal spans, from news data sets.We use the Causal News Corpus subtask 2 training data set to train span-based and sequence tagging models.Our span-based model based on pre-trained BERT base weights achieves an F1 score of 47.48 on the test set with an accuracy score of 36.87 and obtained 3rd place in the Causal News Corpus 2022 shared task.
T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation

Unavailability of parallel corpora for training text style transfer (TST) models is a very challenging yet common scenario.Also, TST models implicitly need to preserve the content while transforming a source sentence into the target style.To tackle these problems, an intermediate representation is often constructed that is devoid of style while still preserving the meaning of the source sentence.In this work, we study the usefulness of Abstract Meaning Representation (AMR) graph as the intermediate style agnostic representation.We posit that semantic notations like AMR are a natural choice for an intermediate representation.Hence, we propose T-STAR: a model comprising of two components, text-to-AMR encoder and a AMR-to-text decoder.We propose several modeling improvements to enhance the style agnosticity of the generated AMR.To the best of our knowledge, T-STAR is the first work that uses AMR as an intermediate representation for TST.With thorough experimental evaluation we show T-STAR significantly outperforms state of the art techniques by achieving on an average 15.2% higher content preservation with negligible loss (∼3%) in style accuracy.Through detailed human evaluation with 90, 000 ratings, we also show that T-STAR has upto 50% lesser hallucinations compared to state of the art TST models.
Decoding a Neural Retriever’s Latent Space for Query Suggestion

Neural retrieval models have superseded classic bag-of-words methods such as BM25 as the retrieval framework of choice.However, neural systems lack the interpretability of bagof-words models; it is not trivial to connect a query change to a change in the latent space that ultimately determines the retrieval results.To shed light on this embedding space, we learn a "query decoder" that, given a latent representation of a neural search engine, generates the corresponding query.We show that it is possible to decode a meaningful query from its latent representation and, when moving in the right direction in latent space, to decode a query that retrieves the relevant paragraph.In particular, the query decoder can be useful to understand "what should have been asked" to retrieve a particular paragraph from the collection.We employ the query decoder to generate a large synthetic dataset of query reformulations for MSMarco, leading to improved retrieval performance.On this data, we train a pseudo-relevance feedback (PRF) T5 model for the application of query suggestion that outperforms both query reformulation and PRF information retrieval baselines.
IM^2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation

Evaluation metrics shine the light on the best models and thus strongly influence the research directions, such as the recently developed dialogue metrics USR, FED, and GRADE.However, most current metrics evaluate the dialogue data as isolated and static because they only focus on a single quality or several qualities.To mitigate the problem, this paper proposes an interpretable, multi-faceted, and controllable framework IM 2 (Interpretable and M ulti-category Integrated M etric) to combine a large number of metrics which are good at measuring different qualities.The IM 2 framework first divides current popular dialogue qualities into different categories and then applies or proposes dialogue metrics to measure the qualities within each category and finally generates an overall IM 2 score.An initial version of IM 2 was submitted to the AAAI 2022 Track5.1@DSTC10challenge 1 and took the 2 nd place on both of the development and test leaderboard.After the competition, we develop more metrics and improve the performance of our model.We compare IM 2 with other 13 current dialogue metrics and experimental results show that IM 2 correlates more strongly with human judgments than any of them on each evaluated dataset 2 .
Language Model Detoxification in Dialogue with Contextualized Stance Control

To reduce the toxic degeneration in a pretrained Language Model (LM), previous work on Language Model detoxification has focused on reducing the toxicity of the generation itself (self-toxicity) without consideration of the context (Dathathri et al., 2020;Krause et al., 2020;Qian et al., 2022).As a result, a type of implicit offensive language where the generations support the offensive language in the context (Figure 1) is ignored.Different from the LM controlling tasks in previous work, where the desired attributes are fixed for generation, the desired stance of the generation depends on the offensiveness of the context.Therefore, we propose a novel control method to do contextdependent detoxification with the stance taken into consideration.We introduce meta prefixes to learn the contextualized stance control strategy and to generate the stance control prefix according to the input context.The generated stance prefix is then combined with the toxicity control prefix to guide the response generation.Experimental results show that our proposed method can effectively learn the contextdependent stance control strategies while keeping a low self-toxicity of the underlying LM.
Meta-Learning Fast Weight Language Models

Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance.However, it requires over 3x more compute than standard inference.We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation much more efficiently by expressing gradient updates as linear attention.A key improvement over dynamic evaluation is that FWLs can also be applied at training time so the model learns to make good use of gradient updates.FWLs can easily be added on top of existing transformer models, require relatively little extra compute or memory to run, and significantly improve language modeling perplexity.
Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding

Weakly supervised phrase grounding aims to learn an alignment between phrases in a caption and objects in a corresponding image using only caption-image annotations, i.e., without phrase-object annotations.Previous methods typically use a caption-image contrastive loss to indirectly supervise the alignment between phrases and objects, which hinders the maximum use of the intrinsic structure of the multimodal data and leads to unsatisfactory performance.In this work, we directly use the phrase-object contrastive loss in the condition that no positive annotation is available in the first place.Specifically, we propose a novel contrastive learning framework based on the expectation-maximization algorithm that adaptively refines the target prediction.Experiments on two widely used benchmarks, Flickr30K Entities and RefCOCO+, demonstrate the effectiveness of our framework.We obtain 63.05% top-1 accuracy on Flickr30K Entities and 59.51%/43.46% on RefCOCO+ TestA/TestB, outperforming the previous methods by a large margin, even surpassing a previous SoTA that uses a pre-trained visionlanguage model.Furthermore, we deliver a theoretical analysis of the effectiveness of our method from the perspective of the maximum likelihood estimate with latent variables.
Learning to Generate Overlap Summaries through Noisy Synthetic Data

Semantic Overlap Summarization (SOS) is a novel and relatively under-explored seq-to-seq task which entails summarizing common information from multiple alternate narratives.One of the major challenges for solving this task is the lack of existing datasets for supervised training.To address this challenge, we propose a novel data augmentation technique, which allows us to create large amount of synthetic data for training a seq-to-seq model that can perform the SOS task.Through extensive experiments using narratives from the news domain, we show that the models finetuned using the synthetic dataset provide significant performance improvements over the pre-trained vanilla summarization techniques and are close to the models fine-tuned on the golden training data; which essentially demonstrates the effectiveness of out proposed data augmentation technique for training seq-to-seq models on the SOS task.
When does Parameter-Efficient Transfer Learning Work for Machine Translation?

Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting large pre-trained models while only tuning a small number of parameters.They have been shown to be competitive with full model fine-tuning for many downstream tasks.However, prior work indicates that PEFTs may not work as well for machine translation (MT), and there is no comprehensive study showing when PEFTs work for MT.We conduct a comprehensive empirical study of PEFTs for MT, considering (1) various parameter budgets, (2) a diverse set of language-pairs, and (3) different pre-trained models.We find that 'adapters', in which small feed-forward networks are added after every layer, are indeed on par with full model fine-tuning when the parameter budget corresponds to 10% of total model parameters.Nevertheless, as the number of tuned parameters decreases, the performance of PEFTs decreases.The magnitude of this decrease depends on the language pair, with PEFTs particularly struggling for distantly related language-pairs.We find that using PEFTs with a larger pre-trained model outperforms full fine-tuning with a smaller model, and for smaller training data sizes, PEFTs outperform full fine-tuning for the same pre-trained model. 1
FormLM: Recommending Creation Ideas for Online Forms by Modelling Semantic and Structural Information

Online forms are widely used to collect data from human and have a multi-billion market.Many software products provide online services for creating semi-structured forms where questions and descriptions are organized by predefined structures.However, the design and creation process of forms is still tedious and requires expert knowledge.To assist form designers, in this work we present FormLM to model online forms (by enhancing pre-trained language model with form structural information) and recommend form creation ideas (including question / options recommendations and block type suggestion).For model training and evaluation, we collect the first public online form dataset with 62K online forms.Experiment results show that FormLM significantly outperforms general-purpose language models on all tasks, with an improvement by 4.71 on Question Recommendation and 10.6 on Block Type Suggestion in terms of ROUGE-1 and Macro-F1, respectively.
Character-centric Story Visualization via Visual Planning and Token Alignment

Story visualization advances the traditional text-to-image generation by enabling multiple image generation based on a complete story.This task requires machines to 1) understand long text inputs and 2) produce a globally consistent image sequence that illustrates the contents of the story.A key challenge of consistent story visualization is to preserve characters that are essential in stories.To tackle the challenge, we propose to adapt a recent work that augments Vector-Quantized Variational Autoencoders (VQ-VAE) with a text-tovisual-token (transformer) architecture.Specifically, we modify the text-to-visual-token module with a two-stage framework: 1) character token planning model that predicts the visual tokens for characters only; 2) visual token completion model that generates the remaining visual token sequence, which is sent to VQ-VAE for finalizing image generations.To encourage characters to appear in the images, we further train the two-stage framework with a character-token alignment objective.Extensive experiments and evaluations demonstrate that the proposed method excels at preserving characters and can produce higher quality image sequences compared with the strong baselines.Code can be found in https: //github.com/PlusLabNLP/VP-CSV
Are representations built from the ground up? An empirical examination of local composition in language models

Compositionality, the phenomenon where the meaning of a phrase can be derived from its constituent parts, is a hallmark of human language.At the same time, many phrases are non-compositional, carrying a meaning beyond that of each part in isolation.Representing both of these types of phrases is critical for language understanding, but it is an open question whether modern language models (LMs) learn to do so; in this work we examine this question.We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents.We find that the representation of a parent phrase can be predicted with some accuracy given an affine transformation of its children.While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case, indicating that LMs may not accurately distinguish between compositional and non-compositional phrases.We perform a variety of analyses, shedding light on when different varieties of LMs do and do not generate compositional representations, and discuss implications for future modeling work. 1
Semi-Supervised Lifelong Language Learning

Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially.However, existing lifelong language learning methods only focus on the supervised learning setting.Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored.In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data.We propose an unlabeled data enhanced lifelong learner to explore SSLL.Specially, we dedicate taskspecific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks.Experimental results on various language tasks demonstrate our model's effectiveness and superiority over competitive baselines under the new setting SSLL.We will release the code and data 1 .
Lexical Generalization Improves with Larger Models and Longer Training

While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap.Excessive utilization of such heuristics can lead to failure on challenging inputs.We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics.We also find that longer training leads models to abandon lexical overlap heuristics.Finally, we provide evidence that the disparity between models size has its source in the pre-trained model. 1