Reviewerly/exHarmony huggingface dataset:
Cross-document Event Coreference Search: Task, Dataset and Modeling

The task of Cross-document Coreference Resolution has been traditionally formulated as requiring to identify all coreference links across a given set of documents.We propose an appealing, and often more applicable, complementary set up for the task -Cross-document Coreference Search, focusing in this paper on event coreference.Concretely, given a mention in context of an event of interest, considered as a query, the task is to find all coreferring mentions for the query event in a large document collection.To support research on this task, we create a corresponding dataset, which is derived from Wikipedia while leveraging annotations in the available Wikipedia Event Coreference dataset (WEC-Eng).Observing that the coreference search setup is largely analogous to the setting of Open Domain Question Answering, we adapt the prominent Deep Passage Retrieval (DPR) model to our setting, as an appealing baseline.Finally, we present a novel model that integrates a powerful coreference scoring scheme into the DPR architecture, yielding improved performance.
A Minimal Model for Compositional Generalization on gSCAN

Whether neural networks are capable of compositional generalization has been a topic of much debate.Most previous studies on this subject investigate the generalization capabilities of state-of-the-art deep learning architectures.We here take a more bottom-up approach and design a minimal model that displays generalization on a compositional benchmark, namely, the gSCAN dataset.The model is a hybrid architecture that combines layers trained with gradient descent and a selective attention mechanism optimized with an evolutionary strategy.The architecture has around 60 times fewer trainable parameters than models previously tested on gSCAN, and achieves comparable accuracies on most test splits, even when trained only on a fraction of the dataset.On adverb to verb generalization accuracy, it outperforms previous approaches by 65 to 86%.Through ablation studies, neuron pruning, and error analyses, we show that weight decay and attention mechanisms facilitate compositional generalization by encouraging sparse representations divorced from irrelevant context.We find that the model's sample efficiency can mainly be attributed to its selective attention mechanism.
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

Despite recent explosion of interests in incontext learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive.Intuitively, groundtruth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the inputlabel correspondence is significantly less important than previously thought.Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in incontext learning.With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations.Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration.Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.
Understanding Cross-modal Interactions in V&amp;L Models that Generate Scene Descriptions

Image captioning models tend to describe images in an object-centric way, emphasising visible objects.But image descriptions can also abstract away from objects and describe the type of scene depicted.In this paper, we explore the potential of a state of the art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions.Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when objectcentric descriptions are generated.We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.
Natural Logic-guided Autoregressive Multi-hop Document Retrieval for Fact Verification

A key component of fact verification is the evidence retrieval, often from multiple documents.Recent approaches use dense representations and condition the retrieval of each document on the previously retrieved ones.The latter step is performed over all the documents in the collection, requiring storing their dense representations in an index, thus incurring a high memory footprint.An alternative paradigm is retrieve-and-rerank, where documents are retrieved using methods such as BM25, their sentences are reranked, and further documents are retrieved conditioned on these sentences, reducing the memory requirements.However, such approaches can be brittle as they rely on heuristics and assume hyperlinks between documents.We propose a novel retrieve-and-rerank method for multi-hop retrieval, that consists of a retriever that jointly scores documents in the knowledge source and sentences from previously retrieved documents using an autoregressive formulation and is guided by a proof system based on natural logic that dynamically terminates the retrieval process if the evidence is deemed sufficient.This method is competitive with current state-of-the-art methods on FEVER, HoVer and FEVEROUS-S, while using 5 to 10 times less memory than competing systems.Evaluation on an adversarial dataset indicates improved stability of our approach compared to commonly deployed thresholdbased methods.Finally, the proof system helps humans predict model decisions correctly more often than using the evidence alone.Autoregressive Retriever
Leveraging Only the Category Name for Aspect Detection through Prompt-based Constrained Clustering

Aspect category detection (ACD) aims to automatically identify user-concerned aspects from online reviews, which is of great value for evaluating the fine-grained performance of a product.The most recent solutions tackle this problem via weakly supervised methods, achieving remarkable improvement over unsupervised methods.However, a closer look at these methods reveals that the required human efforts are nontrivial and can sometimes be hard to obtain.In this study, we explore the possibility of minimizing human guidance while improving detection performance, with a deep clustering method that relies merely on the category name of each aspect and a pretrained language model (LM).The LM, combined with prompt techniques, is employed as a knowledge base to automatically generate constraints for clustering, as well as to provide a representation space to perform the clustering.Our method (1) extracts extensive keywords to expand our understanding of each aspect, (2) automatically generates instance-level and concept-level constraints for clustering, and (3) trains the clustering model with the above constraints.We demonstrate the capability of the proposed framework through extensive experiments on four benchmark datasets across nine domains.Our model not only performs noticeably better than existing unsupervised approaches but also considerably surpasses weakly supervised methods that require more human efforts.Our code is available at:
Sparse Interventions in Language Models with Differentiable Masking

There has been a lot of interest in understanding what information is captured by hidden representations of language models (LMs).Typically, interpretation methods i) do not guarantee that the model actually uses the information found to be encoded, and ii) do not discover small subsets of neurons responsible for a considered phenomenon.Inspired by causal mediation analysis, we propose a method that discovers a small subset of neurons within a neural LM responsible for a particular linguistic phenomenon, i.e., subsets causing a change in the corresponding token emission probabilities.We use a differentiable relaxation to approximately search through the combinatorial space.An L 0 regularization term ensures that the search converges to discrete and sparse solutions.We apply our method to analyze subject-verb number agreement and gender bias detection in LSTMs.We observe that it is fast and finds better solutions than alternatives such as REINFORCE and Integrated Gradients.Our experiments confirm that each of these phenomena is mediated through a small subset of neurons that do not play any other discernible role.
A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach

Relation extraction has the potential for largescale knowledge graph construction, but current methods do not consider the qualifier attributes for each relation triplet, such as time, quantity or location.The qualifiers form hyperrelational facts which better capture the rich and complex knowledge graph structure.For example, the relation triplet (Leonard Parker, Educated At, Harvard University) can be factually enriched by including the qualifier (End Time, 1967).Hence, we propose the task of hyper-relational extraction to extract more specific and complete facts from text.To support the task, we construct HyperRED, a large-scale and general-purpose dataset.Existing models cannot perform hyper-relational extraction as it requires a model to consider the interaction between three entities.Hence, we propose Cu-beRE, a cube-filling model inspired by tablefilling approaches and explicitly considers the interaction between relation triplets and qualifiers.To improve model scalability and reduce negative class imbalance, we further propose a cube-pruning method.Our experiments show that CubeRE outperforms strong baselines and reveal possible directions for future research.
Evaluating the Knowledge Dependency of Questions

The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly.However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disregard their educational value.They fail to evaluate the MCQ's ability to assess the student's knowledge of the corresponding target fact.To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA), which measures the MCQ's answerability given knowledge of the target fact.Specifically, we first show how to measure KDA based on student responses from a human survey.Then, we propose two automatic evaluation metrics, KDA disc and KDA cont , that approximate KDA by leveraging pre-trained language models to imitate students' problem-solving behavior.Through our human studies, we show that KDA disc and KDA cont have strong correlations with both (1) KDA and (2) usability in an actual classroom setting, labeled by experts.Furthermore, when combined with ngram based similarity metrics, KDA disc and KDA cont are shown to have a strong predictive power for various expert-labeled MCQ quality measures.
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis

Pre-trained language models (PLMs) have gained increasing popularity due to their compelling prediction performance in diverse natural language processing (NLP) tasks.When formulating a PLM-based prediction pipeline for NLP tasks, it is also crucial for the pipeline to minimize the calibration error, especially in safety-critical applications.That is, the pipeline should reliably indicate when we can trust its predictions.In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more.Although prior work has looked into some of these considerations, they usually draw conclusions based on a limited scope of empirical studies.There still lacks a holistic analysis on how to compose a wellcalibrated PLM-based prediction pipeline.To fill this void, we compare a wide range of popular options for each consideration based on three prevalent NLP classification tasks and the setting of domain shift.In response, we recommend the following: (1) use ELECTRA for PLM encoding, (2) use larger PLMs if possible, (3) use Temp Scaling as the uncertainty quantifier, and (4) use Focal Loss for fine-tuning.
Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention

Recently, powerful Transformer architectures have proven superior in generating high-quality sentences.Nevertheless, these models tend to produce dull high-frequency phrases, severely hurting the diversity and novelty of generated text.In this work, we dig into the intrinsic mechanism of this problem and found that sparser attention values in Transformer could improve diversity.To understand such a phenomenon, we first conduct both empirical and theoretical analysis and then attribute it to representation degeneration caused by the attentive mixture of the hidden states during training.We term this process the Trap of Mediocrity.To escape from such a trap, we introduce a novel attention regularization loss to control the sharpness of the attention distribution, which is transparent to model structures and can be easily implemented within 20 lines of python code.We prove that this method could be mathematically regarded as learning a Bayesian approximation of posterior attention.Experiments show that our method improved the diversity and novelty of the generated text while maintaining comparable quality on a variety of conditional and unconditional generation tasks.
Do Decoding Algorithms Capture Discourse Structure in Multi-Modal Tasks? A Case Study of Image Paragraph Generation

This paper describes insights into how different inference algorithms structure discourse in image paragraphs.We train a multi-modal transformer and compare 11 variations of decoding algorithms.We propose to evaluate image paragraphs not only with standard automatic metrics, but also with a more extensive, "under the hood" analysis of the discourse formed by sentences.Our results show that while decoding algorithms can be unfaithful to the reference texts, they still generate grounded descriptions, but they also lack understanding of the discourse structure and differ from humans in terms of attentional structure over images.
Explore Unsupervised Structures in Pretrained Models for Relation Extraction

Syntactic trees have been widely applied in relation extraction (RE).However, since parsing qualities are not stable on different text domains and a pre-defined grammar may not well fit the target relation schema, the introduction of syntactic structures sometimes fails to improve RE performances consistently.In this work, we study RE models with various unsupervised structures mined from pre-trained language models (e.g., BERT).We show that, similar to syntactic trees, unsupervised structures are quite informative for RE task: they are able to obtain competitive (even the best) performance scores on benchmark RE datasets (ACE05, WebNLG, SciERC).We also conduct detailed analyses on their abilities of adapting new RE domains and influence of noise links in those structures.The results suggest that unsupervised structures are reasonable alternatives of commonly used syntactic structures in relation extraction models 1 .
DialogueGAT: A Graph Attention Network for Financial Risk Prediction by Modeling the Dialogues in Earnings Conference Calls

Financial risk prediction is an essential task for risk management in capital markets.While traditional prediction models are built based on the hard information of numerical data, recent studies have shown that the soft information of verbal cues in earnings conference calls is significant for predicting market risk due to its less constrained fashion and direct interaction between managers and analysts.However, most existing models mainly focus on extracting useful semantic information from the textual conference call transcripts but ignore their subtle yet important information of dialogue structures.To bridge this gap, we develop a graph attention network called DialogueGAT for financial risk prediction by simultaneously modeling the speakers and their utterances in dialogues in conference calls.Different from previous studies, we propose a new method for constructing the graph of speakers and utterances in a dialogue, and design contextual attention at both speaker and utterance levels for disentangling their effects on the downstream prediction task.For model evaluation, we extend an existing dataset of conference call transcripts by adding the dialogue structure and speaker information.Empirical results on our dataset of S&P1500 companies demonstrate the superiority of our proposed model over competitive baselines from the extant literature.
The Geometry of Multilingual Language Model Representations

We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language.Using XLM-R as a case study, we show that languages occupy similar linear subspaces after mean-centering, evaluated based on causal effects on language modeling performance and direct comparisons between subspaces for 88 languages.The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies.Shifting representations by language means is sufficient to induce token predictions in different languages.However, we also identify stable languageneutral axes that encode information such as token positions and part-of-speech.We visualize representations projected onto languagesensitive and language-neutral axes, identifying language family and part-of-speech clusters, along with spirals, toruses, and curves representing token position information.These results demonstrate that multilingual language models encode information along orthogonal language-sensitive and language-neutral axes, allowing the models to extract a variety of features for downstream tasks and cross-lingual transfer learning.
Predicting Fine-Tuning Performance with Probing

Large NLP models have recently shown impressive performance in language understanding tasks, typically evaluated by their finetuned performance.Alternatively, probing has received increasing attention as being a lightweight method for interpreting the intrinsic mechanisms of large NLP models.In probing, post-hoc classifiers are trained on "out-ofdomain" datasets that diagnose specific abilities.While probing the language models has led to insightful findings, they appear disjointed from the development of models.This paper explores the utility of probing deep NLP models to extract a proxy signal widely used in model development -the fine-tuning performance.We find that it is possible to use the accuracies of only three probing tests to predict the fine-tuning performance with errors 40% -80% smaller than baselines.We further discuss possible avenues where probing can empower the development of deep NLP models.
Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble

Out-of-distribution (OOD) detection aims to discern outliers from the intended data distribution, which is crucial to maintaining high reliability and a good user experience.Most recent studies in OOD detection utilize the information from a single representation that resides in the penultimate layer to determine whether the input is anomalous or not.Although such a method is straightforward, the potential of diverse information in the intermediate layers is overlooked.In this paper, we propose a novel framework based on contrastive learning that encourages intermediate features to learn layerspecialized representations and assembles them implicitly into a single representation to absorb rich information in the pre-trained language model.Extensive experiments in various intent classification and OOD datasets demonstrate that our approach is significantly more effective than other works.The source code for our model is available online.
“Am I Answering My Job Interview Questions Right?”: A NLP Approach to Predict Degree of Explanation in Job Interview Responses

Providing the right amount of explanation in an employment interview can help the interviewee effectively communicate their skills and experience to the interviewer and convince that she/he is the right candidate for the job.This paper examines natural language processing (NLP) approaches, including word-based tokenization, lexicon-based representations, and pre-trained embeddings with deep learning models, for detecting the degree of explanation in a job interview response.These are exemplified in a study of 24 military veterans who are the focal group of this study, since they can experience unique challenges in job interviews due to the unique verbal communication style that is prevalent in the military.Military veterans participated in mock interviews with industry recruiters and data from these interviews were transcribed and analyzed.Results indicate that the feasibility of automated NLP methods for detecting the degree of explanation in an interview response.Features based on tokenizer analysis are the most effective in detecting underexplained responses (i.e., 0.29 F1-score), while lexicon-based methods depict the higher performance in detecting over-explanation (i.e., 0.51 F1-score).Findings from this work lay the foundation for the design of intelligent assistive technologies that can provide personalized learning pathways to job candidates, especially those belonging to sensitive or underrepresented populations, and helping them succeed in employment job interviews, ultimately contributing to an inclusive workforce.
Modular and Parameter-Efficient Fine-Tuning for NLP Models

State-of-the-art language models in NLP perform best when fine-tuned even on small datasets, but due to their increasing size, finetuning and downstream usage have become extremely compute-intensive.Being able to efficiently and effectively fine-tune the largest pretrained models is thus key in order to reap the benefits of the latest advances in NLP.In this tutorial, we provide a comprehensive overview of parameter-efficient fine-tuning methods.We highlight their similarities and differences by presenting them in a unified view.We explore the benefits and usage scenarios of a neglected property of such parameterefficient models-modularity-such as composition of modules to deal with previously unseen data conditions.We finally highlight how both properties-parameter efficiency and modularity-can be useful in the real-world setting of adapting pre-trained models to under-represented languages and domains with scarce annotated data for several downstream applications. 1