Reviewerly/exHarmony huggingface dataset:
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge

We propose a novel open-domain question answering (ODQA) framework for answering single/multi-hop questions across heterogeneous knowledge sources.The key novelty of our method is the introduction of the intermediary modules into the current retriever-reader pipeline.Unlike previous methods that solely rely on the retriever for gathering all evidence in isolation, our intermediary performs a chain of reasoning over the retrieved set.Specifically, our method links the retrieved evidence with its related global context into graphs and organizes them into a candidate list of evidence chains.Built upon pretrained language models, our system achieves competitive performance on two ODQA datasets, OTT-QA and NQ, against tables and passages from Wikipedia.In particular, our model substantially outperforms the previous state-of-the-art on OTT-QA with an exact match score of 47.3 (45 % relative gain).† Part of this work is done during an
Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords

We propose a novel task-agnostic in-domain pre-training method that sits between generic pre-training and fine-tuning.Our approach selectively masks in-domain keywords, i.e., words that provide a compact representation of the target domain.We identify such keywords using KeyBERT (Grootendorst, 2020).We evaluate our approach using six different settings: three datasets combined with two distinct pretrained language models (PLMs).Our results reveal that the fine-tuned PLMs adapted using our in-domain pre-training strategy outperform PLMs that used in-domain pre-training with random masking as well as those that followed the common pre-train-then-fine-tune paradigm.Further, the overhead of identifying in-domain keywords is reasonable, e.g., 7-15% of the pretraining time (for two epochs) for BERT Large (Devlin et al., 2019). 1
Recurrent Attention Networks for Long-text Modeling

Self-attention-based models have achieved remarkable progress in short-text mining.However, the quadratic computational complexities restrict their application in long text processing.Prior works have adopted the chunking strategy to divide long documents into chunks and stack a self-attention backbone with the recurrent structure to extract semantic representation.Such an approach disables parallelization of the attention mechanism, significantly increasing the training cost and raising hardware requirements.Revisiting the self-attention mechanism and the recurrent structure, this paper proposes a novel long-document encoding model, Recurrent Attention Network (RAN), to enable the recurrent operation of self-attention.Combining the advantages from both sides, the welldesigned RAN is capable of extracting global semantics in both token-level and documentlevel representations, making it inherently compatible with both sequential and classification tasks, respectively.Furthermore, RAN is computationally scalable as it supports parallelization on long document processing.Extensive experiments demonstrate the long-text encoding ability of the proposed RAN model on both classification and sequential tasks, showing its potential for a wide range of applications.
Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood

Recognizing flat, overlapped and discontinuous entities uniformly has been paid increasing attention.Among these works, Seq2Seq formulation prevails for its flexibility and effectiveness.It arranges the output entities into a specific target sequence.However, it introduces bias by assigning all the probability mass to the observed sequence.To alleviate the bias, previous works either augment the data with possible sequences or resort to other formulations.In this paper, we stick to the Seq2Seq formulation and propose a reranking-based approach.It redistributes the likelihood among candidate sequences depending on their performance via a contrastive loss.Extensive experiments show that our simple yet effective method consistently boosts the baseline, and yields competitive or better results compared with the state-of-the-art methods on 8 widelyused datasets for Named Entity Recognition.
Evaluating Paraphrastic Robustness in Textual Entailment Models

We present P aRT E, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing.We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning.We use the evaluation set to determine if RTE models' predictions change when examples are paraphrased.In our experiments, contemporary models change their predictions on 8-16% of paraphrased examples, indicating that there is still room for improvement.
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering

Given its effectiveness on knowledge-intensive natural language processing tasks, dense retrieval models have become increasingly popular. Specifically, the de-facto architecture for open-domain question answering uses two isomorphic encoders that are initialized from the same pretrained model but separately parameterized for questions and passages. This biencoder architecture is parameter-inefficient in that there is no parameter sharing between encoders. Further, recent studies show that such dense retrievers underperform BM25 in various settings. We thus propose a new architecture, Task-Aware Specialization for dEnse Retrieval (TASER), which enables parameter sharing by interleaving shared and specialized blocks in a single encoder. Our experiments on five question answering datasets show that TASER can achieve superior accuracy, surpassing BM25, while using about 60% of the parameters as bi-encoder dense retrievers. In out-of-domain evaluations, TASER is also empirically more robust than bi-encoder dense retrievers. Our code is available at https://github.com/microsoft/taser.
AMPERE: AMR-Aware Prefix for Generation-Based Event Argument Extraction Model

Event argument extraction (EAE) identifies event arguments and their specific roles for a given event.Recent advancement in generationbased EAE models has shown great performance and generalizability over classificationbased models.However, existing generationbased EAE models mostly focus on problem reformulation and prompt design, without incorporating additional information that has been shown to be effective for classification-based models, such as the abstract meaning representation (AMR) of the input passages.Incorporating such information into generation-based models is challenging due to the heterogeneous nature of the natural language form prevalently used in generation-based models and the structured form of AMRs.In this work, we study strategies to incorporate AMR into generationbased EAE models.We propose AMPERE, which generates AMR-aware prefixes for every layer of the generation model.Thus, the prefix introduces AMR information to the generationbased EAE model and then improves the generation.We also introduce an adjusted copy mechanism to AMPERE to help overcome potential noises brought by the AMR graph.Comprehensive experiments and analyses on ACE2005 and ERE datasets show that AMPERE can get 4% -10% absolute F1 score improvements with reduced training data and it is in general powerful across different training sizes.Model Type Development Set Test Set 5% 10% 20% 30% 50% 100% 5% 10% 20% 30% 50% 100% Argument Classification F1-Score (%) in ACE05-E DyGIE++ (Wadden et al., 2019) Cls 34.6 48.5 52.5 57.5 57.9 60.0 29.3 42.4 49.5 53.2 54.5 57.4 OneIE (Lin et al., 2020) Cls 38.6 56.0 63.2 67.6 70.4 71.8 34.6 50.0 59.6 63.0 68.4 70.6 Query and Extract (Wang et al., 2022) Cls 10.5 27.7 37.6 50.0 54.6 61.7 11.0 20.9 34.3 44.3 49.6 59.1 AMR-IE (Zhang and Ji, 2021) Cls 40.0 56.3 61.3 67.4 70.6 73.1 36.8 48.5 58.3 62.6 66.1 70.3 PAIE (Ma et al., 2022) Gen 46.6 57.6 64.6 69.3 70.3 74.1 46.3 56.3 62.8 65.8 69.1 72.1 DEGREE (Hsu et al., 2022a) Gen 41.4 56.8 62.5 68.9 70.5 73.8 41.7 57.7 58.9 65.8 68.2 73.0 AMPERE (AMRBART) Gen 52.3 61.5 67.2 71.2 72.7 75.5 52.4 61.0 66.4 69.7 71.1 73.4 AMPERE (RoBERTa) Gen 53.2 61.5 66.6 71.8 72.5 76.6 53.4 61.7 66.4 69.5 71.9 74.2 Argument Classification F1-Score (%) in ERE-EN DyGIE++ (Wadden et al., 2019) Cls 42.2 45.4 49.0 50.1 51.5 56.8 40.0 44.6 49.5 52.0 53.7 56.0 OneIE (Lin et al., 2020) Cls 51.4 59.5 62.0 65.6 68.6 71.2 49.5 56.1 62.3 66.1 67.7 70.1 Query and Extract (Wang et al., 2022) Cls 22.0 37.3 41.2 49.4 57.0 65.0 19.7 34.0 42.4 50.1 57.7 64.3 AMR-IE (Zhang and Ji, 2021) Cls 44.8 55.2 56.8 65.2 67.6 70.1 44.1 53.7 60.4 65.6 68.9 71.5 DEGREE (Hsu et al., 2022a) Gen 57.2 62.5 63.9 67.1 70.2 73.3 57.5 63.9 67.4 69.1 73.3 74.9 AMPERE (AMRBART) Gen 62.4 66.8 66.6 68.8 70.8 73.6 62.9 66.7 68.5 71.
Linear Classifier: An Often-Forgotten Baseline for Text Classification

Large-scale pre-trained language models such as BERT are popular solutions for text classification.Due to the superior performance of these advanced methods, nowadays, people often directly train them for a few epochs and deploy the obtained model.In this opinion paper, we point out that this way may only sometimes get satisfactory results.We argue the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods.First, for many text data, linear methods show competitive performance, high efficiency, and robustness.Second, advanced models such as BERT may only achieve the best results if properly applied.Simple baselines help to confirm whether the results of advanced models are acceptable.Our experimental results fully support these points.
Improving Long Dialogue Summarization with Semantic Graph Representation

Although Large Language Models (LLMs) are successful in abstractive summarization of short dialogues, summarization of long dialogues remains challenging.To address this challenge, we propose a novel algorithm that processes complete dialogues comprising thousands of tokens into topic-segment-level Abstract Meaning Representation (AMR) graphs, which explicitly capture the dialogue structure, highlight salient semantics, and preserve highlevel information.We also develop a new textgraph attention to leverage both graph semantics and a pretrained LLM that exploits the text.Finally, we propose an AMR node selection loss used jointly with conventional crossentropy loss, to create additional training signals that facilitate graph feature encoding and content selection.Experiments show that our system outperforms the state-of-the-art models on multiple long dialogue summarization datasets, especially in low-resource settings, and generalizes well to out-of-domain data.
I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons

We propose a novel task, G4C, to study teacher-student natural language interactions in a goal-driven and grounded environment.Dungeons and Dragons (D&D), a role-playing game, provides an ideal setting to investigate such interactions.Here, the Dungeon Master (DM), i.e., the teacher, guides the actions of several players-students, each with their own personas and abilities-to achieve shared goals grounded in a fantasy world.Our approach is to decompose and model these interactions into (1) the DM's intent to guide players towards a given goal; (2) the DM's guidance utterance to the players expressing this intent; and (3) a theory-of-mind (ToM) model that anticipates the players' reaction to the guidance one turn into the future.We develop a novel reinforcement learning (RL) method for training a DM that generates guidance for players by rewarding utterances where the intent matches the ToM-anticipated player actions.Human and automated evaluations show that a DM trained to explicitly model intents and incorporate ToM of the players using RL generates better-quality guidance that is 3x more likely to fulfill the DM's intent than a vanilla natural language generation (NLG) approach.
Knowledge Graph-augmented Language Models for Complex Question Answering

Large language models have shown impressive abilities to reason over input text, however, they are prone to hallucinations.On the other hand, end-to-end knowledge graph question answering (KGQA) models output responses grounded in facts, but they still struggle with complex reasoning, such as comparison or ordinal questions.In this paper, we propose a new method for complex question answering where we combine a knowledge graph retriever based on an end-to-end KGQA model with a language model that reasons over the retrieved facts to return an answer.We observe that augmenting language model prompts with retrieved KG facts improves performance over using a language model alone by an average of 83%.In particular, we see improvements on complex questions requiring count, intersection, or multi-hop reasoning operations.
NonFactS: NonFactual Summary Generation for Factuality Evaluation in Document Summarization

Pre-trained abstractive summarization models can generate fluent summaries and achieve high ROUGE scores.Previous research has found that these models often generate summaries that are inconsistent with their context document and contain nonfactual information.To evaluate factuality in document summarization, a document-level Natural Language Inference (NLI) classifier can be used.However, training such a classifier requires largescale high-quality factual and nonfactual samples.To that end, we introduce NonFactS, a data generation model to synthesize nonfactual summaries given a context document and a human-annotated (reference) factual summary.Compared to previous methods, our nonfactual samples are more abstractive and more similar to their corresponding factual samples, resulting in state-of-the-art performance on two factuality evaluation benchmarks, FALSESUM and SUMMAC.Our experiments demonstrate that even without human-annotated summaries, NonFactS can use random sentences to generate nonfactual summaries and a classifier trained on these samples generalizes to out-ofdomain documents. 1
TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model

Language is by its very nature incremental in how it is produced and processed.This property can be exploited by NLP systems to produce fast responses, which has been shown to be beneficial for real-time interactive applications.Recent neural network-based approaches for incremental processing mainly use RNNs or Transformers.RNNs are fast but monotonic (cannot correct earlier output, which can be necessary in incremental processing).Transformers, on the other hand, consume whole sequences, and hence are by nature non-incremental.A restartincremental interface that repeatedly passes longer input prefixes can be used to obtain partial outputs, while providing the ability to revise.However, this method becomes costly as the sentence grows longer.In this work, we propose the Two-pass model for AdaPtIve Revision (TAPIR) and introduce a method to obtain an incremental supervision signal for learning an adaptive revision policy.Experimental results on sequence labelling show that our model has better incremental performance and faster inference speed compared to restartincremental Transformers, while showing little degradation on full sequences. 1
Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values

Although Shapley values have been shown to be highly effective for identifying harmful training instances, dataset size and model complexity constraints limit the ability to apply Shapleybased data valuation to fine-tuning large pretrained language models.To address this, we propose TS-DSHAPLEY, an algorithm that reduces computational cost of Shapley-based data valuation through: 1) an efficient samplingbased method that aggregates Shapley values computed from subsets for valuation of the entire training set, and 2) a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model.Our experiments applying TS-DSHAPLEY to select data for fine-tuning BERT-based language models on benchmark natural language understanding (NLU) datasets show that TS-DSHAPLEY outperforms existing data selection methods.Further, TS-DSHAPLEY can filter fine-tuning data to increase language model performance compared to training with the full fine-tuning dataset.
FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information

Dungeons & Dragons (D&D) is a tabletop roleplaying game with complex natural language interactions between players and hidden state information.Recent work has shown that large language models (LLMs) that have access to state information can generate higher quality game turns than LLMs that use dialog history alone.However, previous work used game state information that was heuristically created and was not a true gold standard game state.We present FIREBALL, a large dataset containing nearly 25,000 unique sessions from real D&D gameplay on Discord with true game state info.We recorded game play sessions of players who used the Avrae bot, which was developed to aid people in playing D&D online, capturing language, game commands and underlying game state information.We demonstrate that FIRE-BALL can improve natural language generation (NLG) by using Avrae state information, improving both automated metrics and human judgments of quality.Additionally, we show that LLMs can generate executable Avrae commands, particularly after finetuning.
LexSym: Compositionality as Lexical Symmetry

In tasks like semantic parsing, instruction following, and question answering, standard deep networks fail to generalize compositionally from small datasets.Many existing approaches overcome this limitation with model architectures that enforce a compositional process of sentence interpretation.In this paper, we present a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models.Informally, we prove that whenever a task can be solved by a compositional model, there is a corresponding data augmentation scheme-a procedure for transforming examples into other well-formed examples-that imparts compositional inductive bias on any model trained to solve the same task.We describe a procedure called LEXSYM that discovers these transformations automatically, then applies them to training data for ordinary neural sequence models.Unlike existing compositional data augmentation procedures, LEXSYM can be deployed agnostically across text, structured data, and even images.It matches or surpasses state-of-the-art, task-specific models on COGS semantic parsing, SCAN and ALCHEMY instruction following, and CLEVR-COGENT visual question answering datasets.
On the Limitations of Simulating Active Learning

Active learning (AL) is a human-and-modelin-the-loop paradigm that iteratively selects informative unlabeled data for human annotation, aiming to improve over random sampling.However, performing AL experiments with human annotations on-the-fly is a laborious and expensive process, thus unrealistic for academic research.An easy fix to this impediment is to simulate AL, by treating an already labeled and publicly available dataset as the pool of unlabeled data.In this position paper, we first survey recent literature and highlight the challenges across all different steps within the AL loop.We further unveil neglected caveats in the experimental setup that can significantly affect the quality of AL research.We continue with an exploration of how the simulation setting can govern empirical findings, arguing that it might be one of the answers behind the ever posed question "why do active learning algorithms sometimes fail to outperform random sampling?".We argue that evaluating AL algorithms on available labeled datasets might provide a lower bound as to their effectiveness in real data.We believe it is essential to collectively shape the best practices for AL research, particularly as engineering advancements in LLMs push the research focus towards data-driven approaches (e.g., data efficiency, alignment, fairness).In light of this, we have developed guidelines for future work.Our aim is to draw attention to these limitations within the community, in the hope of finding ways to address them.
Linguistic representations for fewer-shot relation extraction across domains

Recent work has demonstrated the positive impact of incorporating linguistic representations as additional context and scaffolding on the in-domain performance of several NLP tasks.We extend this work by exploring the impact of linguistic representations on cross-domain performance in a few-shot transfer setting.An important question is whether linguistic representations enhance generalizability by providing features that function as cross-domain pivots.We focus on the task of relation extraction on three datasets of procedural text in two domains, cooking and materials science.Our approach augments a popular transformerbased architecture by alternately incorporating syntactic and semantic graphs constructed by freely available off-the-shelf tools.We examine their utility for enhancing generalization, and investigate whether earlier findings, e.g. that semantic representations can be more helpful than syntactic ones, extend to relation extraction in multiple domains.We find that while the inclusion of these graphs results in significantly higher performance in few-shot transfer, both types of graph exhibit roughly equivalent utility.
Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training

Automatic text simplification systems help to reduce textual information barriers on the internet.However, for languages other than English, only few parallel data to train these systems exists.We propose a two-step approach to overcome this data scarcity issue.First, we fine-tuned language models on a corpus of German Easy Language, a specific style of German.Then, we used these models as decoders in a sequence-to-sequence simplification task.We show that the language models adapt to the style characteristics of Easy Language and output more accessible texts.Moreover, with the style-specific pre-training, we reduced the number of trainable parameters in text simplification models.Hence, less parallel data is sufficient for training.Our results indicate that pre-training on unaligned data can reduce the required parallel data while improving the performance on downstream tasks.
Harnessing the Power of BERT in the Turkish Clinical Domain: Pretraining Approaches for Limited Data Scenarios

Recent advancements in natural language processing (NLP) have been driven by large language models (LLMs), thereby revolutionizing the field.Our study investigates the impact of diverse pre-training strategies on the performance of Turkish clinical language models in a multi-label classification task involving radiology reports, with a focus on overcoming language resource limitations.Additionally, for the first time, we evaluated the simultaneous pre-training approach by utilizing limited clinical task data.We developed four models: TurkRadBERT-task v1, TurkRadBERT-task v2, TurkRadBERT-sim v1, and TurkRadBERT-sim v2.Our results revealed superior performance from BERTurk and TurkRadBERT-task v1, both of which leverage a broad general-domain corpus.Although task-adaptive pre-training is capable of identifying domain-specific patterns, it may be prone to overfitting because of the constraints of the task-specific corpus.Our findings highlight the importance of domainspecific vocabulary during pre-training to improve performance.They also affirmed that a combination of general domain knowledge and task-specific fine-tuning is crucial for optimal performance across various categories.This study offers key insights for future research on pre-training techniques in the clinical domain, particularly for low-resource languages.