Reviewerly/exHarmony huggingface dataset:
Few-Shot (Dis)Agreement Identification in Online Discussions with Regularized and Augmented Meta-Learning

Online discussions are abundant with opinions towards a common topic, and identifying (dis)agreement between a pair of comments enables many opinion mining applications.Realizing the increasing needs to analyze opinions for emergent new topics that however tend to lack annotations, we present the first metalearning approach for few-shot (dis)agreement identification that can be quickly applied to analyze opinions for new topics with few labeled instances.Furthermore, we enhance the metalearner's domain generalization ability from two perspectives.The first is domain-invariant regularization, where we design a lexiconbased regularization loss to enable the metalearner to learn domain-invariant cues.The second is domain-aware augmentation, where we propose domain-aware task augmentation for meta-training to learn domain-specific expressions.In addition to using an existing dataset, we also evaluate our approach on two very recent new topics, mask mandate and COVID vaccine, using our newly annotated datasets containing 1.5k and 1.4k SubReddits comment pairs respectively 1 .Extensive experiments on three domains/topics demonstrate the effectiveness of our meta-learning approach 2 .
Knowledge Graph Generation From Text

In this work we propose a novel end-to-end multi-stage Knowledge Graph (KG) generation system from textual inputs, separating the overall process into two stages.The graph nodes are generated first using pretrained language model, followed by a simple edge construction head, enabling efficient KG extraction from the text.For each stage we consider several architectural choices that can be used depending on the available training resources.We evaluated the model on a recent WebNLG 2020 Challenge dataset, matching the state-of-the-art performance on text-to-RDF generation task, as well as on New York Times (NYT) and a largescale TEKGEN datasets, showing strong overall performance, outperforming the existing baselines.We believe that the proposed system can serve as a viable KG construction alternative to the existing linearization or sampling-based graph generation approaches.
A Systematic Investigation of Commonsense Knowledge in Large Language Models

Language models (LMs) trained on large amounts of data (e.g., Brown et al., 2020;Patwary et al., 2021) have shown impressive performance on many NLP tasks under the zeroshot and few-shot setup.Here we aim to better understand the extent to which such models learn commonsense knowledge -a critical component of many NLP applications.We conduct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pretrained LMs, where we: (i) carefully control for the LMs' ability to exploit potential surface cues and annotation artefacts, and (ii) account for variations in performance that arise from factors that are not related to commonsense knowledge.Our findings highlight the limitations of pre-trained LMs in acquiring commonsense knowledge without task-specific supervision; furthermore, using larger models or few-shot evaluation are insufficient to achieve human-level commonsense performance.
DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

We present DiffusionBERT, a new generative masked language model based on discrete diffusion models.Diffusion models and many pretrained language models have a shared training objective, i.e., denoising, making it possible to combine the two powerful models and enjoy the best of both worlds.On the one hand, diffusion models offer a promising training strategy that helps improve the generation quality.On the other hand, pre-trained denoising language models (e.g., BERT) can be used as a good initialization that accelerates convergence.We explore training BERT to learn the reverse process of a discrete diffusion process with an absorbing state and elucidate several designs to improve it.First, we propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step based on the information of each token.Second, we investigate several designs of incorporating the time step into BERT.Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text (e.g., D3PM and Diffusion-LM) and previous generative masked language models in terms of perplexity and BLEU score.Promising results in conditional generation tasks show that DiffusionBERT can generate texts of comparable quality and more diverse than a series of established baselines.
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback

Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input.This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article.In this work we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems.We use reinforcement learning with reference-free, textual-entailment rewards to optimize for factual consistency and explore the ensuing tradeoffs, as improved consistency may come at the cost of less informative or more extractive summaries.Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience and conciseness of the generated summaries.
NLPeer: A Unified Resource for the Computational Study of Peer Review

Peer review constitutes a core component of scholarly publishing; yet it demands substantial expertise and training, and is susceptible to errors and biases.Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review.To remedy this, we introduce NLPEER -the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.In addition to the new datasets of paper drafts, cameraready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task.Our work paves the path towards systematic, multi-faceted, evidencebased study of peer review in NLP and beyond.The data 1 and code 2 are publicly available.
Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages

Building Natural Language Understanding (NLU) capabilities for Indic languages, which have a collective speaker base of more than one billion speakers is absolutely crucial.In this work, we aim to improve the NLU capabilities of Indic languages by making contributions along 3 important axes (i) monolingual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages.Specifically, we curate the largest monolingual corpora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families -a 2.3x increase over prior work, while supporting 12 additional languages.Next, we create a humansupervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages.Across languages and tasks, IndicX-TREME contains a total of 105 evaluation sets, of which 52 are new contributions to the literature.To the best of our knowledge, this is the first effort towards creating a standard benchmark for Indic languages that aims to test the multilingual zero-shot capabilities of pretrained language models.Finally, we train IndicBERT v2, a state-of-the-art model supporting all the languages.Averaged across languages and tasks, the model achieves an absolute improvement of 2 points over a strong baseline.The data and models are available at https:// github.com/AI4Bharat/IndicBERT.
Ranking-Enhanced Unsupervised Sentence Representation Learning

Unsupervised sentence representation learning has progressed through contrastive learning and data augmentation methods such as dropout masking.Despite this progress, sentence encoders are still limited to using only an input sentence when predicting its semantic vector.In this work, we show that the semantic meaning of a sentence is also determined by nearestneighbor sentences that are similar to the input sentence.Based on this finding, we propose a novel unsupervised sentence encoder, RankEncoder.RankEncoder predicts the semantic vector of an input sentence by leveraging its relationship with other sentences in an external corpus, as well as the input sentence itself.We evaluate RankEncoder on semantic textual benchmark datasets.From the experimental results, we verify that 1) RankEncoder achieves 80.07% Spearman's correlation, a 1.1% absolute improvement compared to the previous state-of-the-art performance, 2) RankEncoder is universally applicable to existing unsupervised sentence embedding methods, and 3) RankEncoder is specifically effective for predicting the similarity scores of similar sentence pairs.
RARR: Researching and Revising What Language Models Say, Using Language Models

Language models (LMs) now excel at many tasks such as question answering, reasoning, and dialog.However, they sometimes generate unsupported or misleading content.A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence.To enable attribution while still preserving all the powerful advantages of recent generation models, we propose RARR (Retrofit Attribution using Research and Revision), a system that 1) automatically finds attribution for the output of any text generation model, and 2) post-edits the output to fix unsupported content while preserving the original output as much as possible.When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models.Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.
Where’s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation

Many NLP pipelines split text into sentences as one of the crucial preprocessing steps.Prior sentence segmentation tools either rely on punctuation or require a considerable amount of sentence-segmented training data: both central assumptions might fail when porting sentence segmenters to diverse languages on a massive scale.In this work, we thus introduce a multilingual punctuation-agnostic sentence segmentation method, currently covering 85 languages, trained in a self-supervised fashion on unsegmented text, by making use of newline characters which implicitly perform segmentation into paragraphs.We further propose an approach that adapts our method to the segmentation in a given corpus by using only a small number (64-256) of sentence-segmented examples.The main results indicate that our method outperforms all the prior best sentence-segmentation tools by an average of 6.1% F1 points.Furthermore, we demonstrate that proper sentence segmentation has a point: the use of a (powerful) sentence segmenter makes a considerable difference for a downstream application such as machine translation (MT).By using our method to match sentence segmentation to the segmentation used during training of MT models, we achieve an average improvement of 2.3 BLEU points over the best prior segmentation tool, as well as massive gains over a trivial segmenter that splits text into equally sized blocks.* Work done during the time BM interned at Cohere.
A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction

Document-level relation extraction (DocRE)aims to extract relations among entities within a document, which is crucial for applications like knowledge graph construction.Existing methods usually assume that entities and their mentions are identified beforehand, which falls short of real-world applications.To overcome this limitation, we propose TAG, a novel tableto-graph generation model for joint extraction of entities and relations at document-level.To enhance the learning of task dependencies, TAG induces a latent graph among mentions, with different types of edges indicating different task information, which is further broadcast with a relational graph convolutional network.To alleviate the error propagation problem, we adapt the hierarchical agglomerative clustering algorithm to back-propagate task information at decoding stage.Experiments on the benchmark dataset, DocRED, demonstrate that TAG surpasses previous methods by a large margin and achieves state-of-the-art results 1 .
MvP: Multi-view Prompting Improves Aspect Sentiment Tuple Prediction

Generative methods greatly promote aspectbased sentiment analysis via generating a sequence of sentiment elements in a specified format.However, existing studies usually predict sentiment elements in a fixed order, which ignores the effect of the interdependence of the elements in a sentiment tuple and the diversity of language expression on the results.In this work, we propose Multi-view Prompting (MVP) that aggregates sentiment elements generated in different orders, leveraging the intuition of human-like problem-solving processes from different views.Specifically, MVP introduces element order prompts to guide the language model to generate multiple sentiment tuples, each with a different element order, and then selects the most reasonable tuples by voting.MVP can naturally model multi-view and multi-task as permutations and combinations of elements, respectively, outperforming previous task-specific designed methods on multiple ABSA tasks with a single model.Extensive experiments show that MVP significantly advances the state-of-the-art performance on 10 datasets of 4 benchmark tasks, and performs quite effectively in low-resource settings.Detailed evaluation verified the effectiveness, flexibility, and cross-task transferability of MVP.
Downstream Datasets Make Surprisingly Good Pretraining Corpora

For most natural language processing tasks, the dominant practice is to finetune large pretrained transformer models (e.g., BERT) using smaller downstream datasets.Despite the success of this approach, it remains unclear to what extent these gains are attributable to the massive background corpora employed for pretraining versus to the pretraining objectives themselves.This paper introduces a large-scale study of self-pretraining, where the same (downstream) training data is used for both pretraining and finetuning.In experiments addressing both ELECTRA and RoBERTa models and 10 distinct downstream classification datasets, we observe that self-pretraining rivals standard pretraining on the BookWiki corpus (despite using around 10×-500× less data), outperforming the latter on 7 and 5 datasets, respectively.Surprisingly, these task-specific pretrained models often perform well on other tasks, including the GLUE benchmark.Self-pretraining also provides benefits on structured output prediction tasks such as question answering and commonsense inference, often providing more than 50% improvements compared to standard pretraining.Our results hint that often performance gains attributable to pretraining are driven primarily by the pretraining objective itself and are not always attributable to the use of external pretraining data in massive amounts.These findings are especially relevant in light of concerns about intellectual property and offensive content in web-scale pretraining data. 1
Span-level Aspect-based Sentiment Analysis via Table Filling

In this paper, we propose a novel span-level model for Aspect-Based Sentiment Analysis (ABSA), which aims at identifying the sentiment polarity of the given aspect.In contrast to conventional ABSA models that focus on modeling the word-level dependencies between an aspect and its corresponding opinion expressions, in this paper, we propose Table Filling BERT (TF-BERT), which considers the consistency of multi-word opinion expressions at the span-level.Specially, we learn the span representations with a table filling method, by constructing an upper triangular table for each sentiment polarity, of which the elements represent the sentiment intensities of the specific sentiment polarity for all spans in the sentence.Two methods are then proposed, including tabledecoding and table-aggregation, to filter out target spans or aggregate each table for sentiment polarity classification.In addition, we design a sentiment consistency regularizer to guarantee the sentiment consistency of each span for different sentiment polarities.Experimental results on three benchmarks demonstrate the effectiveness of our proposed model.
Minanto at SemEval-2023 Task 2: Fine-tuning XLM-RoBERTa for Named Entity Recognition on English Data

Within the scope of the shared task Multi-CoNER II our aim was to improve the recognition of named entities in English.We as team Minanto fine-tuned a cross-lingual model for Named Entity Recognition on English data and achieved an average F1 score of 51.47% in the final submission.We found that a monolingual model works better on English data than a cross-lingual and that the input of external data from earlier Named Entity Recognition tasks provides only minor improvements.In this paper we present our system, discuss our results and analyze the impact of external data.
Grounded Multimodal Named Entity Recognition on Social Media

In recent years, Multimodal Named Entity Recognition (MNER) on social media has attracted considerable attention.However, existing MNER studies only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction and insufficient for entity disambiguation.To solve these issues, in this work, we introduce a Grounded Multimodal Named Entity Recognition (GM-NER) task.Given a text-image social post, GMNER aims to identify the named entities in text, their entity types, and their bounding box groundings in image (i.e., visual regions).To tackle the GMNER task, we construct a Twitter dataset based on two existing MNER datasets.Moreover, we extend four well-known MNER methods to establish a number of baseline systems and further propose a Hierarchical Index generation framework named H-Index, which generates the entity-type-region triples in a hierarchical manner with a sequence-tosequence model.Experiment results on our annotated dataset demonstrate the superiority of our H-Index framework over baseline systems on the GMNER task.
LECO: Improving Early Exiting via Learned Exits and Comparison-based Exiting Mechanism

Recently, dynamic early exiting has attracted much attention since it can accelerate the inference speed of pre-trained models (PTMs).However, previous work on early exiting has neglected the intermediate exits' architectural designs.In this work, we propose a novel framework, Learned Exits and COmparison-based early exiting (LECO) to improve PTMs' early exiting performances.First, to fully uncover the potentials of multi-exit BERT, we design a novel search space for intermediate exits and employ the idea of differentiable neural architecture search (DNAS) to design proper exit architectures for different intermediate layers automatically.Second, we propose a simpleyet-effective comparison-based early exiting mechanism (COBEE), which can help PTMs achieve better performance and speedup tradeoffs.Extensive experiments show that our LECO achieves the SOTA performances for multi-exit BERT training and dynamic early exiting.
DiffusionNER: Boundary Diffusion for Named Entity Recognition

In this paper, we propose DIFFUSIONNER, which formulates the named entity recognition task as a boundary-denoising diffusion process and thus generates named entities from noisy spans.During training, DIFFUSIONNER gradually adds noises to the golden entity boundaries by a fixed forward diffusion process and learns a reverse diffusion process to recover the entity boundaries.In inference, DIFFU-SIONNER first randomly samples some noisy spans from a standard Gaussian distribution and then generates the named entities by denoising them with the learned reverse diffusion process.The proposed boundary-denoising diffusion process allows progressive refinement and dynamic sampling of entities, empowering DIFFUSIONNER with efficient and flexible entity generation capability.Experiments on multiple flat and nested NER datasets demonstrate that DIFFUSIONNER achieves comparable or even better performance than previous state-of-the-art models 1 .
Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods

Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks.However, applying prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks and their controllability remains underexplored.Controllability in GEC is crucial for real-world applications, particularly in educational settings, where the ability to tailor feedback according to learner levels and specific error types can significantly enhance the learning process.This paper investigates the performance and controllability of promptbased methods with GPT-3 for GEC tasks using zero-shot and few-shot setting.We explore the impact of task instructions and examples on GPT-3's output, focusing on controlling aspects such as minimal edits, fluency edits, and learner levels.Our findings demonstrate that GPT-3 could effectively perform GEC tasks, outperforming existing supervised and unsupervised approaches.We also showed that GPT-3 could achieve controllability when appropriate task instructions and examples are given.
Large Language Models Are Reasoning Teachers

Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune-CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model’s ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models. Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher.