Reviewerly/exHarmony huggingface dataset:
MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding

Document images are a ubiquitous source of data where the text is organized in a complex hierarchical structure ranging from fine granularity (e.g., words), medium granularity (e.g., regions such as paragraphs or figures), to coarse granularity (e.g., the whole page).The spatial hierarchical relationships between content at different levels of granularity are crucial for document image understanding tasks.Existing methods learn features from either word-level or region-level but fail to consider both simultaneously.Word-level models are restricted by the fact that they originate from pure-text language models, which only encode the wordlevel context.In contrast, region-level models attempt to encode regions corresponding to paragraphs or text blocks into a single embedding, but they perform worse with additional word-level features.To deal with these issues, we propose MGDoc, a new multi-modal multigranular pre-training framework that encodes page-level, region-level, and word-level information at the same time.MGDoc uses a unified text-visual encoder to obtain multi-modal features across different granularities, which makes it possible to project the multi-granular features into the same hyperspace.To model the region-word correlation, we design a crossgranular attention mechanism and specific pretraining tasks for our model to reinforce the model of learning the hierarchy between regions and words.Experiments demonstrate that our proposed model can learn better features that perform well across granularities and lead to improvements in downstream tasks.
Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond

The shift towards publicly available text sources has enabled language processing at unprecedented scale, yet leaves under-serviced the domains where public and openly licensed data is scarce.Proactively collecting text data for research is a viable strategy to address this scarcity, but lacks systematic methodology taking into account the many ethical, legal and confidentiality-related aspects of data collection.Our work presents a case study on proactive data collection in peer reviewa challenging and under-resourced NLP domain.We outline ethical and legal desiderata for proactive data collection and introduce "Yes-Yes-Yes", the first donation-based peer reviewing data collection workflow that meets these requirements.We report on the implementation of Yes-Yes-Yes at ACL Rolling Review 1 and empirically study the implications of proactive data collection for the dataset size and the biases induced by the donation behavior on the peer reviewing platform.
Evaluating Parameter Efficient Learning for Generation

Parameter efficient learning methods (PERMs) have recently gained significant attention as they provide an efficient way for pre-trained language models (PLMs) to adapt to a downstream task.However, these conclusions are mostly drawn from in-domain evaluations over the full training set.In this paper, we present comparisons between PERMs and finetuning from three new perspectives: (1) the effect of sample and model size to in-domain evaluations, (2) generalization to unseen domains and new datasets, and (3) the faithfulness of generations.Our results show that for in-domain settings (a) there is a cross point of sample size for which PERMs will perform better than finetuning when training with fewer samples, and (b) larger PLMs have larger cross points.For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size.We also compare the faithfulness of generations and show that PERMs can achieve better faithfulness score than finetuning, especially for small training set, by as much as 6%.Finally, we apply Adapter to MT-NLG 530b (Smith et al., 2022) and achieve new state-of-the-art results on Xsum (Narayan et al., 2018) for all ROUGE scores (ROUGE-1 49.17, ROUGE-2 27.20, ROUGE-L 40.98).
CNLP-NITS-PP at WANLP 2022 Shared Task: Propaganda Detection in Arabic using Data Augmentation and AraBERT Pre-trained Model

In today's time, online users are regularly exposed to media posts that are propagandistic.Several strategies have been developed to promote safer media consumption in Arabic to combat this.However, there is a limited available multilabel annotated social media dataset.In this work, we have used a pretrained AraBERT twitter-base model on an expanded train data via data augmentation.Our team CNLP-NITS-PP, has achieved the third rank in subtask 1 at WANLP-2022, for propaganda detection in Arabic (shared task) in terms of micro-F1 score of 0.602.
When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks

Humans can reason compositionally whilst grounding language utterances to the real world.Recent benchmarks like ReaSCAN (Wu et al., 2021) use navigation tasks grounded in a grid world to assess whether neural models exhibit similar capabilities.In this work, we present a simple transformer-based model that outperforms specialized architectures on ReaSCAN and a modified version (Qiu et al., 2021) of gSCAN (Ruis et al., 2020).On analyzing the task, we find that identifying the target location in the grid world is the main challenge for the models.Furthermore, we show that a particular split in ReaSCAN, which tests depth generalization, is unfair.On an amended version of this split, we show that transformers can generalize to deeper input structures.Finally, we design a simpler grounded compositional generalization task, RefEx, to investigate how transformers reason compositionally.We show that a single self-attention layer with a single head generalizes to novel combinations of object attributes.Moreover, we derive a precise mathematical construction of the transformer's computations from the learned network.Overall, we provide valuable insights about the grounded compositional generalization task and the behaviour of transformers on it, which would be useful for researchers working in this area.
Investigating the Robustness of Natural Language Generation from Logical Forms via Counterfactual Samples

The aim of Logic2Text is to generate controllable and faithful texts conditioned on tables and logical forms, which not only requires a deep understanding of the tables and logical forms, but also warrants symbolic reasoning over the tables.State-of-the-art methods based on pre-trained models have achieved remarkable performance on the standard test dataset.However, we question whether these methods really learn how to perform logical reasoning, rather than just relying on the spurious correlations between the headers of the tables and operators of the logical form.To verify this hypothesis, we manually construct a set of counterfactual samples, which modify the original logical forms to generate counterfactual logical forms with rarely co-occurred table headers and logical operators.SOTA methods give much worse results on these counterfactual samples compared with the results on the original test dataset, which verifies our hypothesis.To deal with this problem, we firstly analyze this bias from a causal perspective, based on which we propose two approaches to reduce the model's reliance on the shortcut.The first one incorporates the hierarchical structure of the logical forms into the model.The second one exploits automatically generated counterfactual data for training.Automatic and manual experimental results on the original test dataset and the counterfactual dataset show that our method is effective to alleviate the spurious correlation.Our work points out the weakness of previous methods and takes a further step toward developing Logic2Text models with real logical reasoning ability.
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way

Summarization datasets are often assembled either by scraping naturally occurring publicdomain summaries-which are nearly always in difcult-to-work-with technical domainsor by using approximate heuristics to extract them from everyday text-which frequently yields unfaithful summaries.In this work, we turn to a slower but more straightforward approach to developing summarization benchmark data: We hire highly-qualied contractors to read stories and write original summaries from scratch.To amortize reading time, we collect ve summaries per document, with the rst giving an overview and the subsequent four addressing specic questions.We use this protocol to collect SQuAL-ITY, a dataset of question-focused summaries built on the same public-domain short stories as the multiple-choice dataset QuALITY (Pang et al., 2021b).Experiments with stateof-the-art summarization systems show that our dataset is challenging and that existing automatic evaluation metrics are weak indicators of quality.
CANarEx: Contextually Aware Narrative Extraction for Semantically Rich Text-as-data Applications

Narrative modelling is an area of active research, motivated by the acknowledgement of narratives as drivers of societal decision making.These research efforts conceptualize narratives as connected entity chains, and modeling typically focuses on the identification of entities and their connections within a text.An emerging approach to narrative modelling is the use of semantic role labeling (SRL) to extract Entity-Verb-Entity (E-V-Es) tuples from a text, followed by dimensionality reduction to reduce the space of entities and connections separately.This process penalises the semantic richness of narratives and discards much contextual information along the way.Here, we propose an alternate narrative extraction approach -CANarEx, incorporating a pipeline of common contextual constructs through coreference resolution, micro-narrative generation and clustering of these narratives through sentence embeddings.We evaluate our approach through testing the recovery of "narrative time-series clusters", mimicking a desirable text-as-data task.The evaluation framework leverages synthetic data generated using a GPT-3 model.The GPT-3 model is trained to generate similar sentences using a large dataset of news articles.The synthetic data maps to three topics in the news dataset.We then generate narrative time-series document cluster representations by mapping the synthetic data to three distinct signals synthetically injected into the testing corpus.Evaluation results demonstrate the superior ability of CANarEx to recover narrative time-series through reduced MSE and improved precision/recall relative to existing methods.The validity is further reinforced through ablation studies and qualitative analysis 1 .
Scientific and Creative Analogies in Pretrained Language Models

This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2.Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds.As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains.Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs).We find that state-ofthe-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.
PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation

Formal verse poetry imposes strict constraints on the meter and rhyme scheme of poems.Most prior work on generating this type of poetry uses existing poems for supervision, which are difficult to obtain for most languages and poetic forms.In this work, we propose an unsupervised approach to generate poems that follow any given meter and rhyme scheme, without requiring any poetic text for training.Our method works by splitting a regular, non-poetic corpus into phrases, prepending control codes that describe the length and end rhyme of each phrase, and training a transformer language model in the augmented corpus.The transformer learns to link the structure descriptor with the control codes to the number of lines, their length and their end rhyme.During inference, we build control codes for the desired meter and rhyme scheme, and condition our language model on them to generate formal verse poetry.Experiments in Spanish and Basque show that our approach is able to generate valid poems, which are often comparable in quality to those written by humans.
Boundary-Driven Table-Filling for Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) aims to extract the aspect terms along with the corresponding opinion terms and the expressed sentiments in the review, which is an important task in sentiment analysis.Previous research efforts generally address the ASTE task in an endto-end fashion through the table-filling formalization, in which the triplets are represented by a two-dimensional (2D) table of word-pair relations.Under this formalization, a term-level relation is decomposed into multiple independent word-level relations, which leads to relation inconsistency and boundary insensitivity in the face of multi-word aspect terms and opinion terms.To overcome these issues, we propose Boundary-Driven Table-Filling (BDTF), which represents each triplet as a relation region in the 2D table and transforms the ASTE task into detection and classification of relation regions.We also notice that the quality of the table representation greatly affects the performance of BDTF.Therefore, we develop an effective relation representation learning approach to learn the table representation, which can fully exploit both word-to-word interactions and relation-torelation interactions.Experiments on several public benchmarks show that the proposed approach achieves state-of-the-art performances 1 .
Hierarchical Phrase-Based Sequence-to-Sequence Learning

This paper describes a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. Our approach trains two models: a discriminative parser based on a bracketing transduction grammar whose derivation tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one. We use the same seq2seq model to translate at all phrase scales, which results in two inference modes: one mode in which the parser is discarded and only the seq2seq component is used at the sequence-level, and another in which the parser is combined with the seq2seq model. Decoding in the latter mode is done with the cube-pruned CKY algorithm, which is more involved but can make use of new translation rules during inference. We formalize our model as a source-conditioned synchronous grammar and develop an efficient variational inference algorithm for training. When applied on top of both randomly initialized and pretrained seq2seq models, we find that it performs well compared to baselines on small scale machine translation benchmarks.
Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure

With the availability of massive generaldomain dialogue data, pre-trained dialogue generation appears to be super appealing to transfer knowledge from the general domain to downstream applications.In most existing work, such transferable ability is mainly obtained by fitting a large model with hundreds of millions of parameters on massive data in an exhaustive way, leading to inefficient running and poor interpretability.This paper proposes a novel dialogue generation model with a latent structure that is easily transferable from the general domain to downstream tasks in a lightweight and transparent way.Experiments on two benchmarks validate the effectiveness of the proposed model.Thanks to the transferable latent structure, our model is able to yield better dialogue responses than four strong baselines in terms of both automatic and human evaluations, and our model with about 22% parameters particularly delivers a 5x speedup in running time compared with the strongest baseline.Moreover, the proposed model is explainable by interpreting the discrete latent variables.
Parameter-Efficient Tuning Makes a Good Classification Head

In recent years, pretrained models revolutionized the paradigm of natural language understanding (NLU), where we append a randomly initialized classification head after the pretrained backbone, e.g.BERT, and finetune the whole model.As the pretrained backbone makes a major contribution to the improvement, we naturally expect a good pretrained classification head can also benefit the training.However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining (LP-FT) ineffective.In this paper, we find that parameter-efficient tuning makes a good classification head, with which we can simply replace the randomly initialized heads for a stable performance gain.Our experiments demonstrate that the classification head jointly pretrained with parameter-efficient tuning consistently improves the performance on 9 tasks in GLUE and SuperGLUE.
A Hierarchical N-Gram Framework for Zero-Shot Link Prediction

Knowledge graphs typically contain a large number of entities but often cover only a fraction of all relations between them (i.e., incompleteness).Zero-shot link prediction (ZSLP) is a popular way to tackle the problem by automatically identifying unobserved relations between entities.Most recent approaches use textual features of relations (e.g., surface names or textual descriptions) as auxiliary information to improve the encoded representation.These methods lack robustness as they are bound to support only tokens from a fixed vocabulary and are unable to model out-of-vocabulary (OOV) words.Subword units such as character n-grams have the capability of generating more expressive representations for OOV words.Hence, in this paper, we propose a Hierarchical N-gram framework for Zero-Shot Link Prediction (HNZSLP) that leverages character n-gram information for ZSLP.Our approach works by first constructing a hierarchical n-gram graph from the surface name of relations.Subsequently, a new Transformer-based network models the hierarchical n-gram graph to learn a relation embedding for ZSLP.Experimental results show that our proposed HNZSLP method achieves state-of-the-art performance on two standard ZSLP datasets.
Discourse Relation Embeddings: Representing the Relations between Discourse Segments in Social Media

Discourse relations are typically modeled as a discrete class that characterizes the relation between segments of text (e.g.causal explanations, expansions).However, such predefined discrete classes limit the universe of potential relations and their nuanced differences.Adding higher-level semantic structure to modern contextual word embeddings, we propose representing discourse relations as points in high dimensional continuous space.However, unlike words, discourse relations often have no surface form (relations are inbetween two segments, often with no explicit word or phrase marker), presenting a challenge for existing embedding techniques.We present a novel method for automatically creating discourse relation embeddings (DiscRE), addressing the embedding challenge through a weakly supervised, multitask approach.Results show DiscRE representations obtain the best performance on Twitter discourse relation classification (macro F 1 = 0.76) and social media causality prediction (from F 1 = .79to .81),performing beyond modern sentence and word transformers, and capturing novel nuanced relations (e.g. relations at the intersection of causal explanations and counterfactuals).
Hierarchical Multi-Label Classification of Scientific Documents

Automatic topic classification has been studied extensively to assist managing and indexing scientific documents in a digital collection.With the large number of topics being available in recent years, it has become necessary to arrange them in a hierarchy.Therefore, the automatic classification systems need to be able to classify the documents hierarchically.In addition, each paper is often assigned to more than one relevant topic.For example, a paper can be assigned to several topics in a hierarchy tree.In this paper, we introduce a new dataset for hierarchical multi-label text classification (HMLTC) of scientific papers called SciHTC, which contains 186, 160 papers and 1, 233 categories from the ACM CCS tree.We establish strong baselines for HMLTC and propose a multi-task learning approach for topic classification with keyword labeling as an auxiliary task.Our best model achieves a Macro-F1 score of 34.57% which shows that this dataset provides significant research opportunities on hierarchical scientific topic classification.We make our dataset and code available on Github. 1
Language Models Are Poor Learners of Directional Inference

We examine LMs' competence of directional predicate entailments by supervised fine-tuning with prompts.Our analysis shows that contrary to their apparent success on standard NLI, LMs show limited ability to learn such directional inference; moreover, existing datasets fail to test directionality, and/or are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results.In response, we present BoOQA (Boolean Open QA), a robust multi-lingual evaluation benchmark for directional predicate entailments, extrinsic to existing training sets.On BoOQA, we establish baselines and show evidence of existing LM-prompting models being incompetent directional entailment learners, in contrast to entailment graphs, however limited by sparsity.
Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing.Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models.However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context.To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the crossattention modules in the upper layers.Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage.Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs.