Reviewerly/exHarmony huggingface dataset:
Probing for Constituency Structure in Neural Language Models

In this paper, we investigate to which extent contextual neural language models (LMs) implicitly learn syntactic structure.More concretely, we focus on constituent structure as represented in the Penn Treebank (PTB).Using standard probing techniques based on diagnostic classifiers, we assess the accuracy of representing constituents of different categories within the neuron activations of a LM such as RoBERTa.In order to make sure that our probe focuses on syntactic knowledge and not on implicit semantic generalizations, we also experiment on a PTB version that is obtained by randomly replacing constituents with each other while keeping syntactic structure, i.e., a semantically ill-formed but syntactically well-formed version of the PTB.We find that 4 pretrained transfomer LMs obtain high performance on our probing tasks even on manipulated data, suggesting that semantic and syntactic knowledge in their representations can be separated and that constituency information is in fact learned by the LM.Moreover, we show that a complete constituency tree can be linearly separated from LM representations. 1
Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction

Automatic peer-review aspect score prediction (PASP) of academic papers can be a helpful assistant tool for both reviewers and authors.Most existing works on PASP utilize supervised learning techniques.However, the limited number of peer-review data deteriorates the performance of PASP.This paper presents a novel semi-supervised learning (SSL) method that incorporates the Transformer fine-tuning into the Γ-model, a variant of the Ladder network, to leverage contextual features from unlabeled data.Backpropagation simultaneously minimizes the sum of supervised and unsupervised cost functions, it can be easily trained in an end-to-end fashion.The proposed method is evaluated on the PeerRead benchmark.The experimental results demonstrate that our model outperforms the supervised and naive semi-supervised learning baselines.Our source codes are available online 1 .
Dependency Parsing via Sequence Generation

Dependency parsing aims to extract syntactic dependency structure or semantic dependency structure for sentences.Existing methods for dependency parsing include transition-based method, graph-based method and sequence-tosequence method.These methods obtain excellent performance and we notice them belong to labeling method.Therefore, it may be very valuable and interesting to explore the possibility of using generative method to implement dependency parsing.In this paper, we propose to achieve Dependency Parsing (DP) via Sequence Generation (SG) by utilizing only the pre-trained language model without any auxiliary structures.We first explore different serialization designing strategies for converting parsing structures into sequences.Then we design dependency units and concatenate these units into the sequence for DPSG.We verify the DPSG is capable of parsing on widely used DP benchmarks, i.e., PTB, UD2.2, SDP15 and SemEval16.In addition, we also investigate the astonishing low-resource applicability of DPSG, which includes unsupervised cross-domain conducted on CODT and fewshot cross-task conducted on SDP15.Our research demonstrates that sequence generation is one of the effective methods to achieve dependency parsing.Our codes are available now. 1 .
Kernel Ridge Regression-Based Graph Dataset Distillation

The huge volume of emerging graph datasets has become a double-bladed sword for graph machine learning. On the one hand, it empowers the success of a myriad of graph neural networks (GNNs) with strong empirical performance. On the other hand, training modern graph neural networks on huge graph data is computationally expensive. How to distill the given graph dataset while retaining most of the trained models' performance is a challenging problem. Existing efforts try to approach this problem by solving meta-learning-based bilevel optimization objectives. A major hurdle lies in that the exact solutions of these methods are computationally intensive and thus, most, if not all, of them are solved by approximate strategies which in turn hurt the distillation performance. In this paper, inspired by the recent advances in neural network kernel methods, we adopt a kernel ridge regression-based meta-learning objective which has a feasible exact solution. However, the computation of graph neural tangent kernel is very expensive, especially in the context of dataset distillation. As a response, we design a graph kernel, named LiteGNTK, tailored for the dataset distillation problem which is closely related to the classic random walk graph kernel. An effective model named Kernel rıdge regression-based graph Dataset Distillation (KIDD) and its variants are proposed. KIDD shows nice efficiency in both the forward and backward propagation processes. At the same time, KIDD shows strong empirical performance over 7 real-world datasets compared with the state-of-the-art distillation methods. Thanks to the ability to find the exact solution of the distillation objective, the learned training graphs by KIDD can sometimes even outperform the original whole training set with as few as 1.65% training graphs.
FairPrism: Evaluating Fairness-Related Harms in Text Generation

It is critical to measure and mitigate fairnessrelated harms caused by AI text generation systems, including stereotyping and demeaning harms.To that end, we introduce FairPrism, a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality.FairPrism aims to address several limitations of existing datasets for measuring and mitigating fairness-related harms, including improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent.FairPrism's annotations include the extent of stereotyping and demeaning harms, the demographic groups targeted, and appropriateness for different applications.The annotations also include specific harms that occur in interactive contexts and harms that raise normative concerns when the "speaker" is an AI system.Due to its precision and granularity, FairPrism can be used to diagnose (1) the types of fairnessrelated harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods, both of which we illustrate through case studies.Finally, the process we followed to develop FairPrism offers a recipe for building improved datasets for measuring and mitigating harms caused by AI systems.
WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering

Answering non-factoid questions (NFQA) is a challenging task, requiring passage-level answers that are difficult to construct and evaluate.Search engines may provide a summary of a single web page, but many questions require reasoning across multiple documents.Meanwhile, modern models can generate highly coherent and fluent, but often factually incorrect answers that can deceive even non-expert humans.There is a critical need for highquality resources for multi-document NFQA (MD-NFQA) to train new models and evaluate answers' grounding and factual consistency in relation to supporting documents.To bridge this gap, we present WIKIHOWQA, 1 a new multi-document NFQA benchmark built on WikiHow, a website dedicated to answering "how-to" questions.The benchmark includes 11,746 human-written answers along with 74,527 supporting documents.We describe the unique challenges of the resource, provide strong baselines, and propose a novel human evaluation framework that utilizes highlighted relevant supporting passages to mitigate issues such as assessor unfamiliarity with the question topic.All code and data, including the automatic code for preparing the human evaluation, are publicly available.
Clinical Note Owns its Hierarchy: Multi-Level Hypergraph Neural Networks for Patient-Level Representation Learning

Leveraging knowledge from electronic health records (EHRs) to predict a patient's condition is essential to the effective delivery of appropriate care.Clinical notes of patient EHRs contain valuable information from healthcare professionals, but have been underused due to their difficult contents and complex hierarchies.Recently, hypergraph-based methods have been proposed for document classifications.Directly adopting existing hypergraph methods on clinical notes cannot sufficiently utilize the hierarchy information of the patient, which can degrade clinical semantic information by (1) frequent neutral words and (2) hierarchies with imbalanced distribution.Thus, we propose a taxonomy-aware multi-level hypergraph neural network (TM-HGNN), where multi-level hypergraphs assemble useful neutral words with rare keywords via note and taxonomy level hyperedges to retain the clinical semantic information.The constructed patient hypergraphs are fed into hierarchical message passing layers for learning more balanced multi-level knowledge at the note and taxonomy levels.We validate the effectiveness of TM-HGNN by conducting extensive experiments with MIMIC-III dataset on benchmark in-hospital-mortality prediction.
Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation-Nearest Neighbor Neural Machine Translation

k-Nearest neighbor machine translation (kNN-MT) has attracted increasing attention due to its ability to non-parametrically adapt to new translation domains.By using an upstream NMT model to traverse the downstream training corpus, it is equipped with a datastore containing vectorized key-value pairs, which are retrieved during inference to benefit translation.However, there often exists a significant gap between upstream and downstream domains, which hurts the retrieval accuracy and the final translation quality.To deal with this issue, we propose a novel approach to boost the datastore retrieval of kNN-MT by reconstructing the original datastore.Concretely, we design a reviser to revise the key representations, making them better fit for the downstream domain.The reviser is trained using the collected semanticallyrelated key-queries pairs, and optimized by two proposed losses: one is the key-queries semantic distance ensuring each revised key representation is semantically related to its corresponding queries, and the other is an L2-norm loss encouraging revised key representations to effectively retain the knowledge learned by the upstream NMT model.Extensive experiments on domain adaptation tasks demonstrate that our method can effectively boost the datastore retrieval and translation quality of kNN-MT.
Multilingual Conceptual Coverage in Text-to-Image Models

StableDiffusion 2 DallE mega 1 Figure 1: A selection of images generated by DALLE-mega, Stable Diffusion 2, DALLE-2, and AltDiffusion,illustrating their conceptual coverage of "dog," "airplane," and "face" across English, Spanish, German, Chinese (simplified), Japanese, Hebrew, and Indonesian.Coverage of the concepts varies considerably across model and language, and can be observed in the consistency and correctness of images generated under simple prompts.
G3R: A Graph-Guided Generate-and-Rerank Framework for Complex and Cross-domain Text-to-: A Graph-Guided Generate-and-Rerank Framework for Complex and Cross-domain Text-to-SQL Generation

We present a framework called G 3 R for complex and cross-domain Text-to-SQL generation.G 3 R aims to address two limitations of current approaches: (1) The structure of the abstract syntax tree (AST) is not fully explored during the decoding process which is crucial for complex SQL generation; (2) Domain knowledge is not incorporated to enhance their ability to generalise to unseen domains.G 3 R consists of a graph-guided SQL generator and a knowledge-enhanced re-ranking mechanism.Firstly, during the decoding process, an AST-Grammar bipartite graph is constructed for joint modelling the AST and corresponding grammar rules of the generated partial SQL query.The graph-guided SQL generator captures its structural information and fuses heterogeneous information to predict the action sequence, which can uniquely construct the AST for the corresponding SQL query.Then, in the inference stage, a knowledge-enhanced re-ranking mechanism is proposed to introduce domain knowledge to re-rank candidate SQL queries from the beam output and choose the final answer.The SQL re-ranker is based on a pre-trained language model (PLM) and contrastive learning with hybrid prompt tuning is incorporated to stimulate the knowledge of the PLM and make it more discriminative.The proposed approach achieves state-of-theart results on the Spider and Spider-DK benchmarks, which are challenging complex and cross-domain benchmarks for Text-to-SQL semantic analysis.
Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning

Subhajit Chaudhury, Sarathkrishna Swaminathan, Daiki Kimura, Prithviraj Sen, Keerthiram Murugesan, Rosario Uceda-Sosa, Michiaki Tatsubori, Achille Fokoue, Pavan Kapanipathi, Asim Munawar, Alexander Gray. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

Zhiwei Jia, Pradyumna Narayana, Arjun Akula, Garima Pruthi, Hao Su, Sugato Basu, Varun Jampani. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track). 2023.
Gallagher at SemEval-2023 Task 5: Tackling Clickbait with Seq2Seq Models

This paper presents the systems and approaches of the Gallagher team for the SemEval-2023 Task 5: Clickbait Spoiling.We propose a method to classify the type of spoiler (phrase, passage, multi) and a question-answering method to generate spoilers that satisfy the curiosity caused by clickbait posts.We experiment with the state-of-the-art Seq2Seq model T5.To identify the spoiler types we used a finetuned T5 classifier (Subtask 1).A mixture of T5 and Flan-T5 was used to generate the spoilers for clickbait posts (Subtask 2).Our system officially ranks first in generating phrase type spoilers in Subtask 2, and achieves the highest precision score for passage type spoilers in Subtask 1.
Uncertainty-Aware Unlikelihood Learning Improves Generative Aspect Sentiment Quad Prediction

Recently, aspect sentiment quad prediction has received widespread attention in the field of aspect-based sentiment analysis.Existing studies extract quadruplets via pre-trained generative language models to paraphrase the original sentence into a templated target sequence.However, previous works only focus on what to generate but ignore what not to generate.We argue that considering the negative samples also leads to potential benefits.In this work, we propose a template-agnostic method to control the token-level generation, which boosts original learning and reduces mistakes simultaneously.Specifically, we introduce Monte Carlo dropout to understand the built-in uncertainty of pre-trained language models, acquiring the noises and errors.We further propose marginalized unlikelihood learning to suppress the uncertainty-aware mistake tokens.Finally, we introduce minimization entropy to balance the effects of marginalized unlikelihood learning.Extensive experiments on four public datasets demonstrate the effectiveness of our approach on various generation templates 1 .
Delving into the Openness of CLIP

Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descriptions instead of discrete category IDs.This allows for open-vocabulary visual recognition, where the model can recognize images from an open class set (also known as an open vocabulary) in a zero-shot manner.However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice.To address this, we resort to an incremental perspective to assess the openness through vocabulary expansions, and define extensibility to measure a model's ability to handle novel classes.Our evaluation shows that CLIP-like models are not truly open, and their performance deteriorates as the vocabulary expands.We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity.Our investigation reveals that the overestimation of openness is due to confusion among competing text features, rather than a failure to capture the similarity between image features and text features of novel classes.We hope that our investigation and analysis will facilitate future research on the CLIP openness issue. 1 CLIP (RN101) 55.4 REPE (ViT-B/32) 70.9 CoOp (ViT-B/16) 76.9 DeCLIP (ViT-B/32) 70.8 SLIP (ViT-B/16) 51.1 PE (ViT-B/32) 70.
Dialect-robust Evaluation of Generated Text

Text generation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects.In this paper, we introduce a suite of methods to assess whether metrics are dialect robust.These methods show that state-of-the-art metrics are not dialect robust: they often prioritize dialect similarity over semantics, preferring outputs that are semantically incorrect over outputs that match the semantics of the reference but contain dialect differences.As a step towards dialect-robust metrics for text generation, we propose NANO, which introduces regional and language information to the metric's pretraining.NANO significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings.It also enables a more ambitious approach to evaluation, dialect awareness, in which system outputs are scored by both semantic match to the reference and appropriateness in any specified dialect.1 https://ewave-atlas.org/parameters/ 62#2/7.0/7.92 For simplicity we do not include the reference in this definition.A corpus-level reference-based metric could be defined as 1 N i mi(yi) with mi(yi) = δ(yi, ri), with ri indicating the reference for example i and δ : Y × Y → R.
NCUEE-NLP at BioLaySumm Task 2: Readability-Controlled Summarization of Biomedical Articles Using the PRIMERA Models

This study describes the model design of the NCUEE-NLP system for BioLaySumm Task 2 at the BioNLP 2023 workshop.We separately fine-tune pretrained PRIMERA models to independently generate technical abstracts and lay summaries of biomedical articles.A total of seven evaluation metrics across three criteria were used to compare system performance.Our best submission was ranked first for relevance, second for readability, and fourth for factuality, tying first for overall performance.
Jointprop: Joint Semi-supervised Learning for Entity and Relation Extraction with Heterogeneous Graph-based Propagation

Semi-supervised learning has been an important approach to address challenges in extracting entities and relations from limited data.However, current semi-supervised works handle the two tasks (i.e., Named Entity Recognition and Relation Extraction) separately and ignore the cross-correlation of entity and relation instances as well as the existence of similar instances across unlabeled data.To alleviate the issues, we propose Jointprop, a Heterogeneous Graph-based Propagation framework for joint semi-supervised entity and relation extraction, which captures the global structure information between individual tasks and exploits interactions within unlabeled data.Specifically, we construct a unified span-based heterogeneous graph from entity and relation candidates and propagate class labels based on confidence scores.We then employ a propagation learning scheme to leverage the affinities between labelled and unlabeled samples.Experiments on benchmark datasets show that our framework outperforms the state-of-the-art semi-supervised approaches on NER and RE tasks.We show that the joint semi-supervised learning of the two tasks benefits from their codependency and validates the importance of utilizing the shared information between unlabeled data.
ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER

Complex Named Entity Recognition (NER) is the task of detecting linguistically complex named entities in low-context text.In this paper, we present ACLM (Attention-map aware keyword selection for Conditional Language Model fine-tuning), a novel data augmentation approach, based on conditional generation, to address the data scarcity problem in lowresource complex NER.ACLM alleviates the context-entity mismatch issue, a problem existing NER data augmentation techniques suffer from and often generates incoherent augmentations by placing complex named entities in the wrong context.ACLM builds on BART and is optimized on a novel text reconstruction or denoising task -we use selective masking (aided by attention maps) to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entities.Compared with other data augmentation strategies, ACLM can generate more diverse and coherent augmentations preserving the true word sense of complex entities in the sentence.We demonstrate the effectiveness of ACLM both qualitatively and quantitatively on monolingual, crosslingual, and multilingual complex NER across various low-resource settings.ACLM outperforms all our neural baselines by a significant margin (1%-36%).In addition, we demonstrate the application of ACLM to other domains that suffer from data scarcity (e.g., biomedical).In practice, ACLM generates more effective and factual augmentations for these domains than prior methods.1 Keyword Selection he advanced, attacked the enemy 's infantry with the lance, and then retired while the enemy swarmed out of hidden ground where royal artillery GRP guns could attack them.
Extracting Multi-valued Relations from Language Models

The widespread usage of latent language representations via pre-trained language models (LMs) suggests that they are a promising source of structured knowledge.However, existing methods focus only on a single object per subject-relation pair, even though often multiple objects are correct.To overcome this limitation, we analyze these representations for their potential to yield materialized multi-object relational knowledge.We formulate the problem as a rank-then-select task.For ranking candidate objects, we evaluate existing prompting techniques and propose new ones incorporating domain knowledge.Among the selection methods, we find that choosing objects with a likelihood above a learned relation-specific threshold gives a 49.5% F1 score.Our results highlight the difficulty of employing LMs for the multi-valued slot-filling task, and pave the way for further research on extracting relational knowledge from latent language representations.