Reviewerly/exHarmony huggingface dataset:
What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders

The last years have witnessed the emergence of a promising self-supervised learning strategy, referred to as masked autoencoding. However, there is a lack of theoretical understanding of how masking matters on graph autoencoders (GAEs). In this work, we present masked graph autoencoder (MaskGAE), a self-supervised learning framework for graph-structured data. Different from standard GAEs, MaskGAE adopts masked graph modeling (MGM) as a principled pretext task - masking a portion of edges and attempting to reconstruct the missing part with partially visible, unmasked graph structure. To understand whether MGM can help GAEs learn better representations, we provide both theoretical and empirical evidence to comprehensively justify the benefits of this pretext task. Theoretically, we establish close connections between GAEs and contrastive learning, showing that MGM significantly improves the self-supervised learning scheme of GAEs. Empirically, we conduct extensive experiments on a variety of graph benchmarks, demonstrating the superiority of MaskGAE over several state-of-the-arts on both link prediction and node classification tasks.
Doctor Specific Tag Recommendation for Online Medical Record Management

With the rapid growth of online medical platforms, more and more doctors are willing to manage and communicate with patients via online services. Considering the large volume and various patient conditions, identifying and classifying patients' medical records has become a crucial problem. To efficiently index these records, a common practice is to annotate them with semantically meaningful tags. However, manual labeling tags by doctors is impractical due to the possibility of thousands of tag candidates, which necessitates a tag recommender system. Due to the long tail distribution of tags and the dominance of low-activity doctors, as well as the unique uploaded medical records, this task is rather challenging. This paper proposes an efficient doctor specific tag recommendation framework for improved medical record management without side information. Specifically, we first utilize effective language models to learn the text representation. Then, we construct a doctor embedding learning module to enhance the recommendation quality by integrating implicit information within text representations and considering latent tag correlations to make more accurate predictions. Extensive experiment results demonstrate the effectiveness of our framework from the viewpoints of all doctors (20% improvement) or low-activity doctors (10% improvement).
Commonsense Knowledge Graph towards Super APP and Its Applications in Alipay

The recently explosive growth of Super Apps brings great convenience to people's daily life by providing a wide variety of services through mini-programs, including online shopping, travel, finance, and so on. Due to the considerable gap between various scenarios, the restriction of effective information transfer and sharing severely blocks the efficient delivery of online services, potentially affecting the user's app experience. To deeply understand users' needs, we propose SupKG, a commonsense knowledge graph towards Super APP to help comprehensively characterize user behaviors across different business scenarios. In particular, our SupKG is carefully established from multiplex and heterogeneous data source in Alipay (a well-known Super App in China), which also emphasize abundant spatiotemporal relations and intent-related entities to answer the fundamental question in life service ''which service do users need at what time and where''.
Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark

Large language models (LLMs) have demonstrated powerful capabilities in both text understanding and generation.Companies have begun to offer Embedding as a Service (EaaS) based on these LLMs, which can benefit various natural language processing (NLP) tasks for customers.However, previous studies have shown that EaaS is vulnerable to model extraction attacks, which can cause significant losses for the owners of LLMs, as training these models is extremely expensive.To protect the copyright of LLMs for EaaS, we propose an Embedding Watermark method called EmbMarker that implants backdoors on embeddings.Our method selects a group of moderate-frequency words from a general text corpus to form a trigger set, then selects a target embedding as the watermark, and inserts it into the embeddings of texts containing trigger words as the backdoor.The weight of insertion is proportional to the number of trigger words included in the text.This allows the watermark backdoor to be effectively transferred to EaaS-stealer's model for copyright verification while minimizing the adverse impact on the original embeddings' utility.Our extensive experiments on various datasets show that our method can effectively protect the copyright of EaaS models without compromising service quality.Our code is available at https://github.com/yjw1029/EmbMarker.
ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning

This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs.One significant challenge of this task is how to correctly maintain the consistency between the images and the lengthy report.Previous research explored solving this issue through planningbased methods, which generate reports only based on high-level plans.However, these plans usually only contain the major observations from the radiographs (e.g., lung opacity), lacking much necessary information, such as the observation characteristics and preliminary clinical diagnoses.To address this problem, the system should also take the image information into account together with the textual plan and perform stronger reasoning during the generation process.In this paper, we propose an Observation-guided radiology Report GenerAtioN framework (ORGAN).It first produces an observation plan and then feeds both the plan and radiographs for report generation, where an observation graph and a tree reasoning mechanism are adopted to precisely enrich the plan information by capturing the multiformats of each observation.Experimental results demonstrate that our framework outperforms previous state-of-the-art methods regarding text quality and clinical efficacy.
Dynamic Heterogeneous-Graph Reasoning with Language Models and Knowledge Representation Learning for Commonsense Question Answering

Recently, knowledge graphs (KGs) have won noteworthy success in commonsense question answering.Existing methods retrieve relevant subgraphs in the KGs through key entities and reason about the answer with language models (LMs) and graph neural networks.However, they ignore (i) optimizing the knowledge representation and structure of subgraphs and (ii) deeply fusing heterogeneous QA context with subgraphs.In this paper, we propose a dynamic heterogeneous-graph reasoning method with LMs and knowledge representation learning (DHLK), which constructs a heterogeneous knowledge graph (HKG) based on multiple knowledge sources and optimizes the structure and knowledge representation of the HKG using a two-stage pruning strategy and knowledge representation learning (KRL).It then performs joint reasoning by LMs and Relation Mask Self-Attention (RMSA).Specifically, DHLK filters key entities based on the dictionary vocabulary to achieve the first-stage pruning while incorporating the paraphrases in the dictionary into the subgraph to construct the HKG.Then, DHLK encodes and fuses the QA context and HKG using LM, and dynamically removes irrelevant KG entities based on the attention weights of LM for the second-stage pruning.Finally, DHLK introduces KRL to optimize the knowledge representation and perform answer reasoning on the HKG by RMSA.We evaluate DHLK at CommonsenseQA and OpenBookQA, and show its improvement on existing LM and LM+KG methods.
SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created through Human-Machine Collaboration

The potential social harms that large language models pose, such as generating offensive content and reinforcing biases, are steeply rising.Existing works focus on coping with this concern while interacting with ill-intentioned users, such as those who explicitly make hate speech or elicit harmful responses.However, discussions on sensitive issues can become toxic even if the users are well-intentioned.For safer models in such scenarios, we present the Sensitive Questions and Acceptable Response (SQUARE) dataset, a large-scale Korean dataset of 49k sensitive questions with 42k acceptable and 46k non-acceptable responses.The dataset was constructed leveraging HyperCLOVA in a human-in-the-loop manner based on real news headlines.Experiments show that acceptable response generation significantly improves for HyperCLOVA and GPT-3, demonstrating the efficacy of this dataset.
Nonlinear Structural Equation Model Guided Gaussian Mixture Hierarchical Topic Modeling

Hierarchical topic models, which can extract semantically meaningful topics from a text corpus in an unsupervised manner and automatically organise them into a topic hierarchy, have been widely used to discover the underlying semantic structure of documents.However, the existing models often assume in the prior that the topic hierarchy is a tree structure, ignoring symmetrical dependencies between topics at the same level.Moreover, the sparsity of text data often complicate the analysis.To address these issues, we propose NSEM-GMHTM as a deep topic model, with a Gaussian mixture prior distribution to improve the model's ability to adapt to sparse data, which explicitly models hierarchical and symmetric relations between topics through the dependency matrices and nonlinear structural equations.Experiments on widely used datasets show that our NSEM-GMHTM generates more coherent topics and a more rational topic structure when compared to state-of-theart baselines.Our code is available at https: //github.com/nbnbhwyy/NSEM-GMHTM.
Improving Embedding-based Unsupervised Keyphrase Extraction by Incorporating Structural Information

Keyphrase extraction aims to extract a set of keyphrases with the central idea of the document.In a structured document, there are certain locations (e.g., the title or the first sentence) where a keyphrase is most likely to appear.However, when extracting keyphrases from the document, most existing embedding-based unsupervised keyphrase extraction models ignore the indicative role of the highlights in certain locations, leading to wrong keyphrases extraction.In this paper, we propose a new Highlight-Guided Unsupervised Keyphrase Extraction model (HGUKE) to address the above issue.Specifically, HGUKE first models the phrasedocument relevance via the highlights of the documents.Next, HGUKE calculates the crossphrase relevance between all candidate phrases.Finally, HGUKE aggregates the above two relevance as the importance score of each candidate to rank and extract keyphrases.The experimental results on three benchmarks demonstrate that HGUKE outperforms the state-of-the-art unsupervised keyphrase extraction baselines.
CHEER: Centrality-aware High-order Event Reasoning Network for Document-level Event Causality Identification

Document-level Event Causality Identification (DECI) aims to recognize causal relations between events within a document.Recent studies focus on building a document-level graph for cross-sentence reasoning, but ignore important causal structures -there are one or two "central" events that prevail throughout the document, with most other events serving as either their cause or consequence.In this paper, we manually annotate central events for a systematical investigation and propose a novel DECI model, CHEER, which performs highorder reasoning while considering event centrality.First, we summarize a general GNNbased DECI model and provide a unified view for better understanding.Second, we design an Event Interaction Graph (EIG) involving the interactions among events (e.g., coreference) and event pairs, e.g., causal transitivity, cause(A, B) ∧ cause(B, C) ⇒ cause(A, C).Finally, we incorporate event centrality information into the EIG reasoning network via well-designed features and multi-task learning.We have conducted extensive experiments on two benchmark datasets.The results present great improvements (5.9% F1 gains on average) and demonstrate the effectiveness of each main component.
Few-shot Classification with Hypersphere Modeling of Prototypes

Metric-based meta-learning is one of the de facto standards in few-shot learning. It composes of representation learning and metrics calculation designs. Previous works construct class representations in different ways, varying from mean output embedding to covariance and distributions.However, using embeddings in space lacks expressivity and cannot capture class information robustly, while statistical complex modeling poses difficulty to metric designs.In this work, we use tensor fields (“areas”) to model classes from the geometrical perspective for few-shot learning. We present a simple and effective method, dubbed as hypersphere prototypes (HyperProto), where class information is represented by hyperspheres with dynamic sizes with two sets of learnable parameters: the hypersphere’s center and the radius. Extending from points to areas, hyperspheres are much more expressive than embeddings. Moreover, it is more convenient to perform metric-based classification with hypersphere prototypes than statistical modeling, as we only need to calculate the distance from a data point to the surface of the hypersphere. Following this idea, we also develop two variants of prototypes under other measurements. Extensive experiments and analysis on few-shot NLP tasks and comparison with 20+ competitive baselines demonstrate the effectiveness of our approach.
S2ynRE: Two-stage Self-training with Synthetic data for Low-resource Relation Extraction

Current relation extraction methods suffer from the inadequacy of large-scale annotated data.While distant supervision alleviates the problem of data quantities, there still exists domain disparity in data qualities due to its reliance on domain-restrained knowledge bases. In this work, we propose S2ynRE, a framework of two-stage Self-training with Synthetic data for Relation Extraction.We first leverage the capability of large language models to adapt to the target domain and automatically synthesize large quantities of coherent, realistic training data.We then propose an accompanied two-stage self-training algorithm that iteratively and alternately learns from synthetic and golden data together.We conduct comprehensive experiments and detailed ablations on popular relation extraction datasets to demonstrate the effectiveness of the proposed framework.
Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

The performance of automatic speech recognition (ASR) systems has advanced substantially in recent years, particularly for languages for which a large amount of transcribed speech is available.Unfortunately, for low-resource languages, such as minority languages, regional languages or dialects, ASR performance generally remains much lower.In this study, we investigate whether data augmentation techniques could help improve low-resource ASR performance, focusing on four typologically diverse minority languages or language variants (West Germanic: Gronings, West-Frisian; Malayo-Polynesian: Besemah, Nasal).For all four languages, we examine the use of selftraining, where an ASR system trained with the available human-transcribed data is used to generate transcriptions, which are then combined with the original data to train a new ASR system.For Gronings, for which there was a preexisting text-to-speech (TTS) system available, we also examined the use of TTS to generate ASR training data from text-only sources.We find that using a self-training approach consistently yields improved performance (a relative WER reduction up to 20.5% compared to using an ASR system trained on 24 minutes of manually transcribed speech).The performance gain from TTS augmentation for Gronings was even stronger (up to 25.5% relative reduction in WER compared to a system based on 24 minutes of manually transcribed speech).In sum, our results show the benefit of using selftraining or (if possible) TTS-generated data as an efficient solution to overcome the limitations of data availability for resource-scarce languages in order to improve ASR performance.
Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics

Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users.Imagebased NLG systems have progressed to the point where they can begin to address this persistent societal problem, but these systems will not be fully successful unless we evaluate them on metrics that guide their development correctly.Here, we argue against current referenceless metrics -those that don't rely on human-generated ground-truth descriptions -on the grounds that they do not align with the needs of BLV users.The fundamental shortcoming of these metrics is that they do not take context into account, whereas contextual information is highly valued by BLV users.To substantiate these claims, we present a study with BLV participants who rated descriptions along a variety of dimensions.An in-depth analysis reveals that the lack of context-awareness makes current referenceless metrics inadequate for advancing image accessibility.As a proof-of-concept, we provide a contextual version of the referenceless metric CLIPScore which begins to address the disconnect to the BLV data.
Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation

Non-autoregressive translation (NAT) model achieves a much faster inference speed than the autoregressive translation (AT) model because it can simultaneously predict all tokens during inference.However, its translation quality suffers from degradation compared to AT.And existing NAT methods only focus on improving the NAT model's performance but do not fully utilize it.In this paper, we propose a simple but effective method called "Candidate Soups," which can obtain high-quality translations while maintaining the inference speed of NAT models.Unlike previous approaches that pick the individual result and discard the remainders, Candidate Soups (CDS) can fully use the valuable information in the different candidate translations through model uncertainty.Extensive experiments on two benchmarks demonstrate the effectiveness and generality of our proposed method, which can significantly improve the translation quality of various base models.More notably, our best variant outperforms the AT model on three translation tasks with 7.6× speedup. 1
Social-aware Sparse Attention Network for Session-based Social Recommendation

Session-based Social Recommendation (SSR) aims to use users' social networks and historical sessions to provide more personalized recommendations for the current session.Unfortunately, existing SSR methods have two limitations.First, they do not screen users' useless social relationships and noisy irrelevant interactions.However, user preferences are mainly affected by several close friends and key interactions.Second, when modeling the current session, they do not take full advantage of user preference information.To tackle these issues, we propose a novel Social-aware Sparse Attention Network for SSR, abbreviated as SSAN.It mainly consists of the Heterogeneous Graph Embedding (HGE) module and the Social-aware Encoder-decoder Network (SEN) module.In the HGE module, we adopt a modified heterogeneous graph neural network, which focuses more on close friends and key historical interactions, to enhance user/item representations.In the SEN module, we use the user representation as a bridge between the Encoder and Decoder to incorporate user preferences when modeling the current session.Extensive experiments on two benchmark datasets demonstrate the superiority of SSAN over the state-of-the-art models.
Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples

Knowledge-enhanced language representation learning has shown promising results across various knowledge-intensive NLP tasks.However, prior methods are limited in efficient utilization of multilingual knowledge graph (KG) data for language model (LM) pretraining.They often train LMs with KGs in indirect ways, relying on extra entity/relation embeddings to facilitate knowledge injection.In this work, we explore methods to make better use of the multilingual annotation and language agnostic property of KG triples, and present novel knowledge based multilingual language models (KMLMs) trained directly on the knowledge triples.We first generate a large amount of multilingual synthetic sentences using the Wikidata KG triples.Then based on the intra-and inter-sentence structures of the generated data, we design pretraining tasks to enable the LMs to not only memorize the factual knowledge but also learn useful logical patterns.Our pretrained KMLMs demonstrate significant performance improvements on a wide range of knowledge-intensive crosslingual tasks, including named entity recognition (NER), factual knowledge retrieval, relation classification, and a newly designed logical reasoning task. 1
Energy Consumption Evaluation of Post-Quantum TLS 1.3 for Resource-Constrained Embedded Devices

Post-Quantum cryptography (PQC), in the past few years, constitutes the main driving force of the quantum resistance transition for security primitives, protocols and tools. TLS is one of the widely used security protocols that needs to be made quantum safe. However, PQC algorithms integration into TLS introduce various implementation overheads compared to traditional TLS that in battery powered embedded devices with constrained resources, cannot be overlooked. While there exist several works, evaluating the PQ TLS execution time overhead in embedded systems there are only a few that explore the PQ TLS energy consumption cost. In this paper, a thorough power/energy consumption evaluation and analysis of PQ TLS 1.3 on embedded systems has been made. A WolfSSL PQ TLS 1.3 custom implementation is used that integrates all the NIST PQC algorithms selected for standardisation as well as 2 out of 3 of those evaluated in NIST Round 4. Also 1 out of 2 of the BSI recommendations have been included. The PQ TLS 1.3 with the various PQC algorithms is deployed in a STM Nucleo evaluation board under a mutual and a unilateral client-server authentication scenario. The power and energy consumption collected results are analyzed in detail. The performed comparisons and overall analysis provide very interesting results indicating that the choice of the PQC algorithms in TLS 1.3 to be deployed on an embedded system may be very different depending on the device use as an authenticated or not authenticated, client or server. Also, the results indicate that in some cases, PQ TLS 1.3 implementations can be equally or more energy consumption efficient compared to traditional TLS 1.3.