Reviewerly/exHarmony huggingface dataset:
HAG: Hierarchical Attention with Graph Network for Dialogue Act Classification in Conversation

The prediction of dialogue acts (DA) labels on utterance-level in conversations can be treated as a sequence labeling problem, which requires context- and speaker-aware semantic comprehension, especially for Japanese. In this study, we pro-posed a hierarchical attention with the graph neural network (HAG) to consider the contextual interconnections as well as the semantics carried by the sentence itself. Concretely, the model use long-short term memory networks (LSTMs) to perform a context-aware encoding within a dialogue window. Then, we construct the context graph by aggregating the neighboring utterances. Subsequently, a speaker feature transformation is executed with a graph attention network (GAT) to calculate the interconnections, while a context-level feature selection is performed with a gated graph convolutional network (GatedGCN) to select the salient utterances that contribute to the DA classification. Finally, we merge the representations of different levels and conduct a classification with two dense layers. We evaluate the proposed model on Japanese dialogue act dataset (JPS-DA). The experimental results show that our method outperforms the baselines.
Image Inpainting with Semantic-Aware Transformer

Image inpainting has made huge strides benefiting from the advantages of convolutional neural networks (CNNs) in understanding high-level semantics. Recently, some studies have applied transformers to the visual field to solve the problem that the convolution kernel cannot attend to longdistance information. However, unlike other vision tasks, there is much interference from damaged information in image inpainting tasks. We propose a new Semantic-Aware Transformer, which in addition to including a self-attention block like previous vision transformers, also has a block for learning semantics from QSVM. Specifically, to provide more valid information, we design a Quantized Semantic Vector Memory (QSVM) that encodes and saves semantic features in images as quantized vectors in latent space. Experiments on different datasets demonstrate the effectiveness and superiority of our method compared with the existing state-of-the-art.
Visual Graph Reasoning Network

Visual question answering (VQA) is a fundamental and challenging cross-modal task. This task requires the model to fully understand the image’s content and reason out the answer based on the question. Existing VQA models understand visual content mainly based on bottom-up or grid features. However, both types of vision features have some drawbacks. The discreteness and independence of bottom-up features pre-vent models from adequately performing relational reasoning. Image segmentation by grid features leads to the fragmentation of meaningful visual regions, limiting the cross-modal alignment capability of the model. Therefore, we proposed a more flexible method called Visual Graph. It can connect different patches according to semantic similarity and spatial relevance to model the potential relationships and cluster the adjacent homologous patches. Based on the Visual Graph, we designed a Visual Graph Reasoning Network for VQA. We evaluated our model on GQA and VQA-v2. The experimental results show that our models can achieve excellent performance between single models.
Spatial Similarity Guidance for Few-Shot Segmentation

In this work, we address the challenging issue of few-shot segmentation. Existing methods mainly explore the target object through the semantic similarity between the query and support pixels. However, the semantic similarity often fails to deal well with the target objects with large variations in appearance and the error predictions along the boundary. To this end, we propose a novel spatial similarity guidance network (S2GNet), which adaptively integrates spatial information with semantic information for building a target-aware correlation region to enhance the target object localization. To promote the overall spatial position understanding of the target object, we exploit boundaries as crucial guidance for spatial information. Thus we jointly train a boundary detection task and a segmentation task in an end-to-end way. With that, a target-aware attention module is further proposed to capture the target correlation region by combining the spatial similarity with the semantic similarity for each pair of pixels in the query image, which refines the location of the target object effectively and improves the segmentation performance. Extensive experiments on both PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> datasets show that our approach can achieve state-of-the-art performances.
Difficulty-Aware Data Augmentor for Scene Text Recognition

Deep neural network (DNN) based scene text recognition (STR) methods usually require a large amount of annotated data for training, which is time-consuming and cost-expensive in practice. To address this issue, many data augmentation methods have been developed to train recognizers by improving the diversity of training samples. However, most existing methods neglect the difficulty inherent in samples, and easily suffer from the problem of over-diversity, i.e., the distribution of the augmented data significantly deviates from that of clean data. In this paper, we propose a novel difficulty-aware data augmentation framework for scene text recognition, which jointly considers the difficulty of samples and the strength of augmentations. Specifically, our framework first predicts the sample difficulty, followed by an adaptive data augmentation strategy. Furthermore, we build a more diverse set of augmentation methods for STR and integrate it into our augmentation framework. Extensive experiments on scene text recognition benchmarks show that our augmentation framework significantly improves the performance of recognizers.
Diagonal State Space Augmented Transformers for Speech Recognition

We improve on the popular conformer architecture by replacing the depthwise temporal convolutions with diagonal state space (DSS) models. DSS is a recently introduced variant of linear RNNs obtained by discretizing a linear dynamical system with a diagonal state transition matrix. DSS layers project the input sequence onto a space of orthogonal polynomials where the choice of basis functions, metric and support is controlled by the eigenvalues of the transition matrix. We compare neural transducers with either conformer or our proposed DSS-augmented transformer (DSSformer) encoders on three public corpora: Switchboard English conversational telephone speech 300 hours, Switchboard+Fisher 2000 hours, and a spoken archive of holocaust survivor testimonials called MALACH 176 hours. On Switchboard 300/2000 hours, we reach a single model performance of 8.9%/6.7% WER on the combined test set of the Hub5 2000 evaluation, respectively, and on MALACH we improve the WER by 7% relative over the previous best published result. In addition, we present empirical evidence suggesting that DSS layers learn damped Fourier basis functions where the attenuation coefficients are layer specific whereas the frequency coefficients converge to almost identical linearly-spaced values across all layers.
Absolute Decision Corrupts Absolutely: Conservative Online Speaker Diarisation

Our focus lies in developing an online speaker diarisation framework which demonstrates robust performance across diverse domains. In online speaker diarisation, outputs generated in real-time are irreversible, and a few misjudgements in the early phase of an input session can lead to catastrophic results. We hypothesise that cautiously increasing the number of estimated speakers is of paramount importance among many other factors. Thus, our proposed framework includes decreasing the number of speakers by one when the system judges that an increase in the past was faulty. We also adopt dual buffers, checkpoints and centroids, where checkpoints are combined with silhouette coefficients to estimate the number of speakers and centroids represent speakers. Again, we believe that more than one centroid can be generated from one speaker. Thus we design a clustering-based label matching technique to assign labels in realtime. The resulting system is lightweight yet surprisingly effective. The system demonstrates state-of-the-art performance on DIHARD II and III datasets, where it is also competitive in AMI and VoxConverse test sets.
A Study on the Invariance in Security Whatever the Dimension of Images for the Steganalysis by Deep-Learning

In this paper, we study the performance invariance of convolutional neural networks when confronted with variable image sizes in the context of a more "wild steganalysis". First, we propose two algorithms and definitions for a fine experimental protocol with datasets owning "similar difficulty" and "similar security". The "smart crop 2" algorithm allows the introduction of the Nearly Nested Image Datasets (NNID) that ensure "a similar difficulty" between various datasets, and a dichotomous research algorithm allows a "similar security". Second, we show that invariance does not exist in state-of-the-art architectures. We also exhibit a difference in behavior depending on whether we test on images larger or smaller than the training images. Finally, based on the experiments, we propose to use the dilated convolution which leads to an improvement of a state-of-the-art architecture.
Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames

Recently, the unified streaming and non-streaming two-pass (U2/U2++) end-to-end model for speech recognition has shown great performance in terms of streaming capability, accuracy and latency. In this paper, we present fast-U2++, an enhanced version of U2++ to further reduce partial latency. The core idea of fast-U2++ is to output partial results of the bottom layers in its encoder with a small chunk, while using a large chunk in the top layers of its encoder to compensate the performance degradation caused by the small chunk. More-over, we use knowledge distillation method to reduce the token emission latency. We present extensive experiments on Aishell-1 dataset. Experiments and ablation studies show that compared to U2++, fast-U2++ reduces model latency from 320ms to 80ms, and achieves a character error rate (CER) of 5.06% with a streaming setup.
TABLEIE: Capturing the Interactions Among Sub-Tasks in Information Extraction via Double Tables

Information Extraction mainly consists of three sub-tasks, Named Entity Recognition, Relation Extraction and Event Extraction. Although these sub-tasks are highly correlated with each other, most previous works simply focus on part of them and ignore the interactions among different sub-tasks. Recently, some graph-based models are proposed to cover all the interactions among different IE sub-tasks. However, the use of Graph Neural Network brings heavy computation burden, damaging the model efficiency. In this paper, we propose a double-table framework, TableIE, to capture the interactions among IE sub-tasks as well as improve the model efficiency. Specifically, TableIE has an entity-relation table and an event table, based on which we propose both within-table and cross-table interaction through a novel table integration technique. Such technique makes use of an information-aware mask to extract more essential information in the table during the integration, which we call discriminative interaction. Our extensive experiments demonstrate that TableIE outperforms the previous state-of-the-art up to 1.4 on the ACE05 dataset. Besides, since TableIE does not involve the time-consuming graph operation, it is also more efficient than the previous graph-based models, with 13x speed-up in the inference stage. Our code is available at https://github.com/PKUnlp-icler/TableIE
Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Models

Self-supervised learning (SSL) has been dramatically successful not only in monolingual but also in cross-lingual settings. However, since the two settings have been studied individually in general, there has been little research focusing on how effective a cross-lingual model is in comparison with a monolingual model. In this paper, we investigate this fundamental question empirically with Japanese automatic speech recognition (ASR) tasks. First, we begin by comparing the ASR performance of cross-lingual and monolingual models for two different language tasks while keeping the acoustic domain as identical as possible. Then, we examine how much unlabeled data collected in Japanese is needed to achieve performance comparable to a cross-lingual model pre-trained with tens of thousands of hours of English and/or multilingual data. Finally, we extensively investigate the effectiveness of SSL in Japanese and demonstrate state-of-the-art performance on multiple ASR tasks. Since there is no comprehensive SSL study for Japanese, we hope this study will guide Japanese SSL research.
CC-PoseNet: Towards Human Pose Estimation in Crowded Classrooms

Human pose estimation has long been motivated for its application in human behavior understanding and activity recognition. Despite recent advances in multi-person pose estimation, existing solutions remain challenging in crowded scenes, especially in classroom scenarios where students are extremely overlapped and have different poses. In this paper, we focus on improving human pose estimation in crowded classrooms from the perspective of crowd detection and pose refinement. Specifically, we first follow a top-down strategy to detect persons in a multi-instance prediction manner and perform single-person pose estimation on each detected human region. Then, the pose estimation is refined with Transformer blocks by capturing the interactions among multiple persons in the image. Importantly, we replace self-attention in Transformer with a lightweight attention mechanism to reduce computational complexity. Quantitative and qualitative experiments demonstrate that our method remarkably outperforms previous methods with a clear margin on both standard benchmarks and self-collected classroom images.
Rain2Avoid: Self-Supervised Single Image Deraining

The single image deraining task aims to remove rain from a single image, attracting much attention in the field. Recent research on this topic primarily focuses on discriminative deep learning methods, which train models on rainy images with their clean counterparts. However, collecting such paired images for training takes much work. Thus, we present Rain2Avoid (R2A), a training scheme that requires only rainy images for image deraining. We propose a locally dominant gradient prior to reveal possible rain streaks and overlook those rain pixels while training with the input rainy image directly. Understandably, R2A may not perform as well as deraining methods that supervise their models with rain-free ground truth. However, R2A favors when training image pairs are unavailable and can self-supervise only one rainy image for deraining. Experimental results show that the proposed method performs favorably against state-of-the-art few-shot deraining and self-supervised denoising methods.
Graphmad: Graph Mixup for Data Augmentation Using Data-Driven Convex Clustering

We develop a novel data-driven nonlinear mixup mechanism for graph data augmentation and present different mixup functions for sample pairs and their labels. Mixup is a data augmentation method to create new training data by linearly interpolating between pairs of data samples and their labels. Mixup of graph data is challenging since the interpolation between graphs of potentially different sizes is an ill-posed operation. Hence, a promising approach for graph mixup is to first project the graphs onto a common latent feature space and then explore linear and nonlinear mixup strategies in this latent space. In this context, we propose to (i) project graphs onto the latent space of continuous random graph models known as graphons, (ii) leverage convex clustering in this latent space to generate nonlinear data-driven mixup functions, and (iii) investigate the use of different mixup functions for labels and data samples. We evaluate our graph data augmentation performance on benchmark datasets and demonstrate that nonlinear data-driven mixup functions can significantly improve graph classification.
Multi-Task Bias-Variance Trade-Off Through Functional Constraints

Multi-task learning aims to acquire a set of functions, either regressors or classifiers, that perform well for diverse tasks. At its core, the idea behind multi-task learning is to exploit the intrinsic similarity across data sources to aid in the learning process for each individual domain. In this paper we draw intuition from the two extreme learning scenarios – a single function for all tasks, and a task-specific function that ignores the other tasks dependencies – to propose a bias-variance trade-off. To control the relationship between the variance (given by the number of i.i.d. samples), and the bias (coming from data from other task), we introduce a constrained learning formulation that enforces domain specific solutions to be close to a central function. This problem is solved in the dual domain, for which we propose a stochastic primal-dual algorithm. Experimental results for a multi-domain classification problem with real data show that the proposed procedure outperforms both the task specific, as well as the single classifiers.
CAENet: Using Collaborative Attention Transformer and Add-Boost Strategy for Single Image Deraining

In recent years, the Convolutional Neural Network (CNN) based deraining methods have achieved remarkable results. However, these methods rarely used the long-range context information, and thus could not effectively restore the regions damaged by dense rain streaks. Moreover, the rain streaks in an images are usually complex and diverse while few methods fully explore the richness of the information which may improve the network’s feature representation ability. To solve the above issues, we propose a novel Collaborative Attention Enhanced Network (CAENet) for single image deraining. We first design a Residual Collaborative Attention Transformer (RCAT), consisting of several Collaborative Attention Transformer Blocks (CATBs), to effectively build the long-range dependency relations. The CATB employs the self-attention mechanism to recover the contextual information of the derained images and embeds the outline feature with global attention. Further, we develop an Add-Boost Module (ABM) by aggregating the features in different resolutions, with which more details covered by the rain streaks can be effectively restored. Experiments on synthetic and real-world datasets show that our method achieves excellent rain removal performance and outperforms seven state-of-the-art methods in terms of both quantitative evaluation metrics and qualitative visualization effects.
Spteae: A Soft Prompt Transfer Model for Zero-Shot Cross-Lingual Event Argument Extraction

In zero-shot cross-lingual event argument extraction(EAE) task, a model is typically trained on source language datasets and then applied on task language datasets. There is a trend to regard the zero-shot cross-lingual EAE task as a sequence generation task with manual prompts or discrete prompts. However, there are some problems with these prompts, including using suboptimal prompts and difficult to transfer from source language to target language. To overcome these issues, we propose a method called SPTEAE(A Soft Prompt Transfer model for zero-shot cross-lingual Event Argument Extraction). SPTEAE utilizes a sequence of tunable vectors which are tuned in source language as event type prompts. These source language event type prompts can be transferred as target prompts to perform target EAE task by key-value selection mechanism. For each event type, SPTEAE learns a special target prompt by attending to highly relevant source prompts. Experiment results show that the average performance of SPTEAE with soft prompt transfer is 2.6% higher than the current state-of-the-art model on the ACE2005 dataset.
Towards Polymorphic Adversarial Examples Generation for Short Text

NLP models are shown to be vulnerable to adversarial examples. The usual attack methods in NLP fields mainly focus on word-level perturbations. However, the word-substitution based method is not suitable for short text. Short texts are more susceptible to word substitution than long texts, which makes semantic shifting more likely to occur, and the number of words in short texts can be modified is small, making the attack difficult to succeed and hard to guarantee naturality and fluency. To tackle the above problems, we present Polymorphic Adversarial Examples Generation (PAEG) attack, a generative method by combining pre-trained language model BERT and Variational Autoencoder. Compared to attack methods proposed in previous literature, the proposed approach can not only generate polymorphic adversarial examples with different forms but also improve the attack success rate significantly on two popular datasets. Our codes are released at https://github.com/YilingLiang/vMF-VAE-Bert.
Heuristic Masking for Text Representation Pretraining

Masked language model pretraining provides a standardized way to learn contextualized semantic representations, which reconstructs corrupted text sequences by estimating the conditional probabilities of randomly masked tokens given the context. We attempt to exploit language knowledge from the model itself to boost its pretraining in a lightweight and on-the-fly fashion. In this paper, a heuristic token masking scheme is studied, in which those tokens that deep networks and shallow networks have inconsistent predictions for are more likely to be masked. The proposed method can be applied to BERT-like architectures, and its training approach is consistent with BERT, which guarantees training effects and efficiency. Extensive experiments show that the masked language model pretrained with the heuristic masking scheme consistently outperforms previous schemes in various downstream tasks.
EMCLR: Expectation Maximization Contrastive Learning Representations

One of the bottlenecks of self-supervised contrastive learning is the degenerate constant solution, where all the samples are mapped to one single point in representation space. To prevent such collapses, the mainstream paradigm is using negative samples, forcing negative pairs to push away. However, such manner results in ${\mathcal{O}}\left({{N^2}}\right)$ time and space complexities, limiting the expansibility, scalability and efficiency. We observe current negative-requiring objectives can be decomposed to alignment and uniformity, where uniformity dominates the ${\mathcal{O}}\left({{N^2}}\right)$ complexity. To reduce the complexity, inspired by the traditional EM algorithm, we derive the embedding matrix of each batch with optimally uniform distribution and discard the uniformity part in objectives. Specifically, for stacked embedding matrices of two views, we first calculate the optimal solution of one view by the proposed algorithm. Then we align the embedding matrix with the obtained optimal solution. The learning paradigm ingeniously avoids model collapses without ad-hoc negative pairs and reduces the square complexity to linear. Extensive experiments on CIFAR-10/100 and STL-10 show the proposed methods achieve comparable results in ${\mathcal{O}}\left(N\right)$ complexity.