Reviewerly/exHarmony huggingface dataset:
Linguistic Feature Injection for Efficient Natural Language Processing

Transformers have been established as one of the most effective neural approach in performing various Natural Language Processing tasks. However, following common trend in modern deep architectures, their scale has quickly grown to an extent that reduces the concrete possibility for several enterprises to train such models from scratch. Indeed, despite their high-level performances, Transformers have the general drawback of requiring a huge amount of training data, computational resources and energy consumption to be successfully optimized. For this reason, more recent architectures like Bidirectional Encoder Representations from Transformers rely on unlabeled data to pre-train the model, which is later fine-tuned for a specific downstream task using a relatively smaller amount of training data. In a similar fashion, this paper considers a plug-and-play framework that can be used to inject multiple syntactic features, like Part-of-Speech Tagging or Dependency Parsing, into any kind of pre-trained Transformer. This novel approach allows to perform sequence-to-sequence labeling tasks by exploiting: (i) the (more abundant) available training data that is also used to learn the syntactic features, (ii) the language data that is used to pre-train the transformer model. The experimental results show that our approach improves over the baseline performances of the underlying model in different datasets, thus proving the effectiveness of employing syntactic language information for semantic regularization. In addition, we show that our architecture has a huge efficiency advantage over pure large language models. Indeed, by using a model with limited size, but whose input data are enriched with syntactic information, we show that it is possible to obtain a significant reduction of CO2 emissions without decreasing the prediction performances.
Handling Chinese OOV with a Combination of Radical-based Sub-words and Glyph Features

In natural language processing tasks, dictionaries must be built and words must be converted into their corresponding embeddings, creating problems such as out of vocabulary (OOV). To solve this problem, algorithms such as byte pair encoding (BPE) have been widely used to generate sub-words from known words and decompose OOV words into existing sub-words. However, Chinese has many types of characters called Kanji, and their distribution in the text is sparse, reducing the performance of BPE algorithms that rely on the frequency of occurrence. Compared with Kanji characters, radicals are less varied and more densely distributed. Moreover, because Kanji characters are hieroglyphs, their glyphs contain a rich amount of information and are unique. In this paper, we take advantage of these characteristics of Chinese and apply the BPE algorithm after converting the characters to the corresponding radicals, and combining sub-word embedding with the glyph feature vector. Experiments show that converting more Kanji characters to radicals results in a more even distribution of characters in the converted Chinese text, and the number of OOV words continues to decrease. Furthermore, at a 30% conversion ratio, the effect of radical-based embedding can outperform the effect of character-level embedding that keeps all Kanji characters unchanged. The results also indicate that combining with embedding using glyph features can improve performance despite the different conversion ratios.
FLWGAN: Federated Learning with Wasserstein Generative Adversarial Network for Brain Tumor Segmentation

Recently, the potential of deep learning in identifying complex patterns is gaining research interest in medical applications specifically for brain tumor diagnosis. To segment tumors accurately in brain MRIs, there is a need for a large amount of data for training deep learning models. Also, hospitals cannot share patient data for centralization on the server since health records are prone to privacy and ownership challenges. To deal with these challenges, we set up an efficient federated learning (FL) pipeline with Wasserstein generative adversarial networks (FLWGAN) to ensure data privacy and data sufficiency. FL preserves the data privacy of clients by sharing only the trained model parameters to a centralized server instead of raw data. A modified 3D Wasserstein generative adversarial network with gradient penalty (WGAN-GP) and is incorporated at the client side to generate image-segmentation pairs for efficient training segmentation models. Here, 3D-UNet with an attention module is used for the brain MRI segmentation. The attention module is integrated into a 3D-UNet encoder network for effective brain tumor segmentation. Our approach aims to allow each client to benefit from locally available real data and synthetic data. This process enhances the learning performance while respecting data privacy. The efficacy of our proposed pipeline is demonstrated on the brain tumor task of the medical segmentation decathlon (MSD) dataset. We designed FLWGAN frameworks for predicting four segmentation tasks, i.e., whole tumor (WT), enhanced tumor (ET), tumor core (TC), and multiclass. Our proposed approach achieves state of the art performance in terms of various segmentation metrics.
Real-time Blind Deblurring Based on Lightweight Deep-Wiener-Network

In this paper, we address the problem of blind image deblurring with high efficiency. We propose a set of lightweight deep-Wiener-networks to achieve the task with real-time speed. The network contains a deep neural network for estimating the parameters of Wiener networks and a Wiener network for deblurring. Experimental evaluations show that our approaches have an edge on the state-of-the-art in terms of inference speeds and number of parameters. Two of our models can reach a speed of 100 images per second, which is qualified for real-time deblurring. Further research may focus on some real-world applications of deblurring with our models
Neural Time-Invariant Causal Discovery from Time Series Data

Causal structure learning from observational data is an active field of research over the past decades. Although many approaches exist, such as constrained-based methods and score-based methods including the emerging deep learning-based methods, most of them address the static, non-dynamic setting. In this paper, we propose a score-based causal discovery algorithm named Neural Time-invariant Causal Discovery (NTiCD), which learns summary causal graphs from multivariate time series data based on the principle of Granger causality. NTiCD is a continuous optimization-based technique that leverages the power of deep neural networks to compute the score values. To this end, we use an LSTM to obtain the hidden non-linear representations of temporal variables in the time series data. Then, these features are aggregated using graph convolutional networks and decoded using an MLP that outputs the forecast of the future data values in the time series. The model is optimized based on a score function subject to regularized loss. The final output is a summary causal graph that captures the time-invariant causal relations within and between time series. We evaluate the performance of our algorithm on several synthetic and real datasets. The result analysis over a number of different datasets demonstrates the improvement in the accuracy of causal structure discovery of temporal data compared to other state-of-the-art methods.
Anime Character Identification and Tag Prediction by Multimodality Modeling: Dataset and Model

In recent years, some advances have been achieved in classification and object detection related to animation. However, these works do not take full advantage of the tags and text description content attached to the anime data when they are created, which restricts both the related methods and data to unimodality, consequently leading to unsatisfactory performance. In this paper, we propose a novel multimodal deep learning network for Anime character identification and tag prediction by exploiting multimodal data. Considering that in many realistic scenarios, text annotations accompanying anime may be missing, we introduce the concept of curriculum learning in transformers to enable inference with only one modality. Another challenge lies in that the existing dataset does not meet our demand for large-scale multimodal deep learning. To train the proposed network, we construct a new anime dataset Dan: mul that contains over 1.6M images spread across more than 14K categories, with an average of 24 tags per image. To the best of our knowledge, this is the first dataset specifically designed for multimodal anime character identification. With the trained network, we can identify the anime characters in images and generate the related tags. Experiments show that our method achieves state-of-the-art performance on Dan: mul in animation identification.
NSA: Naturalistic Support Artifact to Boost Network Confidence

Visual AI systems are vulnerable to natural and synthetic physical corruption in the real-world. Such corruption often arises unexpectedly and alters the model's performance. In recent years, the primary focus has been on adversarial attacks. However, natural corruptions (e.g., snow, fog, dust) are an omnipresent threat to visual AI systems and should be considered equally important. Many existing works propose interesting solutions to train robust models against natural corruption. These works either leverage image augmentations, which come with the additional cost of model training, or place suspicious patches in the scene to design unadversarial examples. In this work, we propose the idea of naturalistic support artifacts (NSA) for robust prediction. The NSAs are shown to be beneficial in scenarios where model parameters are inaccessible and adding artifacts in the scene is feasible. The NSAs are natural looking objects generated through artifact training using DC-GAN to have high visual fidelity in the scene. We test against natural corruptions on the Imagenette dataset and observe the improvement in prediction confidence score by four times. We also demonstrate NSA's capability to increase adversarial accuracy by 8% on average. Lastly, we qualitatively analyze NSAs using saliency maps to understand how they help improve prediction confidence.
DeepReversion: Reversely Inferring the Original Face from the DeepFake Face

Deepfake techniques can generate realistic fake images and videos. Malicious fake facial images quickly spread through the Internet, posing a potential threat to personal privacy and judicial forensics. However, the defense methods against deepfake proposed so far mainly focus on the discrimination of authenticity, but cannot identify the true source of the forged face, i.e., the original genuine face corresponding to the face-swapped fake face. This paper poses an interesting issue for face deepfake, which is the proactive forensics of “knowing what and knowing how”. In view of the fact that the fake face exhibits high similarity with the original face, especially the facial expression and pose, we argue that the original face can be approximately estimated from the deepfake counterpart. Accordingly, we advocate a deep-learning-based face inversion approach, so-called DeepReversion, which learns the inverse mapping from the deepfake face to the original face. Based on UNet, we design a specific end-to-end DeepReversion network, and conduct comprehensive experiments on public deepfake datasets. The experimental results show that the speculated face is highly consistent with the original face in terms of visual effects, PSNR, SSIM and similarity given by face recognizers.
10X Faster Subgraph Matching: Dual Matching Networks with Interleaved Diffusion Attention

The goal of subgraph matching is to determine the presence of a particular query pattern within a large collection of data graphs. Despite being a hard problem, subgraph matching is essential in various disciplines, including bioinformatics, text matching, and graph retrieval. Although traditional approaches could provide exact solutions, their computations are known to be NP-complete, leading to an overwhelmingly querying latency. While recent neural-based approaches have been shown to improve the response time, the oversimplified assumption of the first-order network may neglect the generalisability of fully capturing patterns in varying sizes, causing the performance to drop significantly in datasets in various domains. To overcome these limitations, this paper proposes xDualSM, a dual matching neural network model with interleaved diffusion attention. Specifically, we first embed the structural information of graphs into different adjacency matrices, which explicitly capture the intra-graph and cross-graph structures between the query pattern and the target graph. Then, we introduce a dual matching network with interleaved diffusion attention to carefully capture intra-graph and cross-graph information while reducing computational complexity. Empirically, our proposed framework not only boosted the speed of subgraph matching more than 10x compared to the fastest baseline but also achieved significant improvements of 47.64% in Recall and 34.39% in F1-score compared to the state-of-the-art approximation approach on COX2 dataset. In addition, our results are comparable with exact methods.
Fair_FM: An Improved Functional Mechanism to Provide Better Privacy Protection and Fairness Guarantee

Machine learning algorithms are currently used in various fields. The success of these algorithms is following the access of large amounts of sensitive personal data, which raises sociological concerns about privacy and fairness. At present, existing works only focus on providing fairness guarantee for the specific attribute that “can affect fairness” without considering the impact of other attributes on that attribute. Therefore, this paper takes the classical logistic regression model as the research object, uses Decision Tree Analysis and Bayesian Network to complete the analysis, and uses functional mechanism to achieve fair privacy protection. For the functional mechanism, we analyze that the Chebyshev polynomial expansion is a less error approach. Theoretical analysis and experimental results show that proposed algorithms can effectively achieve differential privacy and fairness guarantee while maintaining good utility.
Action Reinforcement and Indication Module for Single-frame Temporal Action Localization

Single-frame temporal action localization aims to predict the start time, end time, and categories of action instances in untrimmed videos with only one timestamp label for each action instance. However, recent works have two challenges: incomplete location results caused by various same-class action snippets (or frames) and redundant predictions due to ambiguous background snippets. To tackle the above issues, we design a model consisting of an action reinforcement module (ARM) and an action indication module (AIM). Specifically, the ARM improves the model's generalization ability by augmenting local and global features, thus solving the challenges of incomplete predictions. Simultaneously, the AIM detects the frames that indicate the location of the action instances (called keyframes) in the training period. Then the AIM applies the detected keyframes to filter redundant prediction results in the testing period. Extensive experiments are performed on two benchmark datasets, THUMOS-14 and ActivityNet-v1.3, demonstrating that the model could achieve state-of-the-art performance compared to some competitive approaches.
Evolution Strategies Enhanced Complex Multiagent Coordination

Multi-agent coordination involves both the individual reward and team reward, where the former guides the agent to learn basic skills and the latter measures how well such a team cooperatively completes final tasks. However, in many complex scenarios, these two aspects can be contradictory, due to that one agent excessively pursuing its own profits may suppress the performance of other teammates and lead to the reduction of overall profits. Besides, such dual rewards are generally entangled which make the learning swing between optimizing either the former or latter, which further leads to the sub-optimal and unstable solutions. Moreover, the sparse reward problem commonly encountered in the multi-agent system would further exacerbate this contradiction. In the present work, we address these challenges by proposing CEMARL, a novel framework combining cross-entropy method (CEM) and off-policy multi-agent reinforcement learning (MARL). CEM is gradient-free and learns from the whole episode, whereas MARL is gradient-based and learns from the experiences of agents. The core idea behind CEMARL is that it explicitly decomposes the individual reward and team reward, and deals with them through gradient-based learning and gradient-free evolution, respectively. By means of this, it can simultaneously maximize the individual and team reward, and reconciles the contradiction between individual and team as well as the sparse reward problem. CEMARL shows both conciseness in framework and stability in training, and achieves significantly better performances than state-of-the-art baselines on a range of complex tasks.
Data Augmentation Ensemble Module based on Natural Guidance for X-ray Prohibited Items Detection

Automatic prohibited items detection plays an important role in protecting public security. Till now, object detection powered by deep learning provides a promising solution to automatic security inspection. However, in the one hand, according to the imaging principle of X-ray images, texture information would be lost, and in the other hand, different prohibited items with the same material are easily confused for the similar imaging color, leading to poor detection performance. Thus, to improve the detection performance of the basic object detection models on prohibited items detection task, we first propose the Data Augmentation Ensemble Module (DAEM) based on Nature Guidance for more accurate prohibited items detection. Specifically, inspired by the fact that inspectors detect items based on the characteristics of prohibited items in nature, we introduce natural images as prior knowledge to build X-ray security image - natural image sample pairs for supervising the model training. Besides, we adopt data augmentation strategies to enhance the diversity of the X-ray images, and then we combine the predictions from different data augmentation methods by ensemble learning to yield more accurate results. We verify the DAEM's performance by plug it into three different object detection models, and the experiments demonstrate that our framework can significantly improve the performance compared with the SOTA method on the PIDray dataset.
On a New Type of Neural Computation for Probabilistic Symbolic Reasoning

New types of neural computations, i.e., methods of computing neuron activities based on other neurons' activities and connectivity strengths, are continuously pushing the boundary of more powerful neural networks. For example, the attention mechanism with dynamic weight computation enables the Transformer-family models significantly surpass old models with fixed-weight transformations. We ask whether there exist more powerful types of neural computations that further surpass attention-based models, to reason under uncertainty and manipulate abstract symbols? Using probabilistic programming as the mathematical framework for capturing probabilistic and symbolic reasoning, we develop the first neural computation that allows the execution of probabilistic programs on neural networks. Evaluation on early language acquisition tasks shows the advantage of our method to learn abstract rules from raw data, a capability not possessed by classical neural networks.
Finding Cycles in Graph: A Unified Approach for Various NER Tasks

Named Entity Recognition (NER) is the task of recognizing the entities' locations and types in text, which can be generally categorized into flat NER, overlapped NER, and discontinuous NER. Most previous methods are usually designed specifically for one of the tasks, such as sequence labeling approaches for flat NER and span-based models for overlapped NER. Recently, some new work has begun to propose the unified NER framework that can addresses all three scenarios simultaneously. However, there still has room for improvement in some complex scenarios (long/discontinuous entity). In this paper, we propose a concise framework that supports all types of NER tasks, where entities can be represented by unique cycles that are formed by the directed edges among tokens in the graph. The model integrates a Graph Feature Enhancement module to extract correlations at both the node-level and edge-level. At the node-level, the features are enhanced in the binary and ternary token relations. In edge-level, the model will go further to enhance the relations among token pairs using deformable convolutions. Furthermore, to benefit the completeness of cycle formation, we also propose a novel Cycle Loss that optimizes the independent edge classification in the group of cycles from a global perspective. Experimental results show that our model can achieve competitive and even new state-of-the-art performance on eight popular NER benchmarks, including flat NER, overlapped NER, and discontinuous NER.
Semantic-Aware Mixup for Domain Generalization

Deep neural networks (DNNs) have shown exciting performance in various tasks, yet suffer generalization failures when meeting unknown target domains. One of the most promising approaches to achieve domain generalization (DG) is generating unseen data, e.g., mixup, to cover the unknown target data. However, existing works overlook the challenges induced by the simultaneous appearance of changes in both the semantic and distribution space. Accordingly, such a challenge makes source distributions hard to fit for DNNs. To mitigate the hard-fitting issue, we propose to perform a semantic-aware mixup (SAM) for domain generalization, where whether to perform mixup depends on the semantic and domain information. The feasibility of SAM shares the same spirits with the Fourier-based mixup. Namely, the Fourier phase spectrum is expected to contain semantics information (relating to labels), while the Fourier amplitude retains other information (relating to style information). Built upon the insight, SAM applies different mixup strategies to the Fourier phase spectrum and amplitude information. For instance, SAM merely performs mixup on the amplitude spectrum when both the semantic and domain information changes. Consequently, the overwhelmingly large change can be avoided. We validate the effectiveness of SAM using image classification tasks on several DG benchmarks.
Exploring Machine Learning Privacy/Utility Trade-Off from a Hyperparameters Lens

Machine Learning (ML) architectures have been applied to several applications that involve sensitive data, where a guarantee of users' data privacy is required. Differentially Private Stochastic Gradient Descent (DPSGD) is the state-of-the-art method to train privacy-preserving models. However, DPSGD comes at a considerable accuracy loss leading to sub-optimal privacy/utility trade-offs. Towards investigating new ground for better privacy-utility trade-off, this work questions; (i) if models' hyperparameters have any inherent impact on ML models' privacy-preserving properties, and (ii) if models' hyperparameters have any impact on the privacy/utility trade-off of differentially private models. We propose a comprehensive design space exploration of different hyperparameters such as the choice of activation functions, the learning rate and the use of batch normalization. Interestingly, we found that utility can be improved by using Bounded RELU as activation functions with the same privacy-preserving characteristics. With a drop-in replacement of the activation function, we achieve new state-of-the-art accuracy on MNIST (96.02%), FashionMnist (84.76%), and CIFAR-10 (44.42%) without any modification of the learning procedure fundamentals of DPSGD.