Reviewerly/exHarmony huggingface dataset:
AttResDU-Net: Medical Image Segmentation Using Attention-based Residual Double U-Net

Manually inspecting polyps from a colonoscopy for colorectal cancer or performing a biopsy on skin lesions for skin cancer are time-consuming, laborious, and complex procedures. Automatic medical image segmentation aims to expedite this diagnosis process. However, numerous challenges exist due to significant variations in the appearance and sizes of objects with no distinct boundaries. This paper proposes an attention-based residual Double U-Net architecture (AttResDU-Net) that improves on the existing medical image segmentation networks. Inspired by the Double U - Net, this architecture incorporates attention gates on the skip connections and residual connections in the convolutional blocks. The attention gates allow the model to retain more relevant spatial information by suppressing irrelevant feature representation from the down-sampling path for which the model learns to focus on target regions of varying shapes and sizes. Moreover, the residual connections help to train deeper models by ensuring better gradient flow. We conducted experiments on three datasets: CVC Clinic-DB, ISIC 2018, and the 2018 Data Science Bowl datasets and achieved Dice Coefficient scores of 94.35 %, 91.68 % and 92.45 % respectively. Our results suggest that AttResDU-Net can be facilitated as a reliable method for automatic medical image segmentation in practice.
A Symbolic-Neural Reasoning Model for Visual Question Answering

State-of-the-art Visual Question Answering (VQA) systems have demonstrated promising performance in solving visual relationship-based reasoning problems. However, they struggle in solving complex problems where the answers require sophisticated logical reasoning. In this paper, we introduce a hybrid symbolic-neural reasoning model that integrates deep neural network vision and language features with a symbolic reasoner connected to a knowledge base. The symbolic reasoner effectively combines visual and linguistic information with on-tological relationships and common-sense reasoning to address complex logical questions. We replace the multimodal fusion layer in traditional VQA deep neural networks with an innovative logical reasoning component, generating reasoned answers and clear logical inference chains. Moreover, we propose developing a notion of Question Difficulty, reflecting the logical complexity of VQA questions and their difficulty level in terms of being answered. Current VQA approaches excel at straightforward logic but struggle with increased question difficulty. Our hybrid method performs better as it has access to an additional logical reasoner through the knowledge base to produce answers that require logical inference. Experimental analysis of the answers and the key evidential predicates generated using our unique LoRA (Logical Reasoning Associated VQA) dataset are used to validate our approach and clearly demonstrate its advantages.
Boosting Aspect Sentiment Quad Prediction by Data Augmentation and Self-Training

Aspect Sentiment Quad Prediction (ASQP), which aims to predict the (aspect category, aspect term, opinion term, sentiment polarity) quadruple in a sentence, is an important subtask of Aspect-based Sentiment Analysis (ABSA) and has also gained much research attention. Recently, a number of works adopt generative models to do ASQP and achieve promising performance. Nevertheless, these approaches are limited by data scarcity, due to the challenge of labeling fine-grained quadruple annotations. Even data augmentation techniques do not help much because it is difficult to maintain the quality of augmented samples. Furthermore, the proposed templates of generative models are undesirable in extracting semantic and structural information, thus cannot fully exploit the knowledge of the language models. In this paper, to boost the performance of ASQP, we propose a novel method called DAST (the abbreviation of Data Augmentation and Self-Training). Specifically, DAST first generates large amounts of unlabeled data from the original training data. Then, an iterative self-training mechanism is proposed to fully utilize the unlabeled data to train the ASQP model. In each iteration, we employ a base ASQP model and a discriminator to obtain high-quality samples with pseudo labels, and further enhance the two models with the augmented data. In our method, both the potential of data augmentation and the ASQP model are fully explored thanks to the iterative mechanism. Finally, we advance the ASQP base model via a novel tree-like template that improves the representations of the relations between different sentiment elements, and a double-check mechanism that leverages multi-task learning to detect incorrect predictions. Extensive experiments on two benchmark datasets demonstrate the superiority of our method over the existing ones.
SAGE: Semantic-Aware Global Explanations for Named Entity Recognition

In the last decades, deep learning approaches achieved impressive results in many research fields, such as Computer Vision and Natural Language Processing (NLP). NLP in particular has greatly benefit from unsupervised methods that allow to learn distributed representation of language. On the race for better performances Language Models have reached hundred of billions parameters nowadays. Despite the remarkable results, deep models are still far from being fully exploited in real world applications. Indeed, these approaches are black-boxes, i.e. they are not interpretable by design nor explainable, which is often crucial to make decisions in business. Several task-agnostic methods have been proposed in literature to explain models' decisions. Most techniques rely on the “local” assumption, i.e. explanations are made example-wise. In this paper instead, we present a post-hoc method to produce highly interpretable global rules to explain NLP classifiers. Rules are extracted with a data mining approach on a semantically enriched input representation, instead of using words/wordpieces solely. Semantic information yields more abstract and general rules that are both more explanatory and less complex, while being also better at reflecting the model behaviour. In the experiments we focus on Named Entity Recognition, an NLP task where explainability is under-investigated. We explain the predictions of BERT NER classifiers trained on two popular benchmarks, CoNLL03 and Ontonotes, and compare our model against LIME [1] and Decision Trees.
A Graph-based Context Learning Technique for Image Parsing

The modern deep learning-based architectures have performed well for pixel-wise segmentation tasks. The consideration of context is of vital importance for generation of accurate semantic information. In this research, a deep learning-based image parsing framework is proposed that utilizes novel relation-aware context learning technique. The proposed technique explores the graph constructs from the training data to learn the co-occurring context associations of object category labels using the graph edge connections. The proposed graph-based context learning technique defines the scene specific relation-awareness among semantic object categories, e.g., the probability of sky, road and building to co-exist in a scene is high. The proposed image parsing architecture (including the novel graph-based context learning technique) is evaluated on the benchmark datasets. In addition, a comprehensive comparison with existing image parsing techniques is presented to establish the efficacy of the scene-graph generation. The in-depth investigation of graph generation is presented to demonstrate the improvement in pixel-wise labeling.
Robust Deep Learning Models against Semantic-Preserving Adversarial Attack

Deep learning models can be fooled by small <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$l_{p}$</tex> -norm adversarial perturbations and natural perturbations in terms of attributes. Although the robustness against each perturbation has been explored, it remains a challenge to address the robustness against joint perturbations effectively. In this paper, we study the robustness of deep learning models against joint perturbations by proposing a novel attack mechanism named Semantic-Preserving Adversarial (SPA) attack, which can then be used to enhance adversarial training. Specifically, we introduce an attribute manipulator to generate natural and human-comprehensible perturbations and a noise generator to generate diverse adversarial noises. Based on such combined noises, we optimize both the attribute value and the diversity variable to generate jointly-perturbed samples. For robust training, we adversarially train the deep learning model against the generated joint perturbations. Empirical results on four benchmarks show that the SPA attack causes a larger performance decline with small <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$l_{\infty}$</tex> norm-ball constraints compared to existing approaches. Furthermore, our SPA-enhanced training outperforms existing defense methods against such joint perturbations.
Model Driven Deep Unfolding Network for Extreme Low-Light Image Enhancement and Denoising

Low visibility and severe noise are two main degradations in extreme low-light images. Nevertheless, existing low-light image enhancement methods often fail to handle real low-light images with strong noise. To address this issue, We propose a deep unfolding network based on the robust Retinex model with an additional noise term. In particular, we design an optimization model with implicit priors and employ the proximal gradient descent (PGD) technique to alternately solve three iterative sub-problems of the optimization model in a data-driven manner. The proposed method combines the interpretability of model-based methods with the speed and strong fitting ability of learning-based methods. In addition, we collect an extreme low-light sRGB image dataset (E-LOL) containing noisy low/normal-light image pairs. Extensive experimental results demonstrate that our method outperforms state-of-the-art methods in enhancing noisy low-light images and obtains better-exposed illumination, richer colors and textures.
Bio-Inspired Dual-Network Model to Tackle Statistical Heterogeneity in Federated Learning

The problem of statistical heterogeneity in Federated Learning has been a major challenge, with existing solutions making unrealistic assumptions about the availability of shared datasets and high bandwidth between clients and the server. Solving this problem is crucial for the success of Federated Learning in real-world scenarios. In this work, we propose a biologically inspired dual-network model FedDual, which mimics how the human brain learns and memorizes the information. The model consists of a neocortical and a hippocampal network similar to those in the human brain. The hippocampal network is comprised by an image classification model, while the neocortical network is a variational auto-encoder responsible for long-term and re-callable memory. In this manner, FedDual uses the neocortical network to generate pseudo-patterns (synthetic data) on the server (global model). This allows for the hippocampal network to be trained with these pseudo-patterns. The dual-network architecture allows devices to share information via the weight updates of the neocortical network to the server without sending the actual data. We compare FedDual against alternatives elsewhere in the literature when applied to widely available datasets. FedDual not only achieves a margin of accuracy improvement over the alternatives, but also converges faster, requiring less communication rounds.
LFLD: A Lightweight Facial Landmark Detector Based on Auxiliary Heatmap

In recent years, heatmap regression models have attracted much attention due to their superior performance in locating facial landmarks. These models overcome the lack of spatial and contextual information of coordinate regression models. However, there are still two major problems with these models; (1) they are computationally expensive, (2) they have the quantization error. To address two issues, we present a lightweight facial landmark detector named LFLD that unifies ShuffleNetV2 with the principles of the high-resolution structure for facial landmark detection. Our LFLD has high accuracy and significantly reduced computation costs compared to other heatmap regression models. In addition, the LFLD is equipped with two detection heads that generate the primary heatmaps and the auxiliary heatmaps. The primary heatmaps represent the integer part of the predicted facial landmarks coordinates, and the auxiliary heatmaps indicate the fractional part. Our LFLD uses these two heatmaps to jointly represent the predicted facial landmarks coordinates. The extensive experimental results show that LFLD can achieve satisfactory results in the case of low parameters and FLOPs. Particularly for the NME, our LFLD reaches 4.31% on WFLW and 3.54% on 300W. In heatmap regression, when compared to HRNet, our LFLD has lower NME than HRNet and has 45% fewer parameters and 66% fewer FLOPs. Finally, our model has fewer parameters than large networks, and the detection accuracy is comparable to large networks.
Analyzing the Accuracy, Representations, and Explainability of Various Loss Functions for Deep Learning

Deep learning utilizes a vast amounts of training data and updates weight parameters so as to minimize the loss between a predicted probability and a ground truth label. Generally, we use cross-entropy as the loss function. Although loss functions for image classification other than cross-entropy exist, their efficacy has not been adequately investigated. In this work, we extensively analyze models trained with different loss functions and clarify the properties of each. Specifically, we analyze the feature space and explainability as well as the classification accuracy on various benchmark datasets and network architectures. For feature space and explainability, we investigate the effectiveness of each loss function by quantitative and qualitative evaluations. We then discuss the properties and improvements of each.
Generating Adversarial Examples with Better Transferability via Masking Unimportant Parameters of Surrogate Model

Deep neural networks (DNNs) have been shown to be vulnerable to adversarial examples. Moreover, the transferability of the adversarial examples has received broad attention in recent years, which means that adversarial examples crafted by a surrogate model can also attack unknown models. This phenomenon gave birth to the transfer-based adversarial attacks, which aim to improve the transferability of the generated adversarial examples. In this paper, we propose to improve the transferability of adversarial examples in the transfer-based attack via masking unimportant parameters (MUP). The key idea in MUP is to refine the pretrained surrogate models to boost the transfer-based attack. Based on this idea, a Taylor expansion-based metric is used to evaluate the parameter importance score and the unimportant parameters are masked during the generation of adversarial examples. This process is simple, yet can be naturally combined with various existing gradient-based optimizers for generating adversarial examples, thus further improving the transferability of the generated adversarial examples. Extensive experiments are conducted to validate the effectiveness of the proposed MUP-based methods.
LU-Net: Invertible Neural Networks Based on Matrix Factorization

LU-Net is a simple and fast architecture for invertible neural networks (INN) that is based on the factorization of quadratic weight matrices A=LU, where L is a lower triangular matrix with ones on the diagonal and U an upper triangular matrix. Instead of learning a fully occupied matrix A, we learn L and U separately. If combined with an invertible activation function, such layers can easily be inverted whenever the diagonal entries of U are different from zero. Also, the computation of the determinant of the Jacobian matrix of such layers is cheap. Consequently, the LU architecture allows for cheap computation of the likelihood via the change of variables formula and can be trained according to the maximum likelihood principle. In our numerical experiments, we test the LU-Net architecture as generative model on several academic datasets. We also provide a detailed comparison with conventional invertible neural networks in terms of performance, training as well as run time.
Question Generation via Generative Adversarial Networks

Question Generation (QG) models suffer from exposure bias. Exposure bias refers to the problem that an autoregressive generator is trained given the observable preceding context, though it runs during test conditioned on the self-generated context. Exposure bias unavoidably causes performance degradation. In this paper, we enhance the GAN-based SeqGAN model which slightly contributes to alleviating exposure bias. Instead of RNN and CNN, we leverage UniLMv2 to construct the generator and discriminator in SeqGAN. Different from RNN who generates the current token in terms of the closely adjacent token, UniLMv2 depends on all the preceding contexts to predict the current token. This reduces the risk of severe token-by-token error propagation caused by exposure bias. During the representation learning process, we retain the adversarial mechanism of SeqGAN that drives the generation of human-like languages, as well as the rewarding mechanism which reduces the loss if subsequent tokens are perfectly generated. Though, we carry out a straight rewording using N-grams. The experiments on the benchmark dataset SQuAD show that our model outperforms the state-of-the-art QG models for the metrics BLEU-4, ROUGE-L and METEOR. Besides, our model is less time-consuming compared to the original SeqGAN. The time consumption of 2nd-stage learning is 1h47m on GeForce RTX 2080 Ti.
OSDet: Towards Open-Set Object Detection

In computer vision, almost all object detectors solve close-set object detection, assuming only known objects appear in the test environment. However, open-set object detection remains challenging since unknown objects need to be separated from the background unsupervised. To this aim, most existing state-of-the-art open-set detectors rely on the class-agnostic properties of the Region Proposal Network (RPN) to generate pseudo-labels for unknown objects. However, we notice that RPN generates low-quality pseudo-labels that significantly affect the performance of open-set detectors. Therefore, we propose Focusing on Location for Unknowns (FLU), which consists of class-agnostic pretraining and class-specific training, to improve the quality of pseudo-labels. Class-agnostic pretraining locates objects without learning to classify, and class-specific training generates high-quality pseudo-labels for unknowns. Besides, we implement a simple Object-level Contrastive Learning (OCL). OCL is helpful for the network to learn more discriminative features of objects, further reducing the confusion about known and unknown objects. With FLU and OCL, we present a novel open-set detector called OSDet. Ablations reveal the merits of FLU and OCL. Moreover, extensive experiments show that OSDet can significantly improve the performance of open-set detectors. e.g., OSDet reduces the Wilderness Impact by 20%~25% on seven open-set object detection benchmarks.
DRepT: Anomaly Detection Based on Transfer of Defect Representation with Transmittance Mask

Automated surface anomaly detection is important in industry. The goal with this task is to classify the query image and its pixel as either “Normal” or “Anomaly” with high precision for industrial products. Unlike general two-class classification tasks, we can only obtain normal images (i.e., anomaly images cannot be obtained) for training because anomalies rarely occur in well-controlled production environments such as modern manufacturing factories. We propose a transfer-based anomaly detection method called Defect Representation Transfer-based Anomaly Detection (DRepT). This method can generate natural anomaly images for the target domain (i.e., real factory with only normal images) using knowledge of anomalies obtained in the source domain (i.e., known datasets with normal and anomaly images). We first focused on an anomaly image of an industrial product consisting of two elements: “background texture” and “defect representation”. At the source domain, a background texture image is generated from an anomaly image with texture invariance (i.e., defects are only removed) by using generative adversarial networks, and defect representation is modeled as the difference between an anomaly image and its background texture image with the newly proposed Gaussian mixture transmittance mask. At the target domain, anomaly images are generated by giving defect representation to normal images, and a anomaly detection model is finally trained. Compared with a method without real defects of other domains and a method based on pasting of the defect images, DRepT achieved highly precision.
Improving the Transferability of Adversarial Examples with Diverse Gradients

Previous works have proven the superior performance of ensemble-based black-box attacks on transferability. However, existing methods require significant difference in architecture among the source models to ensure gradient diversity. In this paper, we propose a Diverse Gradient Method (DGM), verifying that knowledge distillation is able to generate diverse gradients from unchangeable model architecture for boosting transferability. The core idea behind our DGM is to obtain transferable adversarial perturbations by fusing diverse gradients provided by a single source model and its distilled versions through an ensemble strategy. Experimental results show that DGM successfully crafts adversarial examples with higher transferability, only requiring extremely low training cost. Furthermore, our proposed method could be used as a flexible module to improve transferability of most of existing black-box attacks.
Towards Robustness of Few-Shot Text Classifiers

Few shot learning algorithms are designed to perform significantly well when the amount of annotated data is scarce. However, recent research shows that these algorithms are highly vulnerable to adversarial examples. Recent works have demonstrated the robustness of few-shot image classifiers. However, there is very little or no attention paid to the robustness of few-shot text classifiers. In this work, we highlight the vulnerability of few-shot text classifiers. We also provide an algorithm for adversarial training of few-shot classifiers that perform well in the presence of adversarial examples. We implemented this algorithm on existing state-of-the-art few-shot classifiers. Experimental results demonstrate that our algorithm resists adversarial attacks and performs better in the presence of adversarial examples.
Adapt-to-Learn Policy Network for Abstractive Multi-document Summarization

Abstractive multi-document summarization (MDS) aims to generate a summary for a set of topic-related documents, in which different documents may contain trivial and redundant information and present complementary or contradictory content. It is essential to extract salient information and detect redundancy across documents for abstractive MDS compared with single-document summarization. However, it is challenging for models to exploit salient information and generate concise summaries. In this paper, we propose an Adapt-to-Learn Policy (ALP) network to seamlessly adapt the key sentence selection and word generation over retrieval and generative agents, guided by the multi-document summarization reward. Moreover, our model learns latent dependencies among textual units and explicitly takes advantage of critical information by focusing on semantic similarity or discourse relations. Extensive experiments on WikiSum and Multi-News datasets confirm that our method is superior to prior competitive baselines, and experimental analyses show that higher-quality summaries and more fluent word order can be generated in our ALP network.