Reviewerly/exHarmony huggingface dataset:
Weakly Supervised Fine-Grained Image Classification via Guassian Mixture Model Oriented Discriminative Learning

Existing weakly supervised fine-grained image recognition (WFGIR) methods usually pick out the discriminative regions from the high-level feature maps directly. We discover that due to the operation of stacking local receptive filed, Convolutional Neural Network causes the discriminative region diffusion in high-level feature maps, which leads to inaccurate discriminative region localization. In this paper, we propose an end-to-end Discriminative Feature-oriented Gaussian Mixture Model (DF-GMM), to address the problem of discriminative region diffusion and find better fine-grained details. Specifically, DF-GMM consists of 1) a low-rank representation mechanism (LRM), which learns a set of low-rank discriminative bases by Gaussian Mixture Model (GMM) to accurately select discriminative details and filter more irrelevant information in high-level semantic feature maps, 2) a low-rank representation reorganization mechanism (LR2M) which resumes the space information of low-rank discriminative bases to reconstruct the low-rank feature maps. By recovering the low-rank discriminative bases into the same embedding space of highlevel feature maps, LR2M alleviates the discriminative region diffusion problem in high-level feature map and discriminative regions can be located more precisely on the new low-rank feature maps. Extensive experiments verify that DF-GMM yields the best performance under the same settings with the most competitive approaches, in CUBBird, Stanford-Cars datasets, and FGVC Aircraft.
E2I: Generative Inpainting From Edge to Image

Deep learning-based methods especially using convolutional neural network (CNN) and generative adversarial network (GAN) have achieved certain success for the task of image inpainting. The previous methods usually try to generate the content in the missing areas from scratch. However, these methods have difficulty in producing salient image structures that appear natural and consistent with the neighborhood, especially when the missing area is large. In this paper, we address the challenge by introducing edges into the convolutional GAN-based inpainting. We split the inpainting task into two steps: first edge generation, then edge-based image generation. We adopt CNN to accomplish the two steps and use GAN-based training, thus our method is named E2I: generative inpainting from edge to image. Specifically, we adopt a deep network-based edge detector to achieve an edgeness map of an incomplete image, then we fill-in the missing areas in the edgeness map, and finally generate the missing pixels with the assistance of the complete edgeness map. We verify the proposed method on three challenging image datasets: Places2, ImageNet, and CelebA. We also compare our method with the state-of-the-arts on the Places2 test set. Our experimental results demonstrate the superior performance of our method in producing more plausible inpainting results.
Pattern-Structure Diffusion for Multi-Task Learning

Inspired by the observation that pattern structures high-frequently recur within intra-task also across tasks, we propose a pattern-structure diffusion (PSD) framework to mine and propagate task-specific and task-across pattern structures in the task-level space for joint depth estimation, segmentation and surface normal prediction. To represent local pattern structures, we model them as small-scale graphlets, and propagate them in two different ways, i.e., intra-task and inter-task PSD. For the former, to overcome the limit of the locality of pattern structures, we use the high-order recursive aggregation on neighbors to multiplicatively increase the spread scope, so that long-distance patterns are propagated in the intra-task space. In the inter-task PSD, we mutually transfer the counterpart structures corresponding to the same spatial position into the task itself based on the matching degree of paired pattern structures therein. Finally, the intra-task and inter-task pattern structures are jointly diffused among the task-level patterns, and encapsulated into an end-to-end PSD network to boost the performance of multi-task learning. Extensive experiments on two widely-used benchmarks demonstrate that our proposed PSD is more effective and also achieves the state-of-the-art or competitive results.
Deep Adversarial Decomposition: A Unified Framework for Separating Superimposed Images

Separating individual image layers from a single mixed image has long been an important but challenging task. We propose a unified framework named "deep adversarial decomposition" for single superimposed image separation. Our method deals with both linear and non-linear mixtures under an adversarial training paradigm. Considering the layer separating ambiguity that given a single mixed input, there could be an infinite number of possible solutions, we introduce a "Separation-Critic" - a discriminative network which is trained to identify whether the output layers are well-separated and thus further improves the layer separation. We also introduce a "crossroad L1" loss function, which computes the distance between the unordered outputs and their references in a crossover manner so that the training can be well-instructed with pixel-wise supervision. Experimental results suggest that our method significantly outperforms other popular image separation frameworks. Without specific tuning, our method achieves the state of the art results on multiple computer vision tasks, including the image deraining, photo reflection removal, and image shadow removal.
Explorable Super Resolution

Single image super resolution (SR) has seen major performance leaps in recent years. However, existing methods do not allow exploring the infinitely many plausible reconstructions that might have given rise to the observed low-resolution (LR) image. These different explanations to the LR image may dramatically vary in their textures and fine details, and may often encode completely different semantic information. In this paper, we introduce the task of explorable super resolution. We propose a framework comprising a graphical user interface with a neural network backend, allowing editing the SR output so as to explore the abundance of plausible HR explanations to the LR input. At the heart of our method is a novel module that can wrap any existing SR network, analytically guaranteeing that its SR outputs would precisely match the LR input, when down- sampled. Besides its importance in our setting, this module is guaranteed to decrease the reconstruction error of any SR network it wraps, and can be used to cope with blur kernels that are different from the one the network was trained for. We illustrate our approach in a variety of use cases, ranging from medical imaging and forensics, to graphics.
Robust Superpixel-Guided Attentional Adversarial Attack

Deep Neural Networks are vulnerable to adversarial samples, which can fool classifiers by adding small perturbations onto the original image. Since the pioneering optimization-based adversarial attack method, many following methods have been proposed in the past several years. However most of these methods add perturbations in a "pixel-wise" and "global" way. Firstly, because of the contradiction between the local smoothness of natural images and the noisy property of these adversarial perturbations, this "pixel-wise" way makes these methods not robust to image processing based defense methods and steganalysis based detection methods. Secondly, we find adding perturbations to the background is less useful than to the salient object, thus the "global" way is also not optimal. Based on these two considerations, we propose the first robust superpixel-guided attentional adversarial attack method. Specifically, the adversarial perturbations are only added to the salient regions and guaranteed to be same within each superpixel. Through extensive experiments, we demonstrate our method can preserve the attack ability even in this highly constrained modification space. More importantly, compared to existing methods, it is significantly more robust to image processing based defense and steganalysis based detection.
NLDN: Non-local dehazing network for dense haze removal

Single image dehazing is one of the most challenging and important tasks in computer vision and image processing. In this paper, we propose a Non-local Dehazing Network (NLDN), which learns the mapping between hazy images and haze-free images. Our network architecture consists three components: the first is full point-wise convolutional part, which extracts Non-local statistical regularities; the second is feature combination part, which learns the spatial relation of statistical regularities; the third is reconstruction part, which recovers the haze-free image by the features extracted from the second part. By using these three components, we obtain a high quality dehazing result. Experimental results show that our method performs favorably against other state-of-the-art methods on both synthetic dataset and real-world images.
A Decentralized Federated Learning Approach for Connected Autonomous Vehicles

In this paper, we propose an autonomous blockchain-based federated learning (BFL) design for privacy-aware and efficient vehicular communication networking, where local on-vehicle machine learning (oVML) model updates are exchanged and verified in a distributed fashion. BFL enables on-vehicle machine learning without any centralized training data or coordination by utilizing the consensus mechanism of the blockchain. Relying on a renewal reward approach, we develop a mathematical framework that features the controllable network and BFL parameters, such as the retransmission limit, block size, block arrival rate, and the frame sizes, so as to capture their impact on the system-level performance. More importantly, our rigorous analysis of oVML system dynamics quantifies the end-to-end delay with BFL, which provides important insights into deriving optimal block arrival rate by considering communication and consensus delays.
DeepHate: Hate Speech Detection via Multi-Faceted Text Representations

Online hate speech is an important issue that breaks the cohesiveness of online social communities and even raises public safety concerns in our societies. Motivated by this rising issue, researchers have developed many traditional machine learning and deep learning methods to detect hate speech in online social platforms automatically. However, most of these methods have only considered single type textual feature, e.g., term frequency, or using word embeddings. Such approaches neglect the other rich textual information that could be utilized to improve hate speech detection. In this paper, we propose DeepHate, a novel deep learning model that combines multi-faceted text representations such as word embeddings, sentiments, and topical information, to detect hate speech in online social platforms. We conduct extensive experiments and evaluate DeepHate on three large publicly available real-world datasets. Our experiment results show that DeepHate outperforms the state-of-the-art baselines on the hate speech detection task. We also perform case studies to provide insights into the salient features that best aid in detecting hate speech in online social platforms.
Industrial Wastewater Management using Blockchain Technology: Architecture, Requirements, and Future Directions

The inefficient disposal of industrial waste causes water and soil pollution; thus, to protect the health of different ecosystems, industrial wastewater management is essential. Wastewater treatment's aim is to clean and protect water and to extract pollutants and remove toxicants before the remaining water is discharged into the environment. Currently, the disposal of industrial wastewater is an important topic because of the serious threat of wastewater to aquatic life and surrounding environment. Therefore, wastewater must be effectively treated and safely discharged into water bodies. In this study, for industrial wastewater management, we aim to use blockchain technology. Thus, we propose a conceptual architecture based on blockchain and present three case studies. Subsequently, we discussed requirements for incorporating blockchain into wastewater management. Finally, we demonstrated directions for future work.
An adaptive Harris hawks optimization technique for two dimensional grey gradient based multilevel image thresholding

A metaheuristic algorithm called Harris hawks optimization (HHO) is gaining its popularity among its clan and useful for optimization. In this algorithm, the prey gets completely exhausted when the escape energy is equal to zero, therefore it fails to explore further. The random operator chosen in the existing method is a wastage of search agents (Harris hawk). To overcome this issue, we propose an adaptive Harris Hawks optimization (AHHO) technique. In this work, the mutation is employed to restrict the escape energy within the range 0,2, except for the mutation interval. Our method adaptively decides the chance of the Harris hawk would do perch along with the other family members or move to a random tall tree with the help of average fitness. The proposed AHHO algorithm is benchmarked with 23 classical test functions and 30 modern test function from CEC 2014 test suite consisting of unimodal, multimodal, hybrid and composite functions. The qualitative and quantitative analysis, which include metrics such as statistical results, convergence curves, p-value from Wilcoxon rank-sum test and Friedman mean rank. It reveals that AHHO provides good results when compared with other well-known nature-inspired algorithms. It can be used for multilevel thresholding which is an optimization problem. Recently, 2D histogram based multilevel image thresholding techniques are becoming more popular for different image processing applications. The local averaging scheme used for the construction of a 2D histogram in existing methods fails to preserve the edge information. The choice of the diagonal pixels only results in the loss of information making the earlier multi-level thresholding methods inefficient to retain the spatial correlation information. Although the computation of 2D histogram based on grey gradient information is a better way to threshold an image, it faces problems due to the presence of the edge magnitude peaks. These problems are solved by investigating an improved 2D grey gradient (I2DGG) method, a new technique is suggested in this paper to suppress high edge magnitudes. The I2DGG is a maximization problem, which requires an exhaustive search process. Therefore, AHHO is used to obtain the optimal threshold values. The result of our proposed AHHO based multilevel thresholding using the I2DGG method is obtained using all the 500 images from the Berkeley Segmentation Data set (BSDS 500). When we compare the proposed method I2DGG with 2D Tsallis entropy and 1D Tsallis entropy based multilevel thresholding, the I2DGG outperforms other methods. The experimental results are also compared with the state-of-art optimization-based multilevel thresholding methods, which shows our proposed method is beneficial to the segmentation field of image processing.
The Influence of Immersive Virtual Reality Systems on Online Social Application

This study presents a face-to-face simulation of social interaction based on scene roaming, real-time voice capture, and action capture. This paper aimed to compare the difference between social and traditional plane social communication, analyzing its advantages and shortcomings. In particular, we designed an experiment to study the performance of face-to-face simulation based on virtual reality technology, observing the adaptability of the user to the system and the accuracy of body language recognition. We developed an immersive virtual reality social application (IVRSA), which has a basic social function. As an experimental platform, IVRSA uses Unity3D as its engine, and HTC VIVE as an external input/output device, with voice communication, immersive virtual tour, and head and hand movement simulation functions. We recruited 30 volunteers for the test. The test consisted of two parts. The first part was to provide a news topic for volunteers to freely communicate in IVRSA and WeChat. After communication, we used questionnaires to obtain feedback results of the test and compared the two social applications. In the second part, some volunteers were given a list of actions, which they were asked to describe to the rest of volunteers in the form of body expression, letting them guess the action they were performing. After the end of test, the accuracy rate and the time used were analyzed. Results showed that user’s intention expression efficiency was very high in the immersive virtual reality social system. Body tracking devices with more tracking points can provide better body expression effect.
GCTI-SN: Geometry-inspired chemical and tissue invariant stain normalization of microscopic medical images

Stain normalization of microscopic images is the first pre-processing step in any computer-assisted automated diagnostic tool. This paper proposes Geometry-inspired Chemical-invariant and Tissue Invariant Stain Normalization method, namely GCTI-SN, for microscopic medical images. The proposed GCTI-SN method corrects for illumination variation, stain chemical, and stain quantity variation in a unified framework by exploiting the underlying color vector space’s geometry. While existing stain normalization methods have demonstrated their results on a single tissue and stain type, GCTI-SN is benchmarked on three cancer datasets of three cell/tissue types prepared with two different stain chemicals. GCTI-SN method is also benchmarked against the existing methods via quantitative and qualitative results, validating its robustness for stain chemical and cell/tissue type. Further, the utility and the efficacy of the proposed GCTI-SN stain normalization method is demonstrated diagnostically in the application of breast cancer detection via a CNN-based classifier.
Encrypted Network Traffic Classification Using Deep and Parallel Network-in-Network Models

Network traffic classification aims to recognize different application or traffic types by analyzing received data packets. This paper presents a neural network model with deep and parallel network-in-network (NIN) structures for classifying encrypted network traffic. Comparing with standard convolutional neural networks (CNN), NIN adopts a micro network after each convolution layer to enhance local modeling. Besides, NIN utilizes a global average pooling instead of traditional fully connected layers before final classification, which reduces the number of model parameters significantly. In our proposed method, deep NIN models with multiple MLP convolutional layers are built to map fixed-length packet vectors towards application or traffic labels. Furthermore, a parallel decision strategy of building two sub-networks to process packet header and packet body separately is designed considering that they may carry different kinds of clues for classification. The results of our experiments on the “ISCX VPN-nonVPN” encrypted traffic dataset show that NIN models can achieve a better balance between classification accuracy and model complexity than conventional CNNs. The parallel decision strategy can further improve the accuracy of using single NIN model for encrypted network traffic classification. Finally, the test set F1 scores of 0.983 and 0.985 are achieved for traffic characterization and application identification respectively.
Dense Steerable Filter CNNs for Exploiting Rotational Symmetry in Histology Images

Histology images are inherently symmetric under rotation, where each orientation is equally as likely to appear. However, this rotational symmetry is not widely utilised as prior knowledge in modern Convolutional Neural Networks (CNNs), resulting in data hungry models that learn independent features at each orientation. Allowing CNNs to be rotation-equivariant removes the necessity to learn this set of transformations from the data and instead frees up model capacity, allowing more discriminative features to be learned. This reduction in the number of required parameters also reduces the risk of overfitting. In this paper, we propose Dense Steerable Filter CNNs (DSF-CNNs) that use group convolutions with multiple rotated copies of each filter in a densely connected framework. Each filter is defined as a linear combination of steerable basis filters, enabling exact rotation and decreasing the number of trainable parameters compared to standard filters. We also provide the first in-depth comparison of different rotation-equivariant CNNs for histology image analysis and demonstrate the advantage of encoding rotational symmetry into modern architectures. We show that DSF-CNNs achieve state-of-the-art performance, with significantly fewer parameters, when applied to three different tasks in the area of computational pathology: breast tumour classification, colon gland segmentation and multi-tissue nuclear segmentation.
Prototypical Siamese Networks for Few-shot Learning

We propose a novel architecture, called Prototypical Siamese Networks, for few-shot learning, where a classifier must generalize to new classes not seen in the training set, given only a few examples of each class. Prototypical Siamese Networks add a new module to siamese networks to learn a high quality prototypical representation of each class. Compared to recent methods for few-shot learning, our method achieves state-of-the-art performance on few-shot learning. Experiments on two benchmarks validate the effectiveness of the proposed method.
Privacy Risks of General-Purpose Language Models

Recently, a new paradigm of building general-purpose language models (e.g., Google’s Bert and OpenAI’s GPT-2) in Natural Language Processing (NLP) for text feature extraction, a standard procedure in NLP systems that converts texts to vectors (i.e., embeddings) for downstream modeling, has arisen and starts to find its application in various downstream NLP tasks and real world systems (e.g., Google’s search engine [6]). To obtain general-purpose text embeddings, these language models have highly complicated architectures with millions of learnable parameters and are usually pretrained on billions of sentences before being utilized. As is widely recognized, such a practice indeed improves the state-of-the-art performance of many downstream NLP tasks. However, the improved utility is not for free. We find the text embeddings from general-purpose language models would capture much sensitive information from the plain text. Once being accessed by the adversary, the embeddings can be reverse-engineered to disclose sensitive information of the victims for further harassment. Although such a privacy risk can impose a real threat to the future leverage of these promising NLP tools, there are neither published attacks nor systematic evaluations by far for the mainstream industry-level language models. To bridge this gap, we present the first systematic study on the privacy risks of 8 state-of-the-art language models with 4 diverse case studies. By constructing 2 novel attack classes, our study demonstrates the aforementioned privacy risks do exist and can impose practical threats to the application of general-purpose language models on sensitive data covering identity, genome, healthcare and location. For example, we show the adversary with nearly no prior knowledge can achieve about 75% accuracy when inferring the precise disease site from Bert embeddings of patients’ medical descriptions. As possible countermeasures, we propose 4 different defenses (via rounding, differential privacy, adversarial training and subspace projection) to obfuscate the unprotected embeddings for mitigation purpose. With extensive evaluations, we also provide a preliminary analysis on the utility-privacy trade-off brought by each defense, which we hope may foster future mitigation researches.