Reviewerly/exHarmony huggingface dataset:
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks

We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data points. We show that, the expected $0$-$1$ loss of a wide enough ReLU network trained with stochastic gradient descent (SGD) and random initialization can be bounded by the training loss of a random feature model induced by the network gradient at initialization, which we call a \textit{neural tangent random feature} (NTRF) model. For data distributions that can be classified by NTRF model with sufficiently small error, our result yields a generalization error bound in the order of $\tilde{\mathcal{O}}(n^{-1/2})$ that is independent of the network width. Our result is more general and sharper than many existing generalization error bounds for over-parameterized neural networks. In addition, we establish a strong connection between our generalization error bound and the neural tangent kernel (NTK) proposed in recent work.
Minimum Error Entropy Kalman Filter

To date, most linear and nonlinear Kalman filters (KFs) have been developed under the Gaussian assumption and the well-known minimum mean square error (MMSE) criterion. In order to improve the robustness with respect to impulsive (or heavy-tailed) non-Gaussian noises, the maximum correntropy criterion (MCC) has recently been used to replace the MMSE criterion in developing several robust Kalman-type filters. To deal with more complicated non-Gaussian noises such as noises from multimodal distributions, in this article, we develop a new Kalman-type filter, called minimum error entropy KF (MEE-KF), by using the minimum error entropy (MEE) criterion instead of the MMSE or MCC. Similar to the MCC-based KFs, the proposed filter is also an online algorithm with the recursive process, in which the propagation equations are used to give prior estimates of the state and covariance matrix, and a fixed-point algorithm is used to update the posterior estimates. In addition, the MEE extended KF (MEE-EKF) is also developed for performance improvement in the nonlinear situations. The high accuracy and strong robustness of MEE-KF and MEE-EKF are confirmed by experimental results.
NGDNet: Nonuniform Gaussian-label distribution learning for infrared head pose estimation and on-task behavior understanding in the classroom

Head pose estimation (HPE) under active infrared (IR) illumination has attracted much attention in the fields of computer vision and machine learning. However, IRHPE often suffers from the problems of low-quality IR images and ambiguous head pose. To tackle these issues, we propose a novel nonuniform Gaussian-label distribution learning network (NGDNet) for the HPE task. First, we reveal the essential properties from two different perspectives: 1) two head pose images change differently in pitch and yaw directions with the same angle increasing on the central pose; 2) the IR head pose variation first increases and then decreases in the pitch direction. Subsequently, the first property indicates the pose image label as a nonuniform label distribution (Gaussian function) with different long and short axes. The second property is leveraged to determine the distribution size in accordance with the similarities of adjacent hand poses. Lastly, the proposed NGDNet is verified on a new IRHPE dataset, which is built by our research group. Experimental results on several datasets demonstrate the effectiveness of the proposed model. Compared with conventional algorithms, our NGDNet model achieves state-of-the-art performance with 77.39% on IRHPE, 99.08% on CAS-PEAL-R1, and 87.41% on Pointing’04. Our code is publicly available at https://github.com/TingtingSL/NGDNet.
Learning Sheared EPI Structure for Light Field Reconstruction

Research in light field reconstruction focuses on synthesizing novel views with the assistance of depth information. In this paper, we present a learning-based light field reconstruction approach by fusing a set of sheared epipolar plane images (EPIs). We start by showing that a patch in a sheared EPI will exhibit a clear structure when the sheared value equals the depth of that patch. By taking advantage of this pattern, a convolutional neural network (CNN) is then trained to evaluate the sheared EPIs, and output a reference score for fusing the sheared EPIs. The proposed CNN is elaborately designed to learn the similarity degree between the input sheared EPI and the ground truth EPI. Therefore, no depth information is required for network training and reasoning. We demonstrate the high performance of the proposed method through evaluations on synthetic scenes, real-world scenes, and challenging microscope light fields. We also show a further application of our proposed network for depth inference.
Uniform convergence may be unable to explain generalization in deep learning

Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learning, all based on the fundamental learning-theoretic technique of uniform convergence. While it is well-known that many of these existing bounds are numerically large, through numerous experiments, we bring to light a more concerning aspect of these bounds: in practice, these bounds can {\em increase} with the training dataset size. Guided by our observations, we then present examples of overparameterized linear classifiers and neural networks trained by gradient descent (GD) where uniform convergence provably cannot ``explain generalization'' -- even if we take into account the implicit bias of GD {\em to the fullest extent possible}. More precisely, even if we consider only the set of classifiers output by GD, which have test errors less than some small $\epsilon$ in our settings, we show that applying (two-sided) uniform convergence on this set of classifiers will yield only a vacuous generalization guarantee larger than $1-\epsilon$. Through these findings, we cast doubt on the power of uniform convergence-based generalization bounds to provide a complete picture of why overparameterized deep networks generalize well.
Improved binary particle swarm optimization for feature selection with new initialization and search space reduction strategies

Feature selection (FS) is an important preprocessing technique for dimensionality reduction in classification problems. Particle swarm optimization (PSO) algorithms have been widely used as the optimizers for FS problems. However, with the increase of data dimensionality, the search space expands dramatically, which proposes significant challenges for optimization methods, including PSO. In this paper, we propose an improved sticky binary PSO (ISBPSO) algorithm for FS. ISBPSO adopts three new mechanisms based on a recently proposed binary PSO variant, sticky binary particle swarm optimization (SBPSO), to improve the evolutionary performance. First, a new initialization strategy using the feature weighting information based on mutual information is proposed. Second, a dynamic bits masking strategy for gradually reducing the search space during the evolutionary process is proposed. Third, based on the framework of memetic algorithms, a refinement procedure conducting genetic operations on the personal best positions of ISBPSO is used to alleviate the premature convergence problem. The results on 12 UCI datasets show that ISBPSO outperforms six benchmark PSO-based FS methods and two conventional FS methods (sequential forward selection and sequential backward selection) — ISBPSO obtains either higher or similar accuracies with fewer features in most cases. Moreover, ISBPSO substantially reduces the computation time compared with benchmark PSO-based FS methods. Further analysis shows that all the three proposed mechanisms are effective for improving the search performance of ISBPSO.
Efficient quantum-based security protocols for information sharing and data protection in 5G networks

Fifth generation (5G) networks aim at utilizing many promising communication technologies, such as Cloud Computing, Network Slicing, and Software Defined Networking. Supporting a massive number of connected devices with 5G advanced technologies and innovating new techniques will surely bring tremendous challenges for trust, security and privacy. Therefore, secure mechanisms and protocols are required as the basis for 5G networks to address this security challenges and follow security-by-design but also security-by-operations rules. In this context, new efficient cryptographic protocols and mechanisms are needed in order to design and achieve information sharing and data protection protocols in 5G networks. In the literature, several security schemes based on unproven assumptions of computational complexity and mathematical models were proposed. However, the cryptanalysis is able to break most of the existing proposals in the presence of several weakest links in the designs. Recently, quantum walks (QWs) have been introduced as an excellent mechanism for generating cryptographic keys due to its nonlinear chaotic dynamical performance, high sensitivity to initial control parameters, stability and very large key space theoretically strong enough to resist various known attacks. This paper firstly proposes two efficient hash function mechanisms for 5G networks applications based on QWs, namely QWHF-1 and QWHF-2. Then, based on these hash functions, two efficient security protocols for securing data in 5G network scenario are proposed. Performance analyses and simulation results show that the proposed approaches are characterized with high security, efficiency, and robustness against several well-known attacks which nominate them as excellent candidates for securing 5G applications.
A Distributed Control Framework of Multiple Unmanned Aerial Vehicles for Dynamic Wildfire Tracking

Wild-land fire fighting is a hazardous job. A key task for firefighters is to observe the “fire front” to chart the progress of the fire and areas that will likely spread next. Lack of information of the fire front causes many accidents. Using unmanned aerial vehicles (UAVs) to cover wildfire is promising because it can replace humans in hazardous fire tracking and significantly reduce operation costs. In this paper, we propose a distributed control framework designed for a team of UAVs that can closely monitor a wildfire in open space, and precisely track its development. The UAV team, designed for flexible deployment, can effectively avoid in-flight collisions and cooperate well with neighbors. They can maintain a certain height level to the ground for safe flight above fire. Experimental results are conducted to demonstrate the capabilities of the UAV team in covering a spreading wildfire.
Latent Ordinary Differential Equations for Irregularly-Sampled Time Series

Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these ODE-based models outperform their RNN-based counterparts on irregularly-sampled data.
GauGAN

We propose GauGAN, a GAN-based image synthesis model that can generate photo-realistic images given an input semantic layout. It is built on spatially-adaptive normalization, a simple but effective normalization layer. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and non-linearity layers. We show that this is sub-optimal as the normalization layers tend to "wash away" semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Our proposed method outperforms the previous methods by a large margin. Furthermore, the new method enables natural extension to control the style of the synthesized images. Given a style guide image, our style encoder network captures the style into a latent code, which our image generator network combines with the semantic layout via spatially-adaptive normalization to generate a photo-realistic image that respects both the style of the guide image and content of the semantic layout. Our method will enable people without drawing skills to effectively express their imagination. GauGAN in the inference time is a simple convolutional neural network. It runs real-time on most modern GPU cards. GauGAN is one of the recent research efforts in advancing GANs for real-time image rendering. We believe this is of interest to the SIGGRAPH and real-time communities.
Graph Transformer Networks

Graph neural networks (GNNs) have been widely used in representation learning on graphs and achieved state-of-the-art performance in tasks such as node classification and link prediction. However, most existing GNNs are designed to learn node representations on the fixed and homogeneous graphs. The limitations especially become problematic when learning representations on a misspecified graph or a heterogeneous graph that consists of various types of nodes and edges. In this paper, we propose Graph Transformer Networks (GTNs) that are capable of generating new graph structures, which involve identifying useful connections between unconnected nodes on the original graph, while learning effective node representation on the new graphs in an end-to-end fashion. Graph Transformer layer, a core layer of GTNs, learns a soft selection of edge types and composite relations for generating useful multi-hop connections so-call meta-paths. Our experiments show that GTNs learn new graph structures, based on data and tasks without domain knowledge, and yield powerful node representation via convolution on the new graphs. Without domain-specific graph preprocessing, GTNs achieved the best performance in all three benchmark node classification tasks against the state-of-the-art methods that require pre-defined meta-paths from domain knowledge.
A General Theory of Equivariant CNNs on Homogeneous Spaces

We present a general theory of Group equivariant Convolutional Neural Networks (G-CNNs) on homogeneous spaces such as Euclidean space and the sphere. Feature maps in these networks represent fields on a homogeneous base space, and layers are equivariant maps between spaces of fields. The theory enables a systematic classification of all existing G-CNNs in terms of their symmetry group, base space, and field type. We also answer a fundamental question: what is the most general kind of equivariant linear map between feature spaces (fields) of given types? We show that such maps correspond one-to-one with generalized convolutions with an equivariant kernel, and characterize the space of such kernels.
Adversarial Weight Perturbation Helps Robust Generalization

The study on improving the robustness of deep neural networks against adversarial examples grows rapidly in recent years. Among them, adversarial training is the most promising one, which flattens the input loss landscape (loss change with respect to input) via training on adversarially perturbed examples. However, how the widely used weight loss landscape (loss change with respect to weight) performs in adversarial training is rarely explored. In this paper, we investigate the weight loss landscape from a new perspective, and identify a clear correlation between the flatness of weight loss landscape and robust generalization gap. Several well-recognized adversarial training improvements, such as early stopping, designing new objective functions, or leveraging unlabeled data, all implicitly flatten the weight loss landscape. Based on these observations, we propose a simple yet effective Adversarial Weight Perturbation (AWP) to explicitly regularize the flatness of weight loss landscape, forming a double-perturbation mechanism in the adversarial training framework that adversarially perturbs both inputs and weights. Extensive experiments demonstrate that AWP indeed brings flatter weight loss landscape and can be easily incorporated into various existing adversarial training methods to further boost their adversarial robustness.
Optimizing Convolutional Neural Network Hyperparameters by Enhanced Swarm Intelligence Metaheuristics

Computer vision is one of the most frontier technologies in computer science. It is used to build artificial systems to extract valuable information from images and has a broad range of applications in various areas such as agriculture, business, and healthcare. Convolutional neural networks represent the key algorithms in computer vision, and in recent years, they have attained notable advances in many real-world problems. The accuracy of the network for a particular task profoundly relies on the hyperparameters’ configuration. Obtaining the right set of hyperparameters is a time-consuming process and requires expertise. To approach this concern, we propose an automatic method for hyperparameters’ optimization and structure design by implementing enhanced metaheuristic algorithms. The aim of this paper is twofold. First, we propose enhanced versions of the tree growth and firefly algorithms that improve the original implementations. Second, we adopt the proposed enhanced algorithms for hyperparameters’ optimization. First, the modified metaheuristics are evaluated on standard unconstrained benchmark functions and compared to the original algorithms. Afterward, the improved algorithms are employed for the network design. The experiments are carried out on the famous image classification benchmark dataset, the MNIST dataset, and comparative analysis with other outstanding approaches that were tested on the same problem is conducted. The experimental results show that both proposed improved methods establish higher performance than the other existing techniques in terms of classification accuracy and the use of computational resources.
A readiness assessment framework for Blockchain adoption: A healthcare case study

Blockchain technology has been gaining traction across different sectors. It has the potential to immensely benefit the healthcare sector, given the sector's inherent complexities, problems, and inefficiencies. However, to date, no comprehensive, evidence-based effort has been made to understand the readiness of this sector for blockchain adoption. We proposed a readiness assessment framework that encompasses the complex interplay of different underlying factors, social structures, and institutional mechanisms and that covers all key stakeholders. Based on a systematic literature review, the framework is applied to the UAE's healthcare sector and its applicability and usefulness is established. The findings show the multifaceted significance of government readiness in driving blockchain initiatives. Large firms are found to be more willing to leverage the opportunities afforded by blockchain. Lack of clarity on blockchain regulations and laws, and issues pertaining to privacy and trust are found to affect the readiness of all stakeholders. The proposed framework and the study's findings will be useful in guiding policy interventions and developing support mechanisms to strengthen areas related to blockchain adoption.
Detection of False Data Injection Attacks in Smart Grid: A Secure Federated Deep Learning Approach

As an important cyber-physical system (CPS), smart grid is highly vulnerable to cyber attacks. Amongst various types of attacks, false data injection attack (FDIA) proves to be one of the top-priority cyber-related issues and has received increasing attention in recent years. However, so far little attention has been paid to privacy preservation issues in the detection of FDIAs in smart grid. Inspired by federated learning, a FDIA detection method based on secure federated deep learning is proposed in this paper by combining Transformer, federated learning and Paillier cryptosystem. The Transformer, as a detector deployed in edge nodes, delves deep into the connection between individual electrical quantities by using its multi-head self-attention mechanism. By using federated learning framework, our approach utilizes the data from all nodes to collaboratively train a detection model while preserving data privacy by keeping the data locally during training. To improve the security of federated learning, a secure federated learning scheme is designed by combing Paillier cryptosystem with federated learning. Through extensive experiments on the IEEE 14-bus and 118-bus test systems, the effectiveness and superiority of the proposed method are verifed.
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average.

Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Zi-Yi Dou, Keyi Yu, Antonios Anastasopoulos. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations

Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including different forms of gradient compression or computing local models and mixing them iteratively. In this paper we propose Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients. We propose both synchronous and asynchronous implementations of Qsparse-local-SGD. We analyze convergence for Qsparse-local-SGD in the distributed case, for smooth non-convex and convex objective functions. We demonstrate that Qsparse-local-SGD converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. We use Qsparse-local-SGD to train ResNet-50 on ImageNet, and show that it results in significant savings over the state-of-the-art, in the number of bits transmitted to reach target accuracy.