OpenRaiser/Intern-Atlas huggingface dataset:
On Latent Distributions Without Finite Mean in Generative Models
Venue: arXiv cs.LG 2018
Year: 2018

We investigate the properties of multidimensional probability distributions in the context of latent space prior distributions of implicit generative models. Our work revolves around the phenomena arising while decoding linear interpolations between two random latent vectors -- regions of latent space in close proximity to the origin of the space are sampled causing distribution mismatch. We show that due to the Central Limit Theorem, this region is almost never sampled during the training process. As a result, linear interpolations may generate unrealistic data and their usage as a tool to check quality of the trained model is questionable. We propose to use multidimensional Cauchy distribution as the latent prior. Cauchy distribution does not satisfy the assumptions of the CLT and has a number of properties that allow it to work well in conjunction with linear interpolations. We also provide two general methods of creating non-linear interpolations that are easily applicable to a large family of common latent distributions. Finally we empirically analyze the quality of data generated from low-probability-mass regions for the DCGAN model on the CelebA dataset.
A Projection Method for Metric-Constrained Optimization
Venue: arXiv cs.NA 2018
Year: 2018

We outline a new approach for solving optimization problems which enforce triangle inequalities on output variables. We refer to this as metric-constrained optimization, and give several examples where problems of this form arise in machine learning applications and theoretical approximation algorithms for graph clustering. Although these problem are interesting from a theoretical perspective, they are challenging to solve in practice due to the high memory requirement of black-box solvers. In order to address this challenge we first prove that the metric-constrained linear program relaxation of correlation clustering is equivalent to a special case of the metric nearness problem. We then developed a general solver for metric-constrained linear and quadratic programs by generalizing and improving a simple projection algorithm originally developed for metric nearness. We give several novel approximation guarantees for using our framework to find lower bounds for optimal solutions to several challenging graph clustering problems. We also demonstrate the power of our framework by solving optimizing problems involving up to 10^{8} variables and 10^{11} constraints.
Multi-Cohort Intelligence Algorithm: An Intra- and Inter-group Learning Behavior based Socio-inspired Optimization Methodology
Venue: arXiv cs.NE 2018
Year: 2018

A Multi-Cohort Intelligence (Multi-CI) metaheuristic algorithm in emerging socio-inspired optimization domain is proposed. The algorithm implements intra-group and inter-group learning mechanisms. It focusses on the interaction amongst different cohorts. The performance of the algorithm is validated by solving 75 unconstrained test problems with dimensions up to 30. The solutions were comparing with several recent algorithms such as Particle Swarm Optimization, Covariance Matrix Adaptation Evolution Strategy, Artificial Bee Colony, Self-adaptive differential evolution algorithm, Comprehensive Learning Particle Swarm Optimization, Backtracking Search Optimization Algorithm and Ideology Algorithm. The Wilcoxon signed rank test was carried out for the statistical analysis and verification of the performance. The proposed Multi-CI outperformed these algorithms in terms of the solution quality including objective function value and computational cost, i.e. computational time and functional evaluations. The prominent feature of the Multi-CI algorithm along with the limitations are discussed as well. In addition, an illustrative example is also solved and every detail is provided.
Accelerating CNN inference on FPGAs: A Survey
Venue: arXiv cs.DC 2018
Year: 2018

Convolutional Neural Networks (CNNs) are currently adopted to solve an ever greater number of problems, ranging from speech recognition to image classification and segmentation. The large amount of processing required by CNNs calls for dedicated and tailored hardware support methods. Moreover, CNN workloads have a streaming nature, well suited to reconfigurable hardware architectures such as FPGAs. The amount and diversity of research on the subject of CNN FPGA acceleration within the last 3 years demonstrates the tremendous industrial and academic interest. This paper presents a state-of-the-art of CNN inference accelerators over FPGAs. The computational workloads, their parallelism and the involved memory accesses are analyzed. At the level of neurons, optimizations of the convolutional and fully connected layers are explained and the performances of the different methods compared. At the network level, approximate computing and datapath optimization methods are covered and state-of-the-art approaches compared. The methods and tools investigated in this survey represent the recent trends in FPGA CNN inference accelerators and will fuel the future advances on efficient hardware deep learning.
X-FDR: A Cross-Layer Routing Protocol for Multi-hop Full-Duplex Wireless Networks
Venue: arXiv cs.NI 2018
Year: 2018

The recent developments in self-interference (SI) cancellation techniques have led to the practical realization of full-duplex (FD) radios that can perform simultaneous transmission and reception. FD technology is attractive for various legacy communications standards. In this paper, after discussing the opportunities of FD technology at the network layer, we present a cross-layer aided routing protocol, termed as X-FDR, for multi-hop FD wireless networks. X-FDR exploits a Physical (PHY) layer model capturing imperfection of SI cancellation. At the medium access control (MAC) layer, X-FDR adopts an optimized MAC protocol which implements a power control mechanism without creating the hidden terminal problem. X-FDR exploits the unique characteristics of FD technology at the network layer to construct energy-efficient and low end-to-end latency routes in the network. Performance evaluation demonstrates the effectiveness of X-FDR in achieving the gains of FD at higher layers of the protocol stack.
Concept-Oriented Deep Learning
Venue: arXiv cs.AI 2018
Year: 2018

Concepts are the foundation of human deep learning, understanding, and knowledge integration and transfer. We propose concept-oriented deep learning (CODL) which extends (machine) deep learning with concept representations and conceptual understanding capability. CODL addresses some of the major limitations of deep learning: interpretability, transferability, contextual adaptation, and requirement for lots of labeled training data. We discuss the major aspects of CODL including concept graph, concept representations, concept exemplars, and concept representation learning systems supporting incremental and continual learning.
Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds
Venue: arXiv cs.CV 2018
Year: 2018

Deep learning systems extensively use convolution operations to process input data. Though convolution is clearly defined for structured data such as 2D images or 3D volumes, this is not true for other data types such as sparse point clouds. Previous techniques have developed approximations to convolutions for restricted conditions. Unfortunately, their applicability is limited and cannot be used for general point clouds. We propose an efficient and effective method to learn convolutions for non-uniformly sampled point clouds, as they are obtained with modern acquisition techniques. Learning is enabled by four key novelties: first, representing the convolution kernel itself as a multilayer perceptron; second, phrasing convolution as a Monte Carlo integration problem, third, using this notion to combine information from multiple samplings at different levels; and fourth using Poisson disk sampling as a scalable means of hierarchical point cloud learning. The key idea across all these contributions is to guarantee adequate consideration of the underlying non-uniform sample distribution function from a Monte Carlo perspective. To make the proposed concepts applicable to real-world tasks, we furthermore propose an efficient implementation which significantly reduces the GPU memory required during the training process. By employing our method in hierarchical network architectures we can outperform most of the state-of-the-art networks on established point cloud segmentation, classification and normal estimation benchmarks. Furthermore, in contrast to most existing approaches, we also demonstrate the robustness of our method with respect to sampling variations, even when training with uniformly sampled data only. To support the direct application of these concepts, we provide a ready-to-use TensorFlow implementation of these layers at https://github.com/viscom-ulm/MCCNN
Mix&Match - Agent Curricula for Reinforcement Learning
Venue: arXiv cs.LG 2018
Year: 2018

We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradually alter how the policy is represented internally. We show the broad applicability of our method by demonstrating significant performance gains in three different experimental setups: (1) We train an agent able to control more than 700 actions in a challenging 3D first-person task; using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.
Relational Deep Reinforcement Learning
Venue: arXiv cs.LG 2018
Year: 2018

We introduce an approach for deep reinforcement learning (RL) that improves upon the efficiency, generalization capacity, and interpretability of conventional approaches through structured perception and relational reasoning. It uses self-attention to iteratively reason about the relations between entities in a scene and to guide a model-free policy. Our results show that in a novel navigation and planning task called Box-World, our agent finds interpretable solutions that improve upon baselines in terms of sample complexity, ability to generalize to more complex scenes than experienced during training, and overall performance. In the StarCraft II Learning Environment, our agent achieves state-of-the-art performance on six mini-games -- surpassing human grandmaster performance on four. By considering architectural inductive biases, our work opens new directions for overcoming important, but stubborn, challenges in deep RL.
TrollSpot: Detecting misbehavior in commenting platforms
Venue: arXiv cs.SI 2018
Year: 2018

Commenting platforms, such as Disqus, have emerged as a major online communication platform with millions of users and posts. Their popularity has also attracted parasitic and malicious behav- iors, such as trolling and spamming. There has been relatively little research on modeling and safeguarding these platforms. As our key contribution, we develop a systematic approach to detect malicious users on commenting platforms focusing on having: (a) interpretable, and (b) fine-grained classification of malice. Our work has two key novelties: (a) we propose two classifications methods, with one following a two stage approach, which first maps observ- able features to behaviors and then maps these behaviors to user roles, and (b) we use a comprehensive set of 73 features that span four dimensions of information. We use 7 million comments during a 9 month period, and we show that our classification methods can distinguish between benign, and malicious roles (spammers, trollers, and fanatics) with a 0.904 AUC. Our work is a solid step to- wards ensuring that commenting platforms are a safe and pleasant medium for the exchange of ideas.
Real-time Surgical Tools Recognition in Total Knee Arthroplasty Using Deep Neural Networks
Venue: arXiv cs.CV 2018
Year: 2018

Total knee arthroplasty (TKA) is a commonly performed surgical procedure to mitigate knee pain and improve functions for people with knee arthritis. The procedure is complicated due to the different surgical tools used in the stages of surgery. The recognition of surgical tools in real-time can be a solution to simplify surgical procedures for the surgeon. Also, the presence and movement of tools in surgery are crucial information for the recognition of the operational phase and to identify the surgical workflow. Therefore, this research proposes the development of a real-time system for the recognition of surgical tools during surgery using a convolutional neural network (CNN). Surgeons wearing smart glasses can see essential information about tools during surgery that may reduce the complication of the procedures. To evaluate the performance of the proposed method, we calculated and compared the Mean Average Precision (MAP) with state-of-the-art methods which are fast R-CNN and deformable part models (DPM). We achieved 87.6% mAP which is better in comparison to the existing methods. With the additional improvements of our proposed method, it can be a future point of reference, also the baseline for operational phase recognition.
A Policy based Security Architecture for Software Defined Networks
Venue: arXiv cs.CR 2018
Year: 2018

As networks expand in size and complexity, they pose greater administrative and management challenges. Software Defined Networks (SDN) offer a promising approach to meeting some of these challenges. In this paper, we propose a policy driven security architecture for securing end to end services across multiple SDN domains. We develop a language based approach to design security policies that are relevant for securing SDN services and communications. We describe the policy language and its use in specifying security policies to control the flow of information in a multi-domain SDN. We demonstrate the specification of fine grained security policies based on a variety of attributes such as parameters associated with users and devices/switches, context information such as location and routing information, and services accessed in SDN as well as security attributes associated with the switches and Controllers in different domains. An important feature of our architecture is its ability to specify path and flow based security policies, which are significant for securing end to end services in SDNs. We describe the design and the implementation of our proposed policy based security architecture and demonstrate its use in scenarios involving both intra and inter-domain communications with multiple SDN Controllers. We analyse the performance characteristics of our architecture as well as discuss how our architecture is able to counteract various security attacks. The dynamic security policy based approach and the distribution of corresponding security capabilities intelligently as a service layer that enable flow based security enforcement and protection of multitude of network devices against attacks are important contributions of this paper.
Calculational Verification of Reactive Programs with Reactive Relations and Kleene Algebra
Venue: arXiv cs.LO 2018
Year: 2018

Reactive programs are ubiquitous in modern applications, and so verification is highly desirable. We present a verification strategy for reactive programs with a large or infinite state space utilising algebraic laws for reactive relations. We define novel operators to characterise interactions and state updates, and an associated equational theory. With this we can calculate a reactive program's denotational semantics, and thereby facilitate automated proof. Of note is our reasoning support for iterative programs with reactive invariants, which is supported by Kleene algebra. We illustrate our strategy by verifying a reactive buffer. Our laws and strategy are mechanised in Isabelle/UTP, which provides soundness guarantees, and practical verification support.
Mitigation of Policy Manipulation Attacks on Deep Q-Networks with Parameter-Space Noise
Venue: arXiv cs.LG 2018
Year: 2018

Recent developments have established the vulnerability of deep reinforcement learning to policy manipulation attacks via intentionally perturbed inputs, known as adversarial examples. In this work, we propose a technique for mitigation of such attacks based on addition of noise to the parameter space of deep reinforcement learners during training. We experimentally verify the effect of parameter-space noise in reducing the transferability of adversarial examples, and demonstrate the promising performance of this technique in mitigating the impact of whitebox and blackbox attacks at both test and training times.
Spatiotemporal Manifold Prediction Model for Anterior Vertebral Body Growth Modulation Surgery in Idiopathic Scoliosis
Venue: arXiv cs.CV 2018
Year: 2018

Anterior Vertebral Body Growth Modulation (AVBGM) is a minimally invasive surgical technique that gradually corrects spine deformities while preserving lumbar motion. However the selection of potential surgical patients is currently based on clinical judgment and would be facilitated by the identification of patients responding to AVBGM prior to surgery. We introduce a statistical framework for predicting the surgical outcomes following AVBGM in adolescents with idiopathic scoliosis. A discriminant manifold is first constructed to maximize the separation between responsive and non-responsive groups of patients treated with AVBGM for scoliosis. The model then uses subject-specific correction trajectories based on articulated transformations in order to map spine correction profiles to a group-average piecewise-geodesic path. Spine correction trajectories are described in a piecewise-geodesic fashion to account for varying times at follow-up exams, regressing the curve via a quadratic optimization process. To predict the evolution of correction, a baseline reconstruction is projected onto the manifold, from which a spatiotemporal regression model is built from parallel transport curves inferred from neighboring exemplars. The model was trained on 438 reconstructions and tested on 56 subjects using 3D spine reconstructions from follow-up exams, with the probabilistic framework yielding accurate results with differences of 2.1 +/- 0.6deg in main curve angulation, and generating models similar to biomechanical simulations.
AIQL: Enabling Efficient Attack Investigation from System Monitoring Data
Venue: arXiv cs.CR 2018
Year: 2018

The need for countering Advanced Persistent Threat (APT) attacks has led to the solutions that ubiquitously monitor system activities in each host, and perform timely attack investigation over the monitoring data for analyzing attack provenance. However, existing query systems based on relational databases and graph databases lack language constructs to express key properties of major attack behaviors, and often execute queries inefficiently since their semantics-agnostic design cannot exploit the properties of system monitoring data to speed up query execution. To address this problem, we propose a novel query system built on top of existing monitoring tools and databases, which is designed with novel types of optimizations to support timely attack investigation. Our system provides (1) domain-specific data model and storage for scaling the storage, (2) a domain-specific query language, Attack Investigation Query Language (AIQL) that integrates critical primitives for attack investigation, and (3) an optimized query engine based on the characteristics of the data and the semantics of the queries to efficiently schedule the query execution. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 857 GB of real system monitoring data (containing 2.5 billion events). Our evaluations on a real-world APT attack and a broad set of attack behaviors show that our system surpasses existing systems in both efficiency (124x over PostgreSQL, 157x over Neo4j, and 16x over Greenplum) and conciseness (SQL, Neo4j Cypher, and Splunk SPL contain at least 2.4x more constraints than AIQL).
Regularization by Denoising: Clarifications and New Interpretations
Venue: arXiv cs.CV 2018
Year: 2018

Regularization by Denoising (RED), as recently proposed by Romano, Elad, and Milanfar, is powerful image-recovery framework that aims to minimize an explicit regularization objective constructed from a plug-in image-denoising function. Experimental evidence suggests that the RED algorithms are state-of-the-art. We claim, however, that explicit regularization does not explain the RED algorithms. In particular, we show that many of the expressions in the paper by Romano et al. hold only when the denoiser has a symmetric Jacobian, and we demonstrate that such symmetry does not occur with practical denoisers such as non-local means, BM3D, TNRD, and DnCNN. To explain the RED algorithms, we propose a new framework called Score-Matching by Denoising (SMD), which aims to match a "score" (i.e., the gradient of a log-prior). We then show tight connections between SMD, kernel density estimation, and constrained minimum mean-squared error denoising. Furthermore, we interpret the RED algorithms from Romano et al. and propose new algorithms with acceleration and convergence guarantees. Finally, we show that the RED algorithms seek a consensus equilibrium solution, which facilitates a comparison to plug-and-play ADMM.
Adaptive feature recombination and recalibration for semantic segmentation: application to brain tumor segmentation in MRI
Venue: arXiv cs.CV 2018
Year: 2018

Convolutional neural networks (CNNs) have been successfully used for brain tumor segmentation, specifically, fully convolutional networks (FCNs). FCNs can segment a set of voxels at once, having a direct spatial correspondence between units in feature maps (FMs) at a given location and the corresponding classified voxels. In convolutional layers, FMs are merged to create new FMs, so, channel combination is crucial. However, not all FMs have the same relevance for a given class. Recently, in classification problems, Squeeze-and-Excitation (SE) blocks have been proposed to re-calibrate FMs as a whole, and suppress the less informative ones. However, this is not optimal in FCN due to the spatial correspondence between units and voxels. In this article, we propose feature recombination through linear expansion and compression to create more complex features for semantic segmentation. Additionally, we propose a segmentation SE (SegSE) block for feature recalibration that collects contextual information, while maintaining the spatial meaning. Finally, we evaluate the proposed methods in brain tumor segmentation, using publicly available data.
Towards Dependability Metrics for Neural Networks
Venue: arXiv cs.LG 2018
Year: 2018

Artificial neural networks (NN) are instrumental in realizing highly-automated driving functionality. An overarching challenge is to identify best safety engineering practices for NN and other learning-enabled components. In particular, there is an urgent need for an adequate set of metrics for measuring all-important NN dependability attributes. We address this challenge by proposing a number of NN-specific and efficiently computable metrics for measuring NN dependability attributes including robustness, interpretability, completeness, and correctness.