mtybilly/Instruct2DS huggingface dataset:
Relation-Shape Convolutional Neural Network for Point Cloud Analysis
Authors: Bin Fan,  Shiming Xiang,  Chunhong Pan Yongcheng Liu
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Point cloud analysis is very challenging, as the shape implied in irregular points is difficult to capture. In this paper, we propose RS-CNN, namely, Relation-Shape Convolutional Neural Network, which extends regular grid CNN to irregular configuration for point cloud analysis. The key to RS-CNN is learning from relation, i.e., the geometric topology constraint among points. Specifically, the convolutional weight for local point set is forced to learn a high-level relation expression from predefined geometric priors, between a sampled point from this point set and the others. In this way, an inductive local representation with explicit reasoning about the spatial layout of points can be obtained, which leads to much shape awareness and robustness. With this convolution as a basic operator, RS-CNN, a hierarchical architecture can be developed to achieve contextual shape-aware learning for point cloud analysis. Extensive experiments on challenging benchmarks across three tasks verify RS-CNN achieves the state of the arts.
Enhancing Diversity of Defocus Blur Detectors via Cross-Ensemble Network
Authors: Bowen Zheng,  Qiuhua Lin,  Huchuan Lu Wenda Zhao
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Defocus blur detection (DBD) is a fundamental yet challenging topic, since the homogeneous region is obscure and the transition from the focused area to the unfocused region is gradual. Recent DBD methods make progress through exploring deeper or wider networks with the expense of high memory and computation. In this paper, we propose a novel learning strategy by breaking DBD problem into multiple smaller defocus blur detectors and thus estimate errors can cancel out each other. Our focus is the diversity enhancement via cross-ensemble network. Specifically, we design an end-to-end network composed of two logical parts: feature extractor network (FENet) and defocus blur detector cross-ensemble network (DBD-CENet). FENet is constructed to extract low-level features. Then the features are fed into DBD-CENet containing two parallel-branches for learning two groups of defocus blur detectors. For each individual, we design cross-negative and self-negative correlations and an error function to enhance ensemble diversity and balance individual accuracy. Finally, the multiple defocus blur detectors are combined with a uniformly weighted average to obtain the final DBD map. Experimental results indicate the superiority of our method in terms of accuracy and speed when compared with several state-of-the-art methods.
BubbleNets: Learning to Select the Guidance Frame in Video Object Segmentation by Deep Sorting Frames
Authors: Jason J. Corso Brent A. Griffin
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Semi-supervised video object segmentation has made significant progress on real and challenging videos in recent years. The current paradigm for segmentation methods and benchmark datasets is to segment objects in video provided a single annotation in the first frame. However, we find that segmentation performance across the entire video varies dramatically when selecting an alternative frame for annotation. This paper addresses the problem of learning to suggest the single best frame across the video for user annotation-this is, in fact, never the first frame of video. We achieve this by introducing BubbleNets, a novel deep sorting network that learns to select frames using a  performance-based loss function that enables the conversion of expansive amounts of training examples from already existing datasets. Using BubbleNets, we are able to achieve an 11% relative improvement in segmentation performance on the DAVIS benchmark without any changes to the underlying method of segmentation.
Collaborative Global-Local Networks for Memory-Efficient Segmentation of Ultra-High Resolution Images
Authors: Ziyu Jiang,  Zhangyang Wang,  Kexin Cui,  Xiaoning Qian Wuyang Chen
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Segmentation of ultra-high resolution images is increasingly demanded, yet poses significant challenges for algorithm efficiency, in particular considering the (GPU) memory limits. Current approaches either downsample an ultra-high resolution image, or crop it into small patches for separate processing. In either way, the loss of local fine details or global contextual information results in limited segmentation accuracy. We propose collaborative Global-Local Networks (GLNet) to effectively preserve both global and local information in a highly memory-efficient manner. GLNet is composed of a global branch and a local branch, taking the downsampled entire image and its cropped local patches as respective inputs. For segmentation, GLNet deeply fuses feature maps from two branches, capturing both the high-resolution fine structures from  zoomed-in local patches and the contextual dependency from the  downsampled input. To further resolve the potential class imbalance problem between background and foreground regions, we present a coarse-to-fine variant of GLNet, also being memory-efficient. Extensive experiments and analyses have been performed on three real-world ultra-high aerial and medical image datasets (resolution up to 30 million pixels). With only one single 1080Ti GPU and less than 2GB memory used, our GLNet yields high-quality segmentation results, and achieves much more competitive accuracy-memory usage trade-offs compared to state-of-the-arts.
Efficient Parameter-Free Clustering Using First Neighbor Relations
Authors: Vivek Sharma,  Rainer Stiefelhagen Saquib Sarfraz
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: We present a new clustering method in the form of a single clustering equation that is able to directly discover groupings in the data. The main proposition is that the first neighbor of each sample is all one needs to discover large chains and finding the groups in the data. In contrast to most existing clustering algorithms our method does not require any hyper-parameters, distance thresholds and/or the need to specify the number of clusters. The proposed algorithm belongs to the family of hierarchical agglomerative methods. The technique has a very low computational overhead, is easily scalable and applicable to large practical problems. Evaluation on well known datasets from different domains ranging between 1077 and 8.1 million samples shows substantial performance gains when compared to the existing clustering techniques.
Learning Personalized Modular Network Guided by Structured Knowledge
Authors: Xiaodan Liang
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: The dominant deep learning approaches use a "one-size-fits-all" paradigm with the hope that underlying characteristics of diverse inputs can be captured via a fixed structure. They also overlook the importance of explicitly modeling feature hierarchy. However, complex real-world tasks often require discovering diverse reasoning paths for different inputs to achieve satisfying predictions, especially for challenging large-scale recognition tasks with complex label relations. In this paper, we treat the structured commonsense knowledge (e.g. concept hierarchy) as the guidance of customizing more powerful and explainable network structures for distinct inputs, leading to dynamic and individualized inference paths. Give an off-the-shelf large network configuration, the proposed Personalized Modular Network (PMN) is learned by selectively activating a sequence of network modules where each of them is designated to recognize particular levels of structured knowledge. Learning semantic configurations and activation of modules to align well with structured knowledge can be regarded as a decision-making procedure, which is solved by a new graph-based reinforcement learning algorithm. Experiments on three semantic segmentation tasks and classification tasks show our PMN can achieve superior performance with the reduced number of network modules while discovering personalized and explainable module configurations for each input.
A Generative Appearance Model for End-To-End Video Object Segmentation
Authors: Martin Danelljan,  Emil Brissman,  Fahad Shahbaz Khan,  Michael Felsberg Joakim Johnander
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: One of the fundamental challenges in video object segmentation is to find an effective representation of the target and background appearance. The best performing approaches resort to extensive fine-tuning of a convolutional neural network for this purpose. Besides being prohibitively expensive, this strategy cannot be truly trained end-to-end since the online fine-tuning procedure is not integrated into the offline training of the network.  To address these issues, we propose a network architecture that learns a powerful representation of the target and background appearance in a single forward pass. The introduced appearance module learns a probabilistic generative model of target and background feature distributions. Given a new image, it predicts the posterior class probabilities, providing a highly discriminative cue, which is processed in later network modules. Both the learning and prediction stages of our appearance module are fully differentiable, enabling true end-to-end training of the entire segmentation pipeline. Comprehensive experiments demonstrate the effectiveness of the proposed approach on three video object segmentation benchmarks. We close the gap to approaches based on online fine-tuning on DAVIS17, while operating at 15 FPS on a single GPU. Furthermore, our method outperforms all published approaches on the large-scale YouTube-VOS dataset.
A Flexible Convolutional Solver for Fast Style Transfers
Authors: Patrick Perez Gilles Puy
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: We propose a new flexible deep convolutional neural network (convnet) to perform fast neural style transfers. Our network is trained to solve approximately, but rapidly, the artistic style transfer problem of [Gatys et al.] for arbritary styles. While solutions already exist, our network is uniquely flexible by design: it can be manipulated at runtime to enforce new constraints on the final output. As examples, we show that it can be modified to perform tasks such as fast photorealistic style transfer, or fast video style transfer with short term consistency, with no retraining. This flexibility stems from the proposed architecture which is obtained by unrolling the gradient descent algorithm used in [Gatys et al.]. Regularisations added to [Gatys et al.] to solve a new task can be reported on-the-fly in our network, even after training.
Cross Domain Model Compression by Structurally Weight Sharing
Authors: Cheng Deng,  Heng Huang Shangqian Gao
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Regular model compression methods focus on RGB input. While cross domain tasks demand more DNN models, each domain often needs its own model. Consequently, for such tasks, the storage cost, memory footprint and computation cost increase dramatically compared to single RGB input. Moreover, the distinct appearance and special structure in cross domain tasks make it difficult to directly apply regular compression methods on it. In this paper, thus, we propose a new robust cross domain model compression method. Specifically, the proposed method compress cross domain models by structurally weight sharing, which is achieved by regularizing the models with graph embedding at training time. Due to the channel wise weights sharing, the proposed method can reduce computation cost without specially designed algorithm. In the experiments, the proposed method achieves state of the art results on two diverse tasks: action recognition and RGB-D scene recognition.
TraVeLGAN: Image-To-Image Translation by Transformation Vector Learning
Authors: Smita Krishnaswamy Matthew Amodio
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Interest in image-to-image translation has grown substantially in recent years with the success of unsupervised models based on the cycle-consistency assumption. The achievements of these models have been limited to a particular subset of domains where this assumption yields good results, namely homogeneous domains that are characterized by style or texture differences. We tackle the challenging problem of image-to-image translation where the domains are defined by high-level shapes and contexts, as well as including significant clutter and heterogeneity. For this purpose, we introduce a novel GAN based on preserving intra-domain vector transformations in a latent space learned by a siamese network. The traditional GAN system introduced a discriminator network to guide the generator into generating images in the target domain. To this two-network system we add a third: a siamese network that guides the generator so that each original image shares semantics with its generated version. With this new three-network system, we no longer need to constrain the generators with the ubiquitous cycle-consistency restraint or any other autoencoding regularization. As a result, the generators can learn mappings between more complex domains that differ from each other by more than just style or texture. We demonstrate our model by mapping between high-resolution, arbitrarily chosen classes from the Imagenet dataset completely without pre-processing such as cropping, centering, or filtering unrepresentative images.
Deep Robust Subjective Visual Property Prediction in Crowdsourcing
Authors: Zhiyong Yang,  Yangbangyan Jiang,  Xiaochun Cao,  Qingming Huang,  Yuan Yao Qianqian Xu
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: The problem of estimating subjective visual properties (SVP) of images (e.g., Shoes A is more comfortable than B) is gaining rising attention. Due to its highly subjective nature, different annotators often exhibit different interpretations of scales when adopting absolute value tests. Therefore, recent investigations turn to collect pairwise comparisons via crowdsourcing platforms. However,  crowdsourcing data usually contains outliers. For this purpose, it is desired to develop a robust model for learning SVP from crowdsourced noisy annotations. In this paper, we construct a deep SVP prediction model which not only leads to better detection of annotation outliers but also enables learning with extremely sparse annotations. Specifically, we construct a comparison multi-graph based on the collected annotations, where different labeling results correspond to edges with different directions between two vertexes. Then, we propose a generalized deep probabilistic framework which consists of an SVP prediction module and an outlier modeling module that work collaboratively and are optimized jointly. Extensive experiments on various benchmark datasets demonstrate that our new approach guarantees promising results.
Transferable AutoML by Model Sharing Over Grouped Datasets
Authors: Junchi Yan,  Rong Yan,  Stephen M. Chu,  Yonggang Hu,  Yonghua Lin Chao Xue
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Automated Machine Learning (AutoML) is an active area on the design of deep neural networks for specific tasks and datasets. Given the complexity of discovering new network designs, methods for speeding up the search procedure are becoming important. This paper presents a so-called transferable AutoML approach that Automated Machine Learning (AutoML) is an active area on the design of deep neural networks for specific tasks and datasets. Given the complexity of discovering new network designs, methods for speeding up the search procedure are becoming important. This paper presents a so-called transferable AutoML approach that leverages previously trained models to speed up the search process for new tasks and datasets. Our approach involves a novel meta-feature extraction technique based on the performance of benchmark models, and a dynamic dataset clustering algorithm based on Markov process and statistical hypothesis test. As such multiple models can share a common structure while with different learned parameters. The transferable AutoML can either be applied to search from scratch, search from predesigned models, or transfer from basic cells according to the difficulties of the given datasets. The experimental results on image classification show notable speedup in overall search time for multiple datasets with negligible loss in accuracy.
Learning Not to Learn: Training Deep Neural Networks With Biased Data
Authors: Hyunwoo Kim,  Kyungsu Kim,  Sungjin Kim,  Junmo Kim Byungju Kim
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: We propose a novel regularization algorithm to train deep neural networks, in which data at training time is severely biased. Since a neural network efficiently learns data distribution, a network is likely to learn the bias information to categorize input data. It leads to poor performance at test time, if the bias is, in fact, irrelevant to the categorization. In this paper, we formulate a regularization loss based on mutual information between feature embedding and bias. Based on the idea of minimizing this mutual information, we propose an iterative algorithm to unlearn the bias information. We employ an additional network to predict the bias distribution and train the network adversarially against the feature embedding network. At the end of learning, the bias prediction network is not able to predict the bias not because it is poorly trained, but because the feature embedding network successfully unlearns the bias information. We also demonstrate quantitative and qualitative experimental results which show that our algorithm effectively removes the bias information from feature embedding.
IRLAS: Inverse Reinforcement Learning for Architecture Search
Authors: Zhao Zhong,  Wei Wu,  Dahua Lin,  Junjie Yan Minghao Guo
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: In this paper, we propose an inverse reinforcement learning method for architecture search (IRLAS), which trains an agent to learn to search network structures that are topologically inspired by human-designed network. Most existing architecture search approaches totally neglect the topological characteristics of architectures, which results in complicated architecture with a high inference latency. Motivated by the fact that human-designed networks are elegant in topology with a fast inference speed, we propose a mirror stimuli function inspired by biological cognition theory to extract the abstract topological knowledge of an expert human-design network (ResNeXt). To avoid raising a too strong prior over the search space, we introduce inverse reinforcement learning to train the mirror stimuli function and exploit it as a heuristic guidance for architecture search, easily generalized to different architecture search algorithms. On CIFAR-10, the best architecture searched by our proposed IRLAS achieves 2.60% error rate. For ImageNet mobile setting, our model achieves a state-of-the-art top-1 accuracy 75.28%, while being 2 4x faster than most auto-generated architectures. A fast version of this model achieves 10% faster than MobileNetV2, while maintaining a higher accuracy.
Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences
Authors: Paul Hongsuck Seo,  Bohyung Han Seonguk Seo
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: We propose a generic framework to calibrate accuracy and confidence of a prediction in deep neural networks through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation between predictive uncertainty of networks and variance of the prediction scores obtained by stochastic inferences for a single example. Our empirical study shows that the accuracy and the score of a prediction are highly correlated with the variance of multiple stochastic inferences given by stochastic depth or dropout. Motivated by this observation, we design a novel variance-weighted confidence-integrated loss function that is composed of two cross-entropy loss terms with respect to ground-truth and uniform distribution, which are balanced by variance of stochastic prediction scores. The proposed loss function enables us to learn deep neural networks that predict confidence calibrated scores using a single inference. Our algorithm presents outstanding confidence calibration performance and improves classification accuracy when combined with two popular stochastic regularization techniques---stochastic depth and dropout---in multiple models and datasets; it alleviates overconfidence issue in deep neural networks significantly by training networks to achieve prediction accuracy proportional to confidence of prediction.
Attention-Based Adaptive Selection of Operations for Image Restoration in the Presence of Unknown Combined Distortions
Authors: Xing Liu,  Takayuki Okatani Masanori Suganuma
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Many studies have been conducted so far on image restoration, the problem of restoring a clean image from its distorted version. There are many different types of distortion affecting image quality. Previous studies have focused on single types of distortion, proposing methods for removing them. However, image quality degrades due to multiple factors in the real world. Thus, depending on applications, e.g., vision for autonomous cars or surveillance cameras, we need to be able to deal with multiple combined distortions with unknown mixture ratios. For this purpose, we propose a simple yet effective layer architecture of neural networks. It performs multiple operations in parallel, which are weighted by an attention mechanism to enable selection of proper operations depending on the input. The layer can be stacked to form a deep network, which is differentiable and thus can be trained in an end-to-end fashion by gradient descent. The experimental results show that the proposed method works better than previous methods by a good margin on tasks of restoring images with multiple combined distortions.
Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
Authors: Meina Kan,  Shiguang Shan,  Xilin Chen Xijun Wang
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Benefitted from its great success on many tasks, deep learning is increasingly used on low-computational-cost devices, e.g. smartphone, embedded devices, etc. To reduce the high computational and memory cost, in this work, we propose a fully learnable group convolution module (FLGC for short) which is quite efficient and can be embedded into any deep neural networks for acceleration. Specifically, our proposed method automatically learns the group structure in the training stage in a fully end-to-end manner, leading to a better structure than the existing pre-defined, two-steps, or iterative strategies. Moreover, our method can be further combined with depthwise separable convolution, resulting in 5 times acceleration than the vanilla Resnet50 on single CPU. An additional advantage is that in our FLGC the number of groups can be set as any value, but not necessarily 2^k as in most existing methods, meaning better tradeoff between accuracy and speed. As evaluated in our experiments, our method achieves better performance than existing learnable group convolution and standard group convolution when using the same number of groups.
EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching From Scratch
Authors: Zhe Li,  Jianchao Yang,  Ning Xu,  Tianbao Yang,  David J. Foran Jian Ren
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Designing the structure of neural networks is considered one of the most challenging tasks in deep learning, especially when there is few prior knowledge about the task domain. In this paper, we propose an Ecologically-Inspired GENetic (EIGEN) approach that uses the concept of succession, extinction, mimicry, and gene duplication to search neural network structure from scratch with poorly initialized simple network and few constraints forced during the evolution, as we assume no prior knowledge about the task domain. Specifically, we first use primary succession to rapidly evolve a population of poorly initialized neural network structures into a more diverse population, followed by a secondary succession stage for fine-grained searching based on the networks from the primary succession. Extinction is applied in both stages to reduce computational cost. Mimicry is employed during the entire evolution process to help the inferior networks imitate the behavior of a superior network and gene duplication is utilized to duplicate the learned blocks of novel structures, both of which help to find better network structures. Experimental results show that our proposed approach can achieve similar or better performance compared to the existing genetic approaches with dramatically reduced computation cost. For example, the network discovered by our approach on CIFAR-100 dataset achieves 78.1% test accuracy under 120 GPU hours, compared to 77.0% test accuracy in more than 65, 536  GPU hours in [35].
Deep Incremental Hashing Network for Efficient Image Retrieval
Authors: Qi Dai,  Jing Liu,  Bo Li,  Weiping Wang Dayan Wu
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Hashing has shown great potential in large-scale image retrieval due to its storage and computation efficiency, especially the recent deep supervised hashing methods. To achieve promising performance, deep supervised hashing methods require a large amount of training data from different classes. However, when images of new categories emerge, existing deep hashing methods have to retrain the CNN model and generate hash codes for all the database images again, which is impractical for large-scale retrieval system. In this paper, we propose a novel deep hashing framework, called Deep Incremental Hashing Network (DIHN), for learning hash codes in an incremental manner. DIHN learns the hash codes for the new coming images directly, while keeping the old ones unchanged. Simultaneously, a deep hash function for query set is learned by preserving the similarities between training points. Extensive experiments on two widely used image retrieval benchmarks demonstrate that the proposed DIHN framework can significantly decrease the training time while keeping the state-of-the-art retrieval accuracy.
Robustness via Curvature Regularization, and Vice Versa
Authors: Alhussein Fawzi,  Jonathan Uesato,  Pascal Frossard Seyed-Mohsen Moosavi-Dezfooli
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: State-of-the-art classifiers have been shown to be largely vulnerable to adversarial perturbations. One of the most effective strategies to improve robustness is adversarial training. In this paper, we investigate the effect of adversarial training on the geometry of the classification landscape and decision boundaries. We show in particular that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs, leading to a drastically more "linear" behaviour of the network. Using a locally quadratic approximation, we provide theoretical evidence on the existence of a strong relation between large robustness and small curvature. To further show the importance of reduced curvature for improving the robustness, we propose a new regularizer that directly minimizes curvature of the loss surface, and leads to adversarial robustness that is on par with adversarial training. Besides being a more efficient and principled alternative to adversarial training, the proposed regularizer confirms our claims on the importance of exhibiting quasi-linear behavior in the vicinity of data points in order to achieve robustness.
SparseFool: A Few Pixels Make a Big Difference
Authors: Seyed-Mohsen Moosavi-Dezfooli,  Pascal Frossard Apostolos Modas
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Deep Neural Networks have achieved extraordinary results on image classification tasks, but have been shown to be vulnerable to attacks with carefully crafted perturbations of the input data. Although most attacks usually change values of many image's pixels, it has been shown that deep networks are also vulnerable to sparse alterations of the input. However, no computationally efficient method has been proposed to compute sparse perturbations. In this paper, we exploit the low mean curvature of the decision boundary, and propose SparseFool, a geometry inspired sparse attack that controls the sparsity of the perturbations. Extensive evaluations show that our approach computes sparse perturbations very fast, and scales efficiently to high dimensional data. We further analyze the transferability and the visual effects of the perturbations, and show the existence of shared semantic information across the images and the networks. Finally, we show that adversarial training can only slightly improve the robustness against sparse additive perturbations computed with SparseFool.
Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks
Authors: Jan Mathias Kohler,  Tobias Gindele,  Leon Hetzel,  Jakob Thaddaus Wiedemer,  Sven Behnke Jorg Wagner
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: To verify and validate networks, it is essential to gain insight into their decisions, limitations as well as possible shortcomings of training data. In this work, we propose a post-hoc, optimization based visual explanation method, which highlights the evidence in the input image for a specific prediction. Our approach is based on a novel technique to defend against adversarial evidence (i.e. faulty evidence due to artefacts) by filtering gradients during optimization. The defense does not depend on human-tuned parameters. It enables explanations which are both fine-grained and preserve the characteristics of images, such as edges and colors. The explanations are interpretable, suited for visualizing detailed evidence and can be tested as they are valid model inputs. We qualitatively and quantitatively evaluate our approach on a multitude of models and datasets.
Structured Pruning of Neural Networks With Budget-Aware Regularization
Authors: Andrew Achkar,  Pierre-Marc Jodoin Carl Lemaire
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Pruning methods have shown to be effective at reducing the size of deep neural networks while keeping accuracy almost intact. Among the most effective methods are those that prune a network while training it with a sparsity prior loss and learnable dropout parameters. A shortcoming of these approaches however is that neither the size nor the inference speed of the pruned network can be controlled directly; yet this is a key feature for targeting deployment of CNNs on low-power hardware. To overcome this, we introduce a budgeted regularized pruning framework for deep CNNs. Our approach naturally fits into traditional neural network training as it consists of a learnable masking layer, a novel budget-aware objective function, and the use of knowledge distillation. We also provide insights on how to prune a residual network and how this can lead to new architectures. Experimental results reveal that CNNs pruned with our method are more accurate and less compute-hungry than state-of-the-art methods. Also, our approach is more effective at preventing accuracy collapse in case of severe pruning; this allows pruning factors of up to 16x without significant accuracy drop.
MBS: Macroblock Scaling for CNN Model Reduction
Authors: Chun-Nan Chou,  Edward Y. Chang Yu-Hsun Lin
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: In this paper we propose the macroblock scaling (MBS) algorithm, which can be applied to various CNN architectures to reduce their model size. MBS adaptively reduces each CNN macroblock depending on its information redundancy measured by our proposed  effective flops. Empirical studies conducted with ImageNet and CIFAR-10 attest that MBS can reduce the model size of some already compact CNN models, e.g., MobileNetV2 (25.03% further reduction) and ShuffleNet (20.74%), and even ultra-deep ones such as ResNet-101 (51.67%) and ResNet-1202 (72.71%) with negligible accuracy degradation. MBS also performs better reduction at a much lower cost than the state-of-the-art optimization-based methods do. MBS's simplicity and efficiency, its flexibility to work with any CNN model, and its scalability to work with models of any depth make it an attractive choice for CNN model size reduction.
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells
Authors: Hao Chen,  Chunhua Shen,  Ian Reid Vladimir Nekrasov
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Automated design of neural network architectures tailored for a specific task is an extremely promising, albeit inherently difficult, avenue to explore. While most results in this domain have been achieved on image classification and language modelling problems, here we concentrate on dense per-pixel tasks, in particular, semantic image segmentation using fully convolutional networks. In contrast to the aforementioned areas, the design choices of a fully convolutional network require several changes, ranging from the sort of operations that need to be used - e.g., dilated convolutions - to a solving of a more difficult optimisation problem. In this work, we are particularly interested in searching for high-performance compact segmentation architectures, able to run in real-time using limited resources. To achieve that, we intentionally over-parameterise the architecture during the training time via a set of auxiliary cells that provide an intermediate supervisory signal and can be omitted during the evaluation phase.  The design of the auxiliary cell is emitted by a controller, a neural network with the fixed structure trained using reinforcement learning. More crucially, we demonstrate how to efficiently search for these architectures within limited time and computational budgets. In particular, we rely on a progressive strategy that terminates non-promising architectures from being further trained, and on Polyak averaging coupled with knowledge distillation to speed-up the convergence. Quantitatively, in 8 GPU-days our approach discovers a set of architectures performing on-par with state-of-the-art among compact models on the semantic segmentation, pose estimation and depth prediction tasks. Code will be made available here: https://github.com/drsleep/nas-segm-pytorch
Generating 3D Adversarial Point Clouds
Authors: Charles R. Qi,  Bo Li Chong Xiang
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Deep neural networks are known to be vulnerable to adversarial examples which are carefully crafted instances to cause the models to make wrong predictions. While adversarial examples for 2D images and CNNs have been extensively studied, less attention has been paid to 3D data such as point clouds. Given many safety-critical 3D applications such as autonomous driving, it is important to study how adversarial point clouds could affect current deep 3D models. In this work, we propose several novel algorithms to craft adversarial point clouds against PointNet, a widely used deep neural network for point cloud processing. Our algorithms work in two ways: adversarial point perturbation and adversarial point generation. For point perturbation, we shift existing points negligibly. For point generation, we generate either a set of independent and scattered points or a small number (1-3) of point clusters with meaningful shapes such as balls and airplanes which could be hidden in the human psyche. In addition, we formulate six perturbation measurement metrics tailored to the attacks in point clouds and conduct extensive experiments to evaluate the proposed algorithms on the ModelNet40 3D shape classification dataset. Overall, our attack algorithms achieve a success rate higher than 99% for all targeted attacks.
Partial Order Pruning: For Best Speed/Accuracy Trade-Off in Neural Architecture Search
Authors: Yiming Zhou,  Zheng Pan,  Jiashi Feng Xin Li
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Achieving good speed and accuracy trade-off on a target platform is very important in deploying deep neural networks in real world scenarios. However, most existing automatic architecture search approaches only concentrate on high performance. In this work, we propose an algorithm that can offer better speed/accuracy trade-off of searched networks, which is termed "Partial Order Pruning". It prunes the architecture search space with a partial order assumption to automatically search for the architectures with the best speed and accuracy trade-off. Our algorithm explicitly takes profile information about the inference speed on the target platform into consideration. With the proposed algorithm, we present several Dongfeng (DF) networks that provide high accuracy and fast inference speed on various application GPU platforms. By further searching decoder architectures, our DF-Seg real-time segmentation networks yield state-of-the-art speed/accuracy trade-off on both the  target embedded device  and the high-end GPU.
Memory in Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity From Spatiotemporal Dynamics
Authors: Jianjin Zhang,  Hongyu Zhu,  Mingsheng Long,  Jianmin Wang,  Philip S. Yu Yunbo Wang
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term. By applying differencing operations appropriately, we may turn time-variant polynomials into a constant, making the deterministic component predictable.   However, most previous recurrent neural networks for spatiotemporal prediction do not use the differential signals effectively, and their relatively simple state transition functions prevent them from learning too complicated variations in spacetime. We propose the Memory In Memory (MIM) networks and corresponding recurrent blocks for this purpose. The MIM blocks exploit the differential signals between adjacent recurrent states to model the non-stationary and approximately stationary properties in spatiotemporal dynamics with two cascaded, self-renewed memory modules. By stacking multiple MIM blocks, we could potentially handle higher-order non-stationarity. The MIM networks achieve the state-of-the-art results on four spatiotemporal prediction tasks across both synthetic and real-world datasets. We believe that the general idea of this work can be potentially applied to other time-series forecasting tasks.
Variational Information Distillation for Knowledge Transfer
Authors: Shell Xu Hu,  Andreas Damianou,  Neil D. Lawrence,  Zhenwen Dai Sungsoo Ahn
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs
Authors: Yang Li,  Samy Bengio,  Si Si Zhourong Chen
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: The concept of conditional computation for deep nets has been proposed previously to improve model performance by selectively using only parts of the model conditioned on the sample it is processing. In this paper, we investigate input-dependent dynamic filter selection in deep convolutional neural networks (CNNs). The problem is interesting because the idea of forcing different parts of the model to learn from different types of samples may help us acquire better filters in CNNs, improve the model generalization performance and potentially increase the interpretability of model behavior. We propose a novel yet simple framework called GaterNet, which involves a backbone and a gater network. The backbone network is a regular CNN that performs the major computation needed for making a prediction, while a global gater network is introduced to generate binary gates for selectively activating filters in the backbone network based on each input. Extensive experiments on CIFAR and ImageNet datasets show that our models consistently outperform the original models with a large margin. On CIFAR-10, our model also improves upon state-of-the-art results.
SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360deg Images
Authors: Jaeseok Jeong,  Jongseob Yun,  Wonjune Cho,  Kuk-Jin Yoon Yeonkun Lee
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: Omni-directional  cameras  have  many  advantages  overconventional cameras in that they have a much wider field-of-view (FOV). Accordingly, several approaches have beenproposed recently to apply convolutional neural networks(CNNs) to omni-directional images for various visual tasks.However, most of them use image representations defined inthe Euclidean space after transforming the omni-directionalviews originally formed in the non-Euclidean space.  Thistransformation leads to shape distortion due to nonuniformspatial resolving power and the loss of continuity.   Theseeffects make existing convolution kernels experience diffi-culties in extracting meaningful information.     This paper presents a novel method to resolve such prob-lems  of  applying  CNNs  to  omni-directional  images.   Theproposed  method  utilizes  a  spherical  polyhedron  to  rep-resent omni-directional views.  This method minimizes thevariance of the spatial resolving power on the sphere sur-face,  and  includes  new  convolution  and  pooling  methodsfor the proposed representation.  The proposed method canalso be adopted by any existing CNN-based methods.  Thefeasibility of the proposed method is demonstrated throughclassification,  detection,  and semantic segmentation taskswith synthetic and real datasets.
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Authors: Mohammad Rastegari,  Linda Shapiro,  Hannaneh Hajishirzi Sachin Mehta
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: We introduce a light-weight, power efficient, and general purpose convolutional neural network, ESPNetv2, for modeling visual and sequential data. Our network uses group point-wise and depth-wise dilated separable convolutions to learn representations from a large effective receptive field with fewer FLOPs and parameters. The performance of our network is evaluated on four different tasks: (1) object classification, (2) semantic segmentation, (3) object detection, and (4) language modeling. Experiments on these tasks, including image classification on the ImageNet and language modeling on the PenTree bank dataset, demonstrate the superior performance of our method over the state-of-the-art methods. Our network outperforms ESPNet by 4-5% and has 2-4x fewer FLOPs on the PASCAL VOC and the Cityscapes dataset. Compared to YOLOv2 on the MS-COCO object detection, ESPNetv2 delivers 4.4% higher accuracy with 6x fewer FLOPs. Our experiments show that ESPNetv2 is much more power efficient than existing state-of-the-art efficient methods including ShuffleNets and MobileNets. Our code is open-source and available at https://github.com/sacmehta/ESPNetv2.
Assisted Excitation of Activations: A Learning Technique to Improve Object Detectors
Authors: Saeed Masoudnia,  Amir Hossein Shaker,  Omid Mersa,  Mohammad Amin Sadeghi,  Mohammad Rastegari,  Babak N. Araabi Mohammad Mahdi Derakhshani
Published in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (Main Conference Track), 2019

Abstract: We present a simple yet effective learning technique that significantly improves mAP of YOLO object detectors without compromising their speed. During network training, we carefully feed in localization information. We excite certain activations in order to help the network learn to better localize (Figure 2). In the later stages of training, we gradually reduce our assisted excitation to zero. We reached a new state-of-the-art in the speed-accuracy trade-off (Figure 1). Our technique improves the mAP of YOLOv2 by 3.8% and mAP of YOLOv3 by 2.2% on MSCOCO dataset. This technique is inspired from curriculum learning. It is simple and effective and it is applicable to most single-stage object detectors.