mtybilly/Instruct2DS huggingface dataset:
Divide and Grow: Capturing Huge Diversity in Crowd Images With Incrementally Growing CNN
Authors: Neeraj N. Sajjan, R. Venkatesh Babu, Mukundhan Srinivasan Deepak Babu Sam
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Automated counting of people in crowd images is a challenging task. The major difficulty stems from the large diversity in the way people appear in crowds. In fact, features available for crowd discrimination largely depend on the crowd density to the extent that people are only seen as blobs in a highly dense scene. We tackle this problem with a growing CNN which can progressively increase its capacity to account for the wide variability seen in crowd scenes. Our model starts from a base CNN density regressor, which is trained in equivalence on all types of crowd images. In order to adapt with the huge diversity, we create two child regressors which are exact copies of the base CNN. A differential training procedure divides the dataset into two clusters and fine-tunes the child networks on their respective specialties. Consequently, without any hand-crafted criteria for forming specialties, the child regressors become experts on certain types of crowds. The child networks are again split recursively, creating two experts at every division. This hierarchical training leads to a CNN tree, where the child regressors are more fine experts than any of their parents. The leaf nodes are taken as the final experts and a classifier network is then trained to predict the correct specialty for a given test image patch. The proposed model achieves higher count accuracy on major crowd datasets. Further, we analyse the characteristics of specialties mined automatically by our method.
Structured Set Matching Networks for One-Shot Part Labeling
Authors: Jayant Krishnamurthy, Aniruddha Kembhavi, Ali Farhadi Jonghyun Choi
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Diagrams often depict complex phenomena and serve as a good test bed for visual and textual reasoning. However, understanding diagrams using natural image understanding approaches requires large training datasets of diagrams, which are very hard to obtain. Instead, this can be addressed as a matching problem either between labeled diagrams, images or both. This problem is very challenging since the absence of significant color and texture renders local cues ambiguous and requires global reasoning. We consider the problem of one-shot part labeling: labeling multiple parts of an object in a target image given only a single source image of that category. For this set-to-set matching problem, we introduce the Structured Set Matching Network (SSMN), a structured prediction model that incorporates convolutional neural networks. The SSMN is trained using global normalization to maximize local match scores between corresponding elements and a global consistency score among all matched elements, while also enforcing a matching constraint between the two sets. The SSMN significantly outperforms several strong baselines on three label transfer scenarios: diagram-to-diagram, evaluated on a new diagram dataset of over 200 categories; image-to-image, evaluated on a dataset built on top of the Pascal Part Dataset; and image-to-diagram, evaluated on transferring labels across these datasets.
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection
Authors: Samuel Albanie, Diane Larlus, Andrea Vedaldi David Novotny
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Self-supervision can dramatically cut back the amount of manually-labelled data required to train deep neural networks. While self-supervision has usually been considered for tasks such as image classification, in this paper we aim at extending it to geometry-oriented tasks such as semantic matching and part detection. We do so by building on several recent ideas in unsupervised landmark detection. Our approach learns dense distinctive visual descriptors from an unlabeled dataset of images using synthetic image transformations. It does so by means of a robust probabilistic formulation that can introspectively determine which image regions are likely to result in stable image matching. We show empirically that a network pre-trained in this manner requires significantly less supervision to learn semantic object parts compared to numerous pre-training alternatives. We also show that the pre-trained representation is excellent for semantic object matching.
Link and Code: Fast Indexing With Graphs and Compact Regression Codes
Authors: Alexandre Sablayrolles, Hervé Jégou Matthijs Douze
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Similarity search approaches based on graph walks have recently attained outstanding speed-accuracy trade-offs, taking aside the memory requirements. In this paper, we revisit these approaches by considering, additionally, the memory constraint required to index billions of images on a single server. This leads us to propose a method based both on graph traversal and compact representations. We encode the indexed vectors using quantization and exploit the graph structure to refine the similarity estimation.   In essence, our method takes the best of these two worlds: the search strategy is based on nested graphs, thereby providing high precision with a relatively small set of comparisons. At the same time it offers a significant memory compression. As a result, our approach outperforms the state of the art on operating points considering 64--128 bytes per vector, as demonstrated by our results on two billion-scale public benchmarks.
Textbook Question Answering Under Instructor Guidance With Memory Networks
Authors: Hang Su, Jun Zhu, Siyu Wang, Bo Zhang Juzheng Li
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Textbook Question Answering (TQA) is a task to choose the most proper answers by reading a multi-modal context of abundant essays and images. TQA serves as a favorable test bed for visual and textual reasoning. However, most of the current methods are incapable of reasoning over the long contexts and images. To address this issue, we propose a novel approach of Instructor Guidance with Memory Networks (IGMN) which conducts the TQA task by finding contradictions between the candidate answers and their corresponding context. We build the Contradiction Entity-Relationship Graph (CERG) to extend the passage-level multi-modal contradictions to an essay level. The machine thus performs as an instructor to extract the essay-level contradictions as the Guidance. Afterwards, we exploit the memory networks to capture the information in the Guidance, and use the attention mechanisms to jointly reason over the global features of the multi-modal input. Extensive experiments demonstrate that our method outperforms the state-of-the-arts on the TQA dataset. The source code is available at https://github.com/freerailway/igmn.
Unsupervised Deep Generative Adversarial Hashing Network
Authors: Feng Zheng, Najmeh Sadoughi, Yanhua Yang, Cheng Deng, Heng Huang Kamran Ghasedi Dizaji
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Unsupervised deep hash functions have not shown satisfactory improvements against the shallow alternatives, and usually, require supervised pretraining to avoid getting stuck in bad local minima. In this paper, we propose a deep unsupervised hashing function, called HashGAN, which outperforms unsupervised hashing models with significant margins without any supervised pretraining. HashGAN consists of three networks, a generator, a discriminator and an encoder. By sharing the parameters of the encoder and discriminator, we benefit from the adversarial loss as a data dependent regularization in training our deep hash function. Moreover, a novel loss function is introduced for hashing real images, resulting in minimum entropy, uniform frequency, consistent and independent hash bits. Furthermore, we train the generator conditioning on random binary inputs and also use these binary variables in a triplet ranking loss for improving hash codes. In our experiments, HashGAN outperforms the previous unsupervised hash functions in image retrieval and achieves the state-of-the-art performance in image clustering. We also provide an ablation study, showing the contribution of each component in our loss function.
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments
Authors: Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton van den Hengel Peter Anderson
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers.  It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering.  Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matterport3D Simulator -- a large-scale reinforcement learning environment based on real imagery. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -- the Room-to-Room (R2R) dataset.
DenseASPP for Semantic Segmentation in Street Scenes
Authors: Kun Yu, Chi Zhang, Zhiwei Li, Kuiyuan Yang Maoke Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Semantic image segmentation is a basic street scene understanding task in autonomous driving, where each pixel in a high resolution image is categorized into a set of semantic labels. Unlike other scenarios, objects in autonomous driving scene exhibit very large scale changes, which poses great challenges for high-level feature representation in a sense that multi-scale information must be correctly encoded. To remedy this problem, atrous convolutioncite{Deeplabv1} was introduced to generate features with larger receptive fields without sacrificing spatial resolution. Built upon atrous convolution, Atrous Spatial Pyramid Pooling (ASPP)cite{Deeplabv2} was proposed to concatenate multiple atrous-convolved features using different dilation rates into a final feature representation. Although ASPP is able to generate multi-scale features, we argue the feature resolution in the scale-axis is not dense enough for the autonomous driving scenario. To this end, we propose Densely connected Atrous Spatial Pyramid Pooling (DenseASPP), which connects a set of atrous convolutional layers in a dense way, such that it generates multi-scale features that not only cover a larger scale range, but also cover that scale range densely, without significantly increasing the model size. We evaluate DenseASPP on the street scene benchmark Cityscapescite{Cityscapes} and achieve state-of-the-art performance.
Efficient Optimization for Rank-Based Loss Functions
Authors: Michal Rolínek, C.V. Jawahar, Vladimir Kolmogorov, M. Pawan Kumar Pritish Mohapatra
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: The accuracy of information retrieval systems is often measured using complex loss functions such as the average precision (AP) or the normalized discounted cumulative gain (NDCG). Given a set of positive and negative samples, the parameters of a retrieval system can be estimated by minimizing these loss functions. However, the non-differentiability and non-decomposability of these loss functions does not allow for simple gradient based optimization algorithms. This issue is generally circumvented by either optimizing a structured hinge-loss upper bound to the loss function or by using asymptotic methods like the direct-loss minimization framework. Yet, the high computational complexity of loss-augmented inference, which is necessary for both the frameworks, prohibits its use in large training data sets. To alleviate this deficiency, we present a novel quicksort flavored algorithm for a large class of non-decomposable loss functions. We provide a complete characterization of the loss functions that are amenable to our algorithm, and show that it includes both AP and NDCG based loss functions. Furthermore, we prove that no comparison based algorithm can improve upon the computational complexity of our approach asymptotically. We demonstrate the effectiveness of our approach in the context of optimizing the structured hinge loss upper bound of AP and NDCG loss for learning models for a variety of vision tasks. We show that our approach provides significantly better results than simpler decomposable loss functions, while requiring a comparable training time.
Wasserstein Introspective Neural Networks
Authors: Weijian Xu, Fan Fan, Zhuowen Tu Kwonjoon Lee
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We present Wasserstein introspective neural networks (WINN) that are both a generator and a discriminator within a single model. WINN provides a significant improvement over the recent introspective neural networks (INN) method by enhancing INN's generative modeling capability. WINN has three interesting properties: (1) A mathematical connection between the formulation of the INN algorithm and that of Wasserstein generative adversarial networks (WGAN) is made. (2) The explicit adoption of the Wasserstein distance into INN results in a large enhancement to INN, achieving compelling results even with a single classifier --- e.g., providing nearly a 20 times reduction in model size over INN for unsupervised generative modeling. (3) When applied to supervised classification, WINN also gives rise to improved robustness against adversarial examples in terms of the error reduction. In the experiments, we report encouraging results on unsupervised learning problems including texture, face, and object modeling, as well as a supervised classification task against adversarial attacks.
Taskonomy: Disentangling Task Transfer Learning
Authors: Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese Amir R. Zamir
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable uses; it is the concept underlying transfer learning and, for example, can provide a principled way for reusing supervision among related tasks, finding what tasks transfer well to an arbitrary target task, or solving many tasks in one system without piling up the complexity.      This paper proposes a fully computational approach for finding the structure of the space of visual tasks. This is done via a sampled dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks, and modeling their (1st and higher order) transfer dependencies in a latent space. The product can be viewed as a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. the nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 while keeping the performance nearly the same. Users can employ a provided Binary Integer Programming solver that leverages the taxonomy to find efficient supervision policies for their own use cases.
Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
Authors: Kohei Watanabe, Yoshitaka Ushiku, Tatsuya Harada Kuniaki Saito
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: In this work, we present a method for unsupervised domain adaptation.  Many adversarial learning methods train domain classifier networks to distinguish the features as either a source or target and train a feature generator network to mimic the discriminator. Two problems exist with these methods. First, the domain classifier only tries to distinguish the features as a source or target and thus does not consider task-specific decision boundaries between classes. Therefore, a trained generator can generate ambiguous features near class boundaries. Second, these methods aim to completely match the feature distributions between different domains, which is difficult because of each domain's characteristics.  To solve these problems, we introduce a new approach that attempts to align distributions of source and target by utilizing the task-specific decision boundaries.  We propose to maximize the discrepancy between two classifiers' outputs to detect target samples that are far from the support of the source. A feature generator learns to generate target features near the support to minimize the discrepancy.  Our method outperforms other methods on several datasets of image classification and semantic segmentation. The codes are available at url{https://github.com/mil-tokyo/MCD_DA}
Unsupervised Feature Learning via Non-Parametric Instance Discrimination
Authors: Yuanjun Xiong, Stella X. Yu, Dahua Lin Zhirong Wu
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Neural net classifiers trained on data with annotated class labels can also capture apparent visual similarity among categories without being directed to do so. We study whether this observation can be extended beyond the conventional domain of supervised learning: Can we learn a good feature representation that captures apparent similarity among instances, instead of classes, by merely asking the feature to be discriminative of individual instances? We formulate this intuition as a non-parametric classification problem at the instance-level, and use noise-contrastive estimation to tackle the computational challenges imposed by the large number of instance classes. Our experimental results demonstrate that, under unsu- pervised learning settings, our method surpasses the state-of-the-art on ImageNet classification by a large margin. Our method is also remarkable for consistently improving test performance with more training data and better network architectures. By fine-tuning the learned feature, we further obtain competitive results for semi-supervised learning and object detection tasks. Our non-parametric model is highly compact: With 128 features per image, our method requires only 600MB storage for a million images, enabling fast nearest neighbour retrieval at the run time.
Multi-Task Adversarial Network for Disentangled Feature Learning
Authors: Zhaowen Wang, Hailin Jin, Ian Wassell Yang Liu
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We address the problem of image feature learning for the applications where multiple factors exist in the image generation process and only some factors are of our interest. We present a novel multi-task adversarial network based on an encoder-discriminator-generator architecture. The encoder extracts a disentangled feature representation for the factors of interest. The discriminators classify each of the factors as individual tasks. The encoder and the discriminators are trained cooperatively on factors of interest, but in an adversarial way on factors of distraction. The generator provides further regularization on the learned feature by reconstructing images with shared factors as the input image. We design a new optimization scheme to stabilize the adversarial optimization process when multiple distributions need to be aligned. The experiments on face recognition and font recognition tasks show that our method outperforms the state-of-the-art methods in terms of both recognizing the factors of interest and generalization to images with unseen variations.
Learning From Synthetic Data: Addressing Domain Shift for Semantic Segmentation
Authors: Yogesh Balaji, Arpit Jain, Ser Nam Lim, Rama Chellappa Swami Sankaranarayanan
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Visual Domain Adaptation is a problem of immense importance in computer vision. Previous approaches showcase the inability of even deep neural networks to learn informative representations across domain shift. This problem is more severe for tasks where acquiring hand labeled data is extremely hard and tedious. In this work, we focus on adapting the representations learned by segmentation networks across synthetic and real domains. Contrary to previous approaches that use a simple adversarial objective or superpixel information to aid the process, we propose an approach based on Generative Adversarial Networks (GANs) that brings the embeddings closer in the learned feature space. To showcase the generality and scalability of our approach, we show that we can achieve state of the art results on two challenging scenarios of synthetic to real domain adaptation. Additional exploratory experiments show that our approach: (1) generalizes to unseen domains and (2) results in improved alignment of source and target distributions.
Empirical Study of the Topology and Geometry of Deep Networks
Authors: Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, Stefano Soatto Alhussein Fawzi
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: The goal of this paper is to analyze the geometric properties of deep neural network image classifiers in the input space. We specifically study the topology of classification regions created by deep networks, as well as their associated decision boundary. Through a systematic empirical study, we show that state-of-the-art deep nets learn connected classification regions, and that the decision boundary in the vicinity of datapoints is flat along most directions. We further draw an essential connection between two seemingly unrelated properties of deep networks: their sensitivity to additive perturbations of the inputs, and the curvature of their decision boundary. The directions where the decision boundary is curved in fact characterize the directions to which the classifier is the most vulnerable. We finally leverage a fundamental asymmetry in the curvature of the decision boundary of deep nets, and propose a method to discriminate between original images, and images perturbed with small adversarial examples. We show the effectiveness of this purely geometric approach for detecting small adversarial perturbations in images, and for recovering the labels of perturbed images.
Boosting Domain Adaptation by Discovering Latent Domains
Authors: Lorenzo Porzi, Samuel Rota Bulò, Barbara Caputo, Elisa Ricci Massimiliano Mancini
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Current Domain Adaptation (DA) methods based on deep architectures assume that the source samples arise from a single distribution. However, in practice most datasets can be regarded as mixtures of multiple domains. In these cases exploiting single-source DA methods for learning target classifiers may lead to sub-optimal, if not poor, results. In addition, in many applications it is difficult to manually provide the domain labels for all source data points, i.e. latent domains should be automatically discovered. This paper introduces a novel Convolutional Neural Network (CNN) architecture which (i) automatically discovers latent domains in visual datasets and (ii) exploits this information to learn robust target classifiers. Our approach is based on the introduction of two main components, which can be embedded into any existing CNN architecture: (i) a side branch that automatically computes the assignment of a source sample to a latent domain and (ii) novel layers that exploit domain membership information to appropriately align the distribution of the CNN internal feature representations to a reference distribution. We test our approach on publicly-available datasets, showing that it outperforms state-of-the-art multi-source DA methods by a large margin.
Shape From Shading Through Shape Evolution
Authors: Jia Deng Dawei Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: In this paper, we address the shape-from-shading problem by training deep networks with synthetic images. Unlike conventional approaches that combine deep learning and synthetic imagery, we propose an approach that does not need any external shape dataset to render synthetic images. Our approach consists of two synergistic processes: the evolution of complex shapes from simple primitives, and the training of a deep network for shape-from-shading. The evolution generates better shapes guided by the network training, while the training improves by using the evolved shapes. We show that our approach achieves state-of-the-art performance on a shape-from-shading benchmark.
Weakly Supervised Instance Segmentation Using Class Peak Response
Authors: Yi Zhu, Qixiang Ye, Qiang Qiu, Jianbin Jiao Yanzhao Zhou
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Weakly supervised instance segmentation with image-level labels, instead of expensive pixel-level masks, remains unexplored. In this paper, we tackle this challenging problem by exploiting class peak responses to enable a classification network for instance mask extraction. With image labels supervision only, CNN classifiers in a fully convolutional manner can produce class response maps, which specify classification confidence at each image location. We observed that local maximums, i.e., peaks, in a class response map typically correspond to strong visual cues residing inside each instance. Motivated by this, we first design a process to stimulate peaks to emerge from a class response map. The emerged peaks are then back-propagated and effectively mapped to highly informative regions of each object instance, such as instance boundaries. We refer to the above maps generated from class peak responses as Peak Response Maps (PRMs). PRMs provide a fine-detailed instance-level representation, which allows instance masks to be extracted even with some off-the-shelf methods. To the best of our knowledge, we for the first time report results for the challenging image-level supervised instance segmentation task. Extensive experiments show that our method also boosts weakly supervised pointwise localization as well as semantic segmentation performance, and reports state-of-the-art results on popular benchmarks, including PASCAL VOC 2012 and MS COCO.
Collaborative and Adversarial Network for Unsupervised Domain Adaptation
Authors: Wanli Ouyang, Wen Li, Dong Xu Weichen Zhang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: In this paper, we propose a new unsupervised domain adaptation approach called Collaborative and Adversarial Network (CAN) through domain-collaborative and domain-adversarial training of neural networks. We use several domain classifiers on multiple CNN feature extraction layers/blocks, in which each domain classifier is connected to the hidden representations from one block and one loss function is defined based on the hidden presentation and the domain labels (e.g., source and target). We design a new loss function by integrating the losses from all blocks in order to learn informative representations from lower layers through collaborative learning and learn uninformative representations from higher layers through adversarial learning. We further extend our CAN method as Incremental CAN (iCAN), in which we iteratively select a set of pseudo-labelled target samples based on the image classifier and the last domain classifier from the previous training epoch and re-train our CAN model using the enlarged training set.  Comprehensive experiments on two benchmark datasets Office and ImageCLEF-DA clearly demonstrate the effectiveness of our newly proposed approaches CAN and iCAN for unsupervised domain adaptation.
Environment Upgrade Reinforcement Learning for Non-Differentiable Multi-Stage Pipelines
Authors: Zitian Chen, Chao Xu, Cewu Lu Shuqin Xie
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Recent advances in multi-stage algorithms have shown great promise, but two important problems still remain. First of all, at inference time, information can't feed back from downstream to upstream. Second, at training time, end-to-end training is not possible if the overall pipeline involves non-differentiable functions, and so different stages can't be jointly optimized. In this paper, we propose a novel environment upgrade reinforcement learning framework to solve the feedback and joint optimization problems. Our framework re-links the downstream stage to the upstream stage by a reinforcement learning agent. While training the agent to improve final performance by refining the upstream stage's output, we also upgrade the downstream stage (environment) according to the agent's policy. In this way, agent policy and environment are jointly optimized. We propose a training algorithm for this framework to address the different training demands of agent and environment. Experiments on instance segmentation and human pose estimation demonstrate the effectiveness of the proposed framework.
Teaching Categories to Human Learners With Visual Explanations
Authors: Shihan Su, Yuxin Chen, Pietro Perona, Yisong Yue Oisin Mac Aodha
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We study the problem of computer-assisted teaching with explanations.   Conventional approaches for machine teaching typically only provide feedback at the instance level e.g., the category or label of the instance.   However, it is intuitive that clear explanations from a knowledgeable teacher can significantly improve a student's ability to learn a new concept.  To address these existing limitations, we propose a teaching framework that provides interpretable explanations as feedback and models how the learner incorporates this additional information.   In the case of images, we show that we can automatically generate explanations that highlight the parts of the image that are responsible for the class label. Experiments on human learners illustrate that, on average, participants achieve better test set performance on challenging categorization tasks when taught with our interpretable approach compared to existing methods.
Density Adaptive Point Set Registration
Authors: Martin Danelljan, Fahad Shahbaz Khan, Per-Erik Forssén, Michael Felsberg Felix Järemo Lawin
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Probabilistic methods for point set registration have demonstrated competitive results in recent years. These techniques estimate a probability distribution model of the point clouds. While such a representation has shown promise, it is highly sensitive to variations in the density of 3D points. This fundamental problem is primarily caused by changes in the sensor location across point sets. We revisit the foundations of the probabilistic registration paradigm. Contrary to previous works, we model the underlying structure of the scene as a latent probability distribution, and thereby induce invariance to point set density changes. Both the probabilistic model of the scene and the registration parameters are inferred by minimizing the Kullback-Leibler divergence in an Expectation Maximization based framework. Our density-adaptive registration successfully handles severe density variations commonly encountered in terrestrial Lidar applications. We perform extensive experiments on several challenging real-world Lidar datasets. The results demonstrate that our approach outperforms state-of-the-art probabilistic methods for multi-view registration, without the need of re-sampling.
Left-Right Comparative Recurrent Model for Stereo Matching
Authors: Pengfei Wang, Yonggen Ling, Bo Zhao, Yunchao Wei, Jiashi Feng, Wei Liu Zequn Jie
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Leveraging the disparity information from both  left and right views is crucial for stereo disparity estimation. Left-right consistency check is an effective way to enhance the disparity estimation by referring to the information from the opposite view. However, the conventional left-right consistency check is an isolated post-processing step and heavily hand-crafted. This paper proposes a novel left-right comparative recurrent model to perform left-right consistency checking jointly with   disparity estimation. At each recurrent step, the model produces disparity results for both views, and then performs online left-right comparison to identify the mismatched regions which may probably contain erroneously labeled pixels. A soft attention mechanism is introduced, which employs the learned error maps for better guiding the model to selectively focus on refining the unreliable regions at the next recurrent step. In this way, the generated disparity maps are progressively improved by the proposed recurrent model. Extensive evaluations on  KITTI 2015, Scene Flow and Middlebury benchmarks validate the effectiveness of our model,  demonstrating that state-of-the-art stereo disparity estimation results can be achieved by this new model.
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
Authors: Andy Zeng, Angel X. Chang, Manolis Savva, Silvio Savarese, Thomas Funkhouser Shuran Song
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image. To make this possible, Im2Pano3D leverages strong contextual priors learned from large-scale synthetic and real-world indoor scenes. To ease the prediction of 3D structure, we propose to parameterize 3D surfaces with their plane equations and train the model to predict these parameters directly. To provide meaningful training supervision, we make use of multiple loss functions that consider both pixel level accuracy and global context consistency. Experiments demonstrate that Im2Pano3D is able to predict the semantics and 3D structure of the unobserved scene with more than 56% pixel accuracy and less than 0.52m average distance error, which is significantly better than alternative approaches.
Polarimetric Dense Monocular SLAM
Authors: Feitong Tan, Ao Li, Zhaopeng Cui, Yasutaka Furukawa, Ping Tan Luwei Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: This paper presents a novel polarimetric dense monocular SLAM (PDMS) algorithm based on a polarization camera. The algorithm exploits both photometric and polarimetric light information to produce more accurate and complete geometry. The polarimetric information allows us to recover the azimuth angle of surface normals from each video frame to facilitate dense reconstruction, especially at textureless or specular regions. There are two challenges in our approach: 1) surface azimuth angles from the polarization camera are very noisy; and 2) we need a near real-time solution for SLAM. Previous successful methods on polarimetric multi-view stereo are offline and require manually pre-segmented object masks to suppress the effects of erroneous angle information along boundaries. Our fully automatic approach efficiently iterates azimuth-based depth propagations, two-view depth consistency check, and depth optimization to produce a depthmap in real-time, where all the algorithmic steps are carefully designed to enable a GPU implementation. To our knowledge, this paper is the first to propose a photometric and polarimetric method for dense SLAM. We have qualitatively and quantitatively evaluated our algorithm against a few of competing methods, demonstrating the superior performance on various indoor and outdoor scenes.
A Unifying Contrast Maximization Framework for Event Cameras, With Applications to Motion, Depth, and Optical Flow Estimation
Authors: Henri Rebecq, Davide Scaramuzza Guillermo Gallego
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We present a unifying framework to solve several computer vision problems with event cameras: motion, depth and optical flow estimation. The main idea of our framework is to find the point trajectories on the image plane that are best aligned with the event data by maximizing an objective function: the contrast of an image of warped events. Our method implicitly handles data association between the events, and therefore, does not rely on additional appearance information about the scene. In addition to accurately recovering the motion parameters of the problem, our framework produces motion-corrected edge-like images with high dynamic range that can be used for further scene analysis. The proposed method is not only simple, but more importantly, it is, to the best of our knowledge, the first method that can be successfully applied to such a diverse set of important vision tasks with event cameras.
Modeling Facial Geometry Using Compositional VAEs
Authors: Chenglei Wu, Jason Saragih, Pascal Fua, Yaser Sheikh Timur Bagautdinov
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We propose a method for learning non-linear face geometry  representations using deep generative models.  Our model is a variational autoencoder with multiple levels of  hidden variables where lower layers capture global geometry  and higher ones encode more local deformations.  Based on that, we propose a new parameterization of facial  geometry that naturally decomposes the structure of the human face  into a set of semantically meaningful levels of detail.  This parameterization enables us to do model fitting while  capturing varying level of detail under different types of geometrical  constraints.
Tangent Convolutions for Dense Prediction in 3D
Authors: Jaesik Park, Vladlen Koltun, Qian-Yi Zhou Maxim Tatarchenko
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We present an approach to semantic scene analysis using deep convolutional networks. Our approach is based on tangent convolutions - a new construction for convolutional networks on 3D data. In contrast to volumetric approaches, our method operates directly on surface geometry. Crucially, the construction is applicable to unstructured point clouds and other noisy real-world data. We show that tangent convolutions can be evaluated efficiently on large-scale point clouds with millions of points. Using tangent convolutions, we design a deep fully-convolutional network for semantic segmentation of 3D point clouds, and apply it to challenging real-world datasets of indoor and outdoor 3D environments. Experimental results show that the presented approach outperforms other recent deep network constructions in detailed analysis of large 3D scenes.
RayNet: Learning Volumetric 3D Reconstruction With Ray Potentials
Authors: Osman Ulusoy, Carolin Schmitt, Luc Van Gool, Andreas Geiger Despoina Paschalidou
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from data. However, they do not incorporate the physics of image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. RayNet integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.
Neural 3D Mesh Renderer
Authors: Yoshitaka Ushiku, Tatsuya Harada Hiroharu Kato
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.
Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation
Authors: Wei Wang, Hao Tang, Hong Liu, Nicu Sebe, Elisa Ricci Dan Xu
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Recent works have shown the benefit of integrating Conditional Random Fields (CRFs) models into deep architectures for improving pixel-level prediction tasks. Following this line of research, in this paper we introduce a novel approach for monocular depth estimation. Similarly to previous works, our method employs a continuous CRF to fuse multi-scale information derived from different layers of a front-end Convolutional Neural Network (CNN). Differently from past works, our approach benefits from a structured attention model which automatically regulates the amount of information transferred between corresponding features at different scales. Importantly, the proposed attention model is seamlessly integrated into the CRF, allowing end-to-end training of the entire architecture. Our extensive experimental evaluation demonstrates the effectiveness of the proposed method which is competitive with previous methods on the KITTI benchmark and outperforms the state of the art on the NYU Depth V2 dataset.
Automatic 3D Indoor Scene Modeling From Single Panorama
Authors: Shi Jin, Ruiyang Liu, Sing Bing Kang, Jingyi Yu Yang Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: We describe a system that automatically extracts 3D geometry of an indoor scene from a single 2D panorama. Our system recovers the spatial layout by finding the floor, walls, and ceiling; it also recovers shapes of typical indoor objects such as furniture. Using sampled perspective sub-views, we extract geometric cues (lines, vanishing points, orientation map, and surface normals) and semantic cues (saliency and object detection information). These cues are used for ground plane estimation and occlusion reasoning. The global spatial layout is inferred through a constraint graph on line segments and planar superpixels. The recovered layout is then used to guide shape estimation of the remaining objects using their normal information. Experiments on synthetic and real datasets show that our approach is state-of-the-art in both accuracy and efficiency. Our system can handle cluttered scenes with complex geometry that are challenging to existing techniques.
Extreme 3D Face Reconstruction: Seeing Through Occlusions
Authors: Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, Gérard Medioni Anh Tuấn Trần
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Existing single view, 3D face reconstruction methods can produce beautifully detailed 3D results, but typically only for near frontal, unobstructed viewpoints. We describe a system designed to provide detailed 3D reconstructions of faces viewed under extreme conditions, out of plane rotations, and occlusions. Motivated by the concept of bump mapping, we propose a layered approach which decouples estimation of a global shape from its mid-level details (e.g., wrinkles). We estimate a coarse 3D face shape which acts as a foundation and then separately layer this foundation with details represented by a bump map. We show how a deep convolutional encoder-decoder can be used to estimate such bump maps. We further show how this approach naturally extends to generate plausible details for occluded facial regions. We test our approach and its components extensively, quantitatively demonstrating the invariance of our estimated facial details. We further provide numerous qualitative examples showing that our method produces detailed 3D face shapes in viewing conditions where existing state of the art often break down.
Beyond Grobner Bases: Basis Selection for Minimal Solvers
Authors: Magnus Oskarsson, Kalle Astrom, Alge Wallis, Zuzana Kukelova, Tomas Pajdla Viktor Larsson
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Main Conference Track), 2018

Abstract: Many computer vision applications require robust estimation of the underlying geometry, in terms of camera motion and 3D structure of the scene. These robust methods often rely on running minimal solvers in a RANSAC framework. In this paper we show how we can make polynomial solvers based on the action matrix method faster, by careful selection of the monomial bases. These monomial bases have traditionally been based on a Grobner basis for the polynomial ideal. Here we describe how we can enumerate all such bases in an efficient way. We also show that going beyond Grobner bases leads to more efficient solvers in many cases. We present a novel basis sampling scheme that we evaluate on a number of problems.