mtybilly/Instruct2DS huggingface dataset:
Face Alignment Across Large Poses: A 3D Solution
Authors: Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, Stan Z. Li
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in CV community. However, most algorithms are designed for faces in small to medium poses (below 45 degree), lacking the ability to align faces in large-pose up to 90 degree. The challenges are three-fold: Firstly, the commonly used landmark-based face model assumes that all the landmarks are visible and is therefore not suitable for profile views. Secondly, the face appearance varies more dramatically in large poses, ranging from frontal view to profile view. Thirdly, labelling landmarks in large poses is an extremely challenging work since the invisible landmarks have to be guessed. In this paper, we propose a solution to the three problems in an new alignment framework, called 3D Dense Face Alignment (3DDFA), in which a dense 3D face model is fitted to the image via convolutional neutral network (CNN). We also propose a method to synthesize large-scale training samples in profile views to solve the third problem of data labelling. Experiments on the challenging AFLW database show that our approach achieves significant improvements over state-of-the-art methods.
Interactive Segmentation on RGBD Images via Cue Selection
Authors: Jie Feng, Brian Price, Scott Cohen, Shih-Fu Chang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Interactive image segmentation is an important problem in computer vision with many applications including image editing, object recognition and image retrieval. Most existing interactive segmentation methods only operate on color images. Until recently, very few works have been proposed to leverage depth information from low-cost sensors to improve interactive segmentation. While these methods achieve better results than color-based methods, they are still limited in either using depth as an additional color channel or simply combining depth with color in a linear way. We propose a novel interactive segmentation algorithm which can incorporate multiple feature cues like color, depth, and normals in an unified graph cut framework to leverage these cues more effectively. A key contribution of our method is that it automatically selects a single cue to be used at each pixel, based on the intuition that only one cue is necessary to determine the segmentation label locally.  This is achieved by optimizing over both segmentation labels and cue labels, using terms designed to decide where both the segmentation and label cues should change. Our algorithm thus produces not only the segmentation mask but also a cue label map that indicates where each cue contributes to the final result. Extensive experiments on five large scale RGBD datasets show that our proposed algorithm performs significantly better than both other color-based and RGBD based algorithms in reducing the amount of user inputs as well as increasing segmentation accuracy.
Layered Scene Decomposition via the Occlusion-CRF
Authors: Chen Liu, Pushmeet Kohli, Yasutaka Furukawa
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: This paper addresses the challenging problem of perceiving the hidden or occluded geometry of the scene depicted in any given RGBD image. Unlike other image labeling problems such as image segmentation where each pixel needs to be assigned a single label, layered decomposition requires us to assign multiple labels to pixels. We propose a novel "Occlusion-CRF" model that allows for the integration of sophisticated priors to regularize the solution space and enables the automatic inference of the layer decomposition. We use a generalization of the Fusion Move algorithm to perform Maximum a Posterior (MAP) inference on the model that can handle the large label sets needed to represent multiple surface assignments to each pixel. We have evaluated the proposed model and the inference algorithm on many RGBD images of cluttered indoor scenes. Our experiments show that not only is our model able to explain occlusions but it also enables automatic inpainting of occluded/invisible surfaces.
Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding
Authors: Michael Maire, Takuya Narihira, Stella X. Yu
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Spectral embedding provides a framework for solving perceptual organization problems, including image segmentation and figure/ground organization.  From an affinity matrix describing pairwise relationships between pixels, it clusters pixels into regions, and, using a complex-valued extension, orders pixels according to layer.  We train a convolutional neural network (CNN) to directly predict the pairwise relationships that define this affinity matrix.  Spectral embedding then resolves these predictions into a globally-consistent segmentation and figure/ground organization of the scene.  Experiments demonstrate significant benefit to this direct coupling compared to prior works which use explicit intermediate stages, such as edge detection, on the pathway from image to affinities.  Our results suggest spectral embedding as a powerful alternative to the conditional random field (CRF)-based globalization schemes typically coupled to deep neural networks.
Weakly Supervised Object Boundaries
Authors: Anna Khoreva, Rodrigo Benenson, Mohamed Omran, Matthias Hein, Bernt Schiele
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: State-of-the-art learning based boundary detection methods require extensive training data. Since labelling object boundaries is one of the most expensive types of annotations, there is a need to relax the requirement to carefully annotate images to make both the training more affordable and to extend the amount of training data. In this paper we propose a technique to generate weakly supervised annotations and show that bounding box annotations alone suffice to reach high-quality object boundaries without using any object-specific boundary annotations. With the proposed weak supervision techniques we achieve the top performance on the object boundary detection task, outperforming by a large margin the current fully supervised state-of-the-art methods.
Object Contour Detection With a Fully Convolutional Encoder-Decoder Network
Authors: Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, Ming-Hsuan Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We develop a deep learning algorithm for contour detection with a fully convolutional encoder-decoder network. Different from previous low-level edge detection, our algorithm focuses on detecting higher-level object contours. Our network is trained end-to-end on PASCAL VOC with refined ground truth from inaccurate polygon annotations, yielding much higher precision in object contour detection than previous methods. We find that the learned model generalizes well to unseen object classes from the same supercategories on MS COCO and can match state-of-the-art edge detection on BSDS500 with fine-tuning. By combining with the multiscale combinatorial grouping algorithm, our method can generate high-quality segmented object proposals, which significantly advance the state-of-the-art on PASCAL VOC (improving average recall from 0.62 to 0.67) with a relatively small amount of candidates (1660 per image).
What Value Do Explicit High Level Concepts Have in Vision to Language Problems?
Authors: Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton van den Hengel
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Much recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we investigate whether this direct approach succeeds due to, or despite, the fact that it avoids the explicit representation of high-level information. We propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. We achieve the best reported results on both image captioning and VQA on several benchmark datasets, and provide an analysis of the value of explicit high-level concepts in V2L problems.
Fast Detection of Curved Edges at Low SNR
Authors: Nati Ofir, Meirav Galun, Boaz Nadler, Ronen Basri
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Detecting edges is a fundamental problem in computer vision with many applications, some involving very noisy images. While most edge detection methods are fast, they perform well only on relatively clean images. Unfortunately, sophisticated methods that are robust to high levels of noise are quite slow. In this paper we develop a novel multiscale method to detect curved edges in noisy images. Even though our algorithm searches for edges over an exponentially large set of candidate curves, its runtime is nearly linear in the total number of image pixels. As we demonstrate experimentally, our algorithm is orders of magnitude faster than previous methods designed to deal with high noise levels.  At the same time it obtains comparable and often superior results to existing methods on a variety of challenging noisy images.
Object Skeleton Extraction in Natural Images by Fusing Scale-Associated Deep Side Outputs
Authors: Wei Shen, Kai Zhao, Yuan Jiang, Yan Wang, Zhijiang Zhang, Xiang Bai
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Object skeleton is a useful cue for object detection, complementary to the object contour, as it provides a structural representation to describe the relationship among object parts. While object skeleton extraction in natural images is a very challenging problem, as it requires the extractor to be able to capture both local and global image context to determine the intrinsic scale of each skeleton pixel. Existing methods rely on per-pixel based multi-scale feature computation, which results in difficult modeling and high time consumption. In this paper, we present a fully convolutional network with multiple scale-associated side outputs to address this problem. By observing the relationship between the receptive field sizes of the sequential stages in the network and the skeleton scales they can capture, we introduce a scale-associated side output to each stage. We impose supervision to different stages by guiding the scale-associated side outputs toward groundtruth skeletons of different scales. The responses of the multiple scale-associated side outputs are then fused in a scale-specific way to localize skeleton pixels with multiple scales effectively. Our method achieves promising results on two skeleton extraction datasets, and significantly outperforms other competitors.
Learning Relaxed Deep Supervision for Better Edge Detection
Authors: Yu Liu, Michael S. Lew
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We propose using relaxed deep supervision (RDS) within convolutional neural networks for edge detection. The conventional deep supervision utilizes the general ground-truth to guide intermediate predictions. Instead, we build hierarchical supervisory signals with additional relaxed labels to consider the diversities in deep neural networks. We begin by capturing the relaxed labels from simple detectors (e.g. Canny).Then we merge them with the general ground-truth to generate the RDS. Finally we employ the RDS to supervise the edge network following a coarse-to-fine paradigm. These relaxed labels can be seen as some false positives that are difficult to be classified. We consider these false positives in the supervision, and are able to achieve high performance for better edge detection. We compensate for the lack of training images by capturing coarse edge annotations from a large dataset of image segmentations to pretrain the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on the well-known BSDS500 dataset (ODS F-score of .792) and obtains superior cross-dataset generalization results on NYUD dataset.
Occlusion Boundary Detection via Deep Exploration of Context
Authors: Huan Fu, Chaohui Wang, Dacheng Tao, Michael J. Black
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Occlusion boundaries contain rich perceptual information about the underlying scene structure. They also provide important cues in many visual perception tasks such as scene understanding, object recognition, and segmentation. In this paper, we improve occlusion boundary detection via enhanced exploration of contextual information (e.g., local structural boundary patterns, observations from surrounding regions, and temporal context), and in doing so develop a novel approach based on convolutional neural networks (CNNs) and conditional random fields (CRFs). Experimental results demonstrate that our detector significantly outperforms the state-of-the-art (e.g., improving the F-measure from 0.62 to 0.71 on the commonly used CMU benchmark). Last but not least, we empirically assess the roles of several important components of the proposed detector, so as to validate the rationale behind this approach.
SemiContour: A Semi-Supervised Learning Approach for Contour Detection
Authors: Zizhao Zhang, Fuyong Xing, Xiaoshuang Shi, Lin Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Supervised contour detection methods usually require many labeled training images to obtain satisfactory performance. However, a large set of annotated data might be unavailable or extremely labor intensive. In this paper, we investigate the usage of semi-supervised learning (SSL) to obtain competitive detection accuracy with very limited training data (three labeled images). Specifically, we propose a semi-supervised structured ensemble learning approach for contour detection built on structured random forests (SRF). To allow SRF to be applicable to unlabeled data, we present an effective sparse representation approach to capture inherent structure in image patches by finding a compact and discriminative low-dimensional subspace representation in an unsupervised manner, enabling the incorporation of abundant unlabeled patches with their estimated structured labels to help SRF perform better node splitting. We re-examine the role of sparsity and propose a novel and fast sparse coding algorithm to boost the overall learning efficiency. To the best of our knowledge, this is the first attempt to apply SSL for contour detection. Extensive experiments on the BSDS500 segmentation dataset and the NYU Depth dataset demonstrate the superiority of the proposed method.
Learning to Localize Little Landmarks
Authors: Saurabh Singh, Derek Hoiem, David Forsyth
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We interact everyday with tiny objects such as the door handle of a car or the light switch in a room. These little landmarks are barely visible and hard to localize in images. We describe a method to find such landmarks by finding a sequence of latent landmarks, each with a prediction model. Each latent landmark predicts the next in sequence, and the last localizes the target landmark. For example, to find the door handle of a car, our method learns to start with a latent landmark near the wheel, as it is globally distinctive; subsequent latent landmarks use the context from the earlier ones to get closer to the target. Our method is supervised solely by the location of the little landmark and displays strong performance on more difficult variants of established tasks and on two new tasks.
InterActive: Inter-Layer Activeness Propagation
Authors: Lingxi Xie, Liang Zheng, Jingdong Wang, Alan L. Yuille, Qi Tian
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: An increasing number of computer vision tasks can be tackled with deep features, which are the intermediate outputs of a pre-trained Convolutional Neural Network. Despite the astonishing performance, deep features extracted from low-level neurons are still below satisfaction, arguably because they cannot access the spatial context contained in the higher layers. In this paper, we present InterActive, a novel algorithm which computes the activeness of neurons and network connections. Activeness is propagated through a neural network in a top-down manner, carrying high-level context and improving the descriptive power of low-level and mid-level neurons. Visualization indicates that neuron activeness can be interpreted as spatial-weighted neuron responses. We achieve state-of-the-art classification performance on a wide range of image datasets.
Exploit Bounding Box Annotations for Multi-Label Object Recognition
Authors: Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, Jianfei Cai
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Convolutional neural networks (CNNs) have shown great performance as general feature representations for object recognition applications. However, for multi-label images that contain multiple objects from different categories, scales and locations, global CNN features are not optimal. In this paper, we incorporate local information to enhance the feature discriminative power. In particular, we first extract object proposals from each image. With each image treated as a bag and object proposals extracted from it treated as instances, we transform the multi-label recognition problem into a multi-class multi-instance learning problem. Then, in addition to extracting the typical CNN feature representation from each proposal, we propose to make use of ground-truth bounding box annotations (strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline. The proposed multi-view multi-instance framework utilizes both weak and strong labels effectively, and more importantly it has the generalization ability to even boost the performance of unseen categories by partial strong labels from other categories. Our framework is extensively compared with state-of-the-art hand-crafted feature based methods and CNN based methods on two multi-label benchmark datasets. The experimental results validate the discriminative power and the generalization ability of the proposed framework. With strong labels, our framework is able to achieve state-of-the-art results in both datasets.
TI-Pooling: Transformation-Invariant Pooling for Feature Learning in Convolutional Neural Networks
Authors: Dmitry Laptev, Nikolay Savinov, Joachim M. Buhmann, Marc Pollefeys
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: In this paper we present a deep neural network topology that incorporates a simple to implement transformation-invariant pooling operator (TI-pooling). This operator is able to efficiently handle prior knowledge on nuisance variations in the data, such as rotation or scale changes. Most current methods usually make use of dataset augmentation to address this issue, but this requires larger number of model parameters and more training data, and results in significantly increased training time and larger chance of under- or overfitting. The main reason for these drawbacks is that that the learned model needs to capture adequate features for all the possible transformations of the input. On the other hand, we formulate features in convolutional neural networks to be transformation-invariant. We achieve that using parallel siamese architectures for the considered transformation set and applying the TI-pooling operator on their outputs before the fully-connected layers. We show that this topology internally finds the most optimal "canonical" instance of the input image for training and therefore limits the redundancy in learned features. This more efficient use of training data results in better performance on popular benchmark datasets with smaller number of parameters when comparing to standard convolutional neural networks with dataset augmentation and to other baselines.
Fashion Style in 128 Floats: Joint Ranking and Classification Using Weak Data for Feature Extraction
Authors: Edgar Simo-Serra, Hiroshi Ishikawa
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We propose a novel approach for learning features from weakly-supervised data by joint ranking and classification. In order to exploit data with weak labels, we jointly train a feature extraction network with a ranking loss and a classification network with a cross-entropy loss. We obtain high-quality compact discriminative features with few parameters, learned on relatively small datasets without additional annotations. This enables us to tackle tasks with specialized images not very similar to the more generic ones in existing fully-supervised datasets. We show that the resulting features in combination with a linear classifier surpass the state-of-the-art on the Hipster Wars dataset despite using features only 0.3% of the size. Our proposed features significantly outperform those obtained from networks trained on ImageNet, despite being 32 times smaller (128 single-precision floats), trained on noisy and weakly-labeled data, and using only 1.5% of the number of parameters.
Equiangular Kernel Dictionary Learning With Applications to Dynamic Texture Analysis
Authors: Yuhui Quan, Chenglong Bao, Hui Ji
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Most existing dictionary learning algorithms consider a linear sparse model, which often  cannot effectively characterize the nonlinear properties present in many types of visual data, e.g. dynamic texture (DT). Such nonlinear properties can be exploited by the so-called  kernel sparse coding. This paper proposed an equiangular kernel dictionary learning method with optimal mutual coherence to exploit  the nonlinear sparsity of  high-dimensional visual data. Two main issues are addressed in the proposed method: (1) coding stability for redundant dictionary of infinite-dimensional space; and (2) computational efficiency for computing kernel matrix of training samples of high-dimensional data. The proposed kernel sparse coding method is applied to dynamic texture analysis with both local DT pattern extraction and global DT pattern characterization. The experimental results showed its performance gain over existing methods.
Compact Bilinear Pooling
Authors: Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition. However, bilinear features are high dimensional, typically on the order of hundreds of thousands to a few million, which makes them impractical for subsequent analysis. We propose two compact bilinear representations with the same discriminative power as the full bilinear representation but with only a few thousand dimensions. Our compact representations allow back-propagation of classification errors enabling an end-to-end optimization of the visual recognition system. The compact bilinear representations are derived through a novel kernelized analysis of bilinear pooling which provide insights into the discriminative power of bilinear pooling, and a platform for further research in compact pooling methods. Experimentation illustrate the utility of the proposed representations for image classification and few-shot learning across several datasets.
Accumulated Stability Voting: A Robust Descriptor From Descriptors of Multiple Scales
Authors: Tsun-Yi Yang, Yen-Yu Lin, Yung-Yu Chuang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: This paper proposes a novel local descriptor through accumulated stability voting (ASV). The stability of feature dimensions is measured by their differences across scales. To be more robust to noise, the stability is further quantized by thresholding. The principle of maximum entropy is utilized for determining the best thresholds for maximizing discriminant power of the resultant descriptor. Accumulating stability renders a real-valued descriptor and it can be converted into a binary descriptor by an additional thresholding process. The real-valued descriptor attains high matching accuracy while the binary descriptor makes a good compromise between storage and accuracy. Our descriptors are simple yet effective, and easy to implement. In addition, our descriptors require no training. Experiments on popular benchmarks demonstrate the effectiveness of our descriptors and their superiority to the state-of-the-art descriptors.
CoMaL: Good Features to Match on Object Boundaries
Authors: Swarna K. Ravindran, Anurag Mittal
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Traditional Feature Detectors and Trackers use information aggregation in 2D patches to detect and match discriminative patches.  However, this information does not remain the same at object boundaries when there is object motion against a significantly varying background.  In this paper, we propose a new approach for feature detection, tracking and re-detection that gives significantly improved results at the object boundaries.  We utilize level lines or iso-intensity curves that often remain stable and can be reliably detected even at the object boundaries, which they often trace. Stable portions of long level lines are detected and points of high curvature  are detected on such curves for corner detection.  Further, this level line is used to separate the portions belonging to the two objects, which is then used for robust matching of such points.  While such CoMaL (Corners on Maximally-stable Level Line Segments) points were found to be much more reliable at the object boundary regions, they perform comparably at the interior regions as well. This is illustrated in exhaustive experiments on real-world datasets.
Progressive Feature Matching With Alternate Descriptor Selection and Correspondence Enrichment
Authors: Yuan-Ting Hu, Yen-Yu Lin
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We address two difficulties in establishing an accurate system for image matching. First, image matching relies on the descriptor for feature extraction, but the optimal descriptor often varies from image to image, or even patch to patch. Second, conventional matching approaches carry out geometric checking on a small set of correspondence candidates due to the concern of efficiency. It may result in restricted performance in recall. We aim at tackling the two issues by integrating adaptive descriptor selection and progressive candidate enrichment into image matching. We consider that the two integrated components are complementary: The high-quality matching yielded by adaptively selected descriptors helps in exploring more plausible candidates, while the enriched candidate set serves as a better reference for descriptor selection. It motivates us to formulate image matching as a joint optimization problem, in which adaptive descriptor selection and progressive correspondence enrichment are alternately conducted. Our approach is comprehensively evaluated and compared with the state-of-the-art approaches on two benchmarks. The promising results manifest its effectiveness.
A New Finsler Minimal Path Model With Curvature Penalization for Image Segmentation and Closed Contour Detection
Authors: Da Chen, Jean-Marie Mirebeau, Laurent D. Cohen
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: In this paper, we propose a new curvature penalized minimal path model for image segmentation via closed contour detection based on the weighted Euler elastica curves, firstly introduced to the field of computer vision in [22]. Our image segmentation method extracts a collection of curvature penalized minimal geodesics, concatenated to form a closed contour, by connecting a set of user-specified points. Globally optimal minimal paths can be computed by solving an Eikonal equation. This first order PDE is traditionally regarded as unable to penalize curvature, which is related to the path acceleration in active contour models. We introduce here a new approach that enables finding a global minimum of the geodesic energy including a curvature term. We achieve this through the use of a novel Finsler metric adding to the image domain the orientation as an extra space dimension. This metric is non-Riemannian and asymmetric, defined on an orientation lifted space, incorporating the curvature penalty in the geodesic energy. Experiments show that the proposed Finsler minimal path model indeed outperforms state-of-the-art minimal path models in both synthetic and real images.
Scale-Aware Alignment of Hierarchical Image Segmentation
Authors: Yuhua Chen, Dengxin Dai, Jordi Pont-Tuset, Luc Van Gool
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Image segmentation is a key component in many computer vision systems, and it is recovering a prominent spot in the literature as methods improve and overcome their limitations. The outputs of most recent algorithms are in the form of a hierarchical segmentation, which provides segmentation at different scales in a single tree-like structure. Commonly, these hierarchical methods start from some low-level features, and are not aware of the scale information of the different regions in them. As such, one might need to work on many different levels of the hierarchy to find the objects in the scene. This work tries to modify the existing hierarchical algorithm by improving their alignment, that is, by trying to modify the depth of the regions in the tree to better couple depth and scale. To do so, we first train a regressor to predict the scale of regions using mid-level features. We then define the anchor slice as the set of regions that better balance between over-segmentation and under-segmentation. The output of our method is an improved hierarchy, re-aligned by the anchor slice. To demonstrate the power of our method, we perform comprehensive experiments, which show that our method, as a post-processing step, can significantly improve the quality of the hierarchical segmentation representations, and ease the usage of hierarchical image segmentation to high-level vision tasks such as object segmentation. We also prove that the improvement generalizes well across different algorithms and datasets, with a low computational cost.
Deep Interactive Object Selection
Authors: Ning Xu, Brian Price, Scott Cohen, Jimei Yang, Thomas S. Huang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Interactive object selection is a very important research problem and has many applications. Previous algorithms require substantial user interactions to estimate the foreground and background distributions. In this paper, we present a novel deep-learning-based algorithm which has much better understanding of objectness and can reduce user interactions to just a few clicks. Our algorithm transforms user-provided positive and negative clicks into two Euclidean distance maps which are then concatenated with the RBG channels of images to compose (image, user interactions) pairs. We generate many of such pairs by combining several random sampling strategies to model users' click patterns and use them to finetune deep Fully Convolutional Networks (FCNs). Finally the output probability maps of our FCN-8s model is integrated with graph cut optimization to refine the boundary segments. Our model is trained on the PASCAL segmentation dataset and evaluated on other datasets with different object classes. Experimental results on both seen and unseen objects clearly demonstrate that our algorithm has a good generalization ability and is superior to all existing interactive object selection approaches.
Pull the Plug? Predicting If Computers or Humans Should Segment Images
Authors: Danna Gurari, Suyog Jain, Margrit Betke, Kristen Grauman
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Foreground object segmentation is a critical step for many image analysis tasks.  While automated methods can produce high-quality results, their failures disappoint users in need of practical solutions.  We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods.  The framework is based on a proposed prediction module that estimates the quality of given algorithm-drawn segmentations.  We demonstrate the value of the framework for two novel tasks related to "pulling the plug" on computer and human annotators.  Specifically, we implement two systems that automatically decide, for a batch of images, when to replace 1) humans with computers to create coarse segmentations required to initialize segmentation tools and 2) computers with humans to create final, fine-grained segmentations.  Experiments demonstrate the advantage of relying on a mix of human and computer efforts over relying on either resource alone for segmenting objects in three diverse datasets representing visible, phase contrast microscopy, and fluorescence microscopy images.
In the Shadows, Shape Priors Shine: Using Occlusion to Improve Multi-Region Segmentation
Authors: Yuka Kihara, Matvey Soloviev, Tsuhan Chen
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We present a new algorithm for multi-region segmentation of 2D images with objects that may partially occlude each other. Our algorithm is based on the observation that human performance on this task is based both on prior knowledge about plausible shapes and taking into account the presence of occluding objects whose shape is already known - once an occluded region is identified, the shape prior can be used to guess the shape of the missing part. We capture the former aspect using a deep learning model of shape; for the latter, we simultaneously minimize the energy of all regions and consider only unoccluded pixels for data agreement.   Existing algorithms incorporating object shape priors consider every object separately in turn and can't distinguish genuine deviation from the expected shape from parts missing due to occlusion. We show that our method significantly improves on the performance of a representative algorithm, as evaluated on both preprocessed natural and synthetic images. Furthermore, on the synthetic images, we recover the ground truth segmentation with good accuracy.
Convexity Shape Constraints for Image Segmentation
Authors: Loic A. Royer, David L. Richmond, Carsten Rother, Bjoern Andres, Dagmar Kainmueller
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Segmenting an image into multiple components is a central task in computer vision. In many practical scenarios, prior knowledge about plausible components is available. Incorporating such prior knowledge into models and algorithms for image segmentation is highly desirable, yet can be non-trivial. In this work, we introduce a new approach that allows, for the first time, to constrain some or all components of a segmentation to have convex shapes. Specifically, we extend the Minimum Cost Multicut Problem by a class of constraints that enforce convexity. To solve instances of this NP-hard integer linear program to optimality, we separate the proposed constraints in the branch-and-cut loop of a state-of-the-art ILP solver. Results on photographs and micrographs demonstrate the effectiveness of the approach as well as its advantages over the state-of-the-art heuristic.
MCMC Shape Sampling for Image Segmentation With Nonparametric Shape Priors
Authors: Ertunc Erdil, Sinan Yildirim, Mujdat Cetin, Tolga Tasdizen
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Segmenting images of low quality or with missing data is a challenging problem. Integrating statistical prior information about the shapes to be segmented can improve the segmentation results significantly. Most shape-based segmentation algorithms optimize an energy functional and find a point estimate for the object to be segmented. This does not provide a measure of the degree of confidence in that result, neither does it provide a picture of other probable solutions based on the data and the priors. With a statistical view, addressing these issues would involve the problem of characterizing the posterior densities of the shapes of the objects to be segmented. For such characterization, we propose a Markov chain Monte Carlo (MCMC) sampling-based image segmentation algorithm that uses statistical shape priors. In addition to better characterization of the statistical structure of the problem, such an approach would also have the potential to address issues with getting stuck at local optima, suffered by existing shape-based segmentation methods. Our approach is able to characterize the posterior probability density in the space of shapes through its samples, and to return multiple solutions, potentially from different modes of a multimodal probability density, which would be encountered, e.g., in segmenting objects from multiple shape classes. We present promising results on a variety of data sets. We also provide an extension for segmenting shapes of objects with parts that can go through independent shape variations. This extension involves the use of local shape priors on object parts and provides robustness to limitations in shape training data size.
From Noise Modeling to Blind Image Denoising
Authors: Fengyuan Zhu, Guangyong Chen, Pheng-Ann Heng
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Traditional image denoising algorithms always assume the noise to be homogeneous white Gaussian distributed. However, the noise on real images can be much more complex empirically. This paper addresses this problem and proposes a novel blind image denoising algorithm which can cope with real-world noisy images even when the noise model is not provided. It is realized by modeling image noise with mixture of Gaussian distribution (MoG) which can approximate large varieties of continuous distributions. As the number of components for MoG is unknown practically, this work adopts Bayesian nonparametric technique and proposes a novel Low-rank MoG filter (LR-MoG) to recover clean signals (patches) from noisy ones contaminated by MoG noise. Based on LR-MoG, a novel blind image denoising approach is developed. To test the proposed method, this study conducts extensive experiments on synthesis and real images. Our method achieves the state-of-the-art performance consistently.
Efficient and Robust Color Consistency for Community Photo Collections
Authors: Jaesik Park, Yu-Wing Tai, Sudipta N. Sinha, In So Kweon
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: We present an efficient technique to optimize color consistency of a collection of images depicting a common scene. Our method first recovers sparse pixel correspondences in the input images and stacks them into a matrix with many missing entries. We show that this matrix satisfies a rank two constraint under a simple color correction model. These parameters can be viewed as pseudo white balance and gamma correction parameters for each input image. We present a robust low-rank matrix factorization method to estimate the unknown parameters of this model. Using them, we improve color consistency of the input images or perform color transfer with any input image as the source. Our approach is insensitive to outliers in the pixel correspondences thereby precluding the need for complex pre-processing steps. We demonstrate high-quality color consistency results on large photo collections of popular tourist landmarks and personal photo collections containing images of people.
Needle-Match: Reliable Patch Matching Under High Uncertainty
Authors: Or Lotan, Michal Irani
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Reliable patch-matching forms the basis for many algorithms (super-resolution, denoising, inpainting, etc.) However, when the image quality deteriorates (by noise, blur or  geometric distortions), the reliability of patch-matching deteriorates as well. Matched patches in the degraded image, do not necessarily imply similarity of the underlying patches in the (unknown) high-quality image. This restricts the applicability of patch-based methods. In this paper we present a patch representation called "Needle", which consists of small multi-scale versions of the patch and its immediate surrounding region. While the patch at the finest image scale is severely degraded, the degradation decreases dramatically in coarser needle scales, revealing reliable information for matching. We show that the Needle is  robust to many types of image degradations, leads to matches faithful to the underlying high-quality patches, and to improvement in existing patch-based methods.
ReconNet: Non-Iterative Reconstruction of Images From Compressively Sensed Measurements
Authors: Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, Amit Ashok
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: The goal of this paper is to present a non-iterative and more importantly an extremely fast algorithm to reconstruct images from compressively sensed (CS) random measurements.  To this end, we propose a novel convolutional neural network (CNN) architecture which takes in CS measurements of an image as input and outputs  an intermediate  reconstruction.  We call this network, ReconNet. The intermediate reconstruction is fed into an off-the-shelf denoiser to obtain the final reconstructed image. On a standard dataset of  images we show significant improvements in reconstruction results (both in terms of PSNR and time complexity) over state-of-the-art iterative CS reconstruction algorithms at various measurement rates. Further, through qualitative experiments on real data collected using our block SPC (single pixel camera), we show that our network is highly robust to sensor noise and can recover visually better quality images than competitive algorithms at extremely low sensing rates of 0.1 and 0.04. To demonstrate that our algorithm can recover semantically informative images even at a low measurement rate of 0.01, we present a very robust proof of concept real-time visual tracking application.
Soft-Segmentation Guided Object Motion Deblurring
Authors: Jinshan Pan, Zhe Hu, Zhixun Su, Hsin-Ying Lee, Ming-Hsuan Yang
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: Object motion blur is a challenging problem as the foreground and the background in the scenes undergo different types of image degradation due to movements in various directions and speed. Most object motion deblurring methods address this problem by segmenting blurred images into regions where different kernels are estimated and applied for restoration. Segmentation on blurred images is difficult due to ambiguous pixels between regions, but it plays an important role for object motion deblurring. To address these problems, we propose a novel model for object motion deblurring. The proposed model is developed based on a maximum a posterior formulation in which soft-segmentation is incorporated for object layer estimation. We propose an efficient algorithm to jointly estimate object segmentation and camera motion where each layer can be deblurred well under the guidance of the soft-segmentation. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art object motion deblurring methods on challenging scenarios.
Two Illuminant Estimation and User Correction Preference
Authors: Dongliang Cheng, Abdelrahman Abdelhamed, Brian Price, Scott Cohen, Michael S. Brown
Published in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 (Main Conference Track), 2016

Abstract: This paper examines the problem of white-balance correction when a scene contains two illuminations.  This is a two step process: 1) estimate the two illuminants; and 2) correct the image.   Existing methods attempt to estimate a spatially varying illumination map, however, results are error prone and the resulting illumination maps are too low-resolution to be used for proper spatially varying white-balance correction.  In addition, the spatially varying nature of these methods make them computationally intensive.   We show that this problem can be effectively addressed by not attempting to obtain a spatially varying illumination map, but instead by performing illumination estimation on large sub-regions of the image.   Our approach is able to detect when distinct illuminations are present in the image and accurately measure these illuminants.   Since our proposed strategy is not suitable for spatially varying image correction, a user study is performed to see if there is a preference for how the image should be corrected when two illuminants are present, but only a global correction can be applied.   The user study shows that when the illuminations are distinct, there is a preference for the outdoor illumination to be corrected resulting in warmer final result.  We use these collective findings to demonstrate an effective two illuminant estimation scheme that produces corrected images that users prefer.