mtybilly/Instruct2DS huggingface dataset:
Efficient Solution to the Epipolar Geometry for Radially Distorted Cameras
Authors: Jan Heller, Martin Bujnak, Andrew Fitzgibbon, Tomas Pajdla Zuzana Kukelova
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: The estimation of the epipolar geometry of two cameras from image matches is a fundamental problem of computer vision with many applications. While the closely related problem of estimating relative pose of two different uncalibrated cameras with radial distortion is of particular importance, none of the previously published methods is suitable for practical applications. These solutions are either numerically unstable, sensitive to noise, based on a large number of point correspondences, or simply too slow for real-time applications. In this paper, we present a new efficient solution to this problem that uses 10 image correspondences. By manipulating ten input polynomial equations, we derive a degree 10 polynomial equation in one variable. The solutions to this equation are efficiently found using the Sturm sequences method. In the experiments, we show that the proposed solution is stable, noise resistant, and fast, and as such efficiently usable in a practical Structure-from-Motion pipeline.
Learning a Descriptor-Specific 3D Keypoint Detector
Authors: Federico Tombari, Riccardo Spezialetti, Luigi Di Stefano Samuele Salti
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Keypoint detection represents the first stage in the majority of modern computer vision pipelines based on automatically established correspondences between local descriptors. However, no standard solution has emerged yet in the case of 3D data such as point clouds or meshes, which exhibit high variability in level of detail and noise. More importantly, existing proposals for 3D keypoint detection rely on geometric saliency functions that attempt to maximize repeatability rather than distinctiveness of the selected regions, which may lead to sub-optimal performance of the overall pipeline. To overcome these shortcomings, we cast 3D keypoint detection as a binary classification between points whose support can be correctly matched by a predefined 3D descriptor or not, thereby learning a descriptor-specific detector that adapts seamlessly to different scenarios. Through  experiments on several public datasets, we show that this novel approach to the design of a keypoint detector represents a flexible solution that, nonetheless, can provide state-of-the-art descriptor matching performance.
Component-Wise Modeling of Articulated Objects
Authors: Marta Sanzari, Bruno Cafaro, Federico Nardi, Fabrizio Natola, Fiora Pirri, Manuel Ruiz Valsamis Ntouskos
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We introduce a novel framework for modeling articulated objects based on the aspects of their components. By decomposing the object into components, we divide the problem in smaller modeling tasks. After obtaining 3D models for each component aspect by employing a shape deformation paradigm, we merge them together, forming the object components. The final model is obtained by assembling the components using an optimization scheme which fits the respective 3D models to the corresponding apparent contours in a reference pose.  The results suggest that our approach can produce realistic 3D models of articulated objects in reasonable time.
A Collaborative Filtering Approach to Real-Time Hand Pose Estimation
Authors: Ayan Sinha, Joon Hee Choi, Sujin Jang, Karthik Ramani Chiho Choi
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Collaborative filtering aims to predict unknown user ratings in a recommender system by collectively assessing known user preferences. In this paper, we first draw analogies between collaborative filtering and the pose estimation problem. Specifically, we recast the hand pose estimation problem as the cold-start problem for a new user with unknown item ratings in a recommender system. Inspired by fast and accurate matrix factorization techniques for collaborative filtering, we develop a real-time algorithm for estimating the hand pose from RGB-D data of a commercial depth camera. First, we efficiently identify nearest neighbors using local shape descriptors in the RGB-D domain from a library of hand poses with known pose parameter values. We then use this information to evaluate the unknown pose parameters using a joint matrix factorization and completion (JMFC) approach. Our quantitative and qualitative results suggest that our approach is robust to variation in hand configurations while achieving real time performance (29 FPS) on a standard computer.
On the Equivalence of Moving Entrance Pupil and Radial Distortion for Camera Calibration
Authors: Narendra Ahuja Avinash Kumar
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Radial distortion for ordinary (non-fisheye) camera lenses has traditionally been modeled as an infinite series function of radial location of an image pixel from the image center. While there has been enough empirical evidence  to show that such a model is accurate and sufficient for radial distortion calibration, there has not been much analysis on the geometric/physical understanding of radial distortion from a camera calibration perspective. In this paper, we show using a thick-lens imaging model, that the variation of entrance pupil location as a function of incident image ray angle is directly responsible for radial distortion in captured images. Thus, unlike as proposed in the current state-of-the-art in camera calibration, radial distortion  and entrance pupil movement are equivalent and need not be modeled together.  By modeling only entrance pupil motion instead of radial distortion,  we achieve two main benefits;  first, we obtain comparable if not better pixel re-projection error than traditional methods; second, and more importantly, we directly back-project a radially distorted image pixel along the  true image ray which formed it. Using a thick-lens setting, we show that such a back-projection is more  accurate than the two-step method of undistorting an image pixel and  then back-projecting it. We have applied this calibration method to the problem of generative depth-from-focus using focal stack to get accurate depth estimates.
A Linear Generalized Camera Calibration From Three Intersecting Reference Planes
Authors: Shohei Nobuhara, Takashi Matsuyama, Shinya Shimizu, Kensaku Fujii Mai Nishimura
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: This paper presents a new generalized (or ray-pixel, raxel) camera calibration algorithm for camera systems involving distortions by unknown refraction and reflection processes. The key idea is use of intersections of calibration planes, while conventional methods utilized collinearity constraints of points on the planes. We show that intersections of calibration planes can realize a simple linear algorithm, and that our method can be applied to any ray-distributions while conventional methods require knowing the ray-distribution class in advance. Evaluations using synthesized and real datasets demonstrate the performance of our method quantitatively and qualitatively.
Towards Pointless Structure From Motion: 3D Reconstruction and Camera Parameters From General 3D Curves
Authors: Andrew Fitzgibbon Irina Nurutdinova
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Modern structure from motion (SfM) remains dependent on point features to recover camera positions, meaning that reconstruction is severely hampered in low-texture environments, for example scanning a plain coffee cup on an uncluttered table.  We show how 3D curves can be used to refine camera position estimation in challenging low-texture scenes.  In contrast to previous work, we allow the curves to be partially observed in all images, meaning that for the first time, curve-based SfM can be demonstrated in realistic scenes.  The algorithm is based on bundle adjustment, so needs an initial estimate, but even a poor estimate from a few point correspondences can be substantially improved by including curves, suggesting that this method would benefit many existing systems.
Attributed Grammars for Joint Estimation of Human Attributes, Part and Pose
Authors: Song-Chun Zhu Seyoung Park
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper, we are interested in developing compositional models to explicit representing  pose, parts and attributes and tackling the tasks of attribute recognition, pose estimation and part localization jointly. This is different from the recent trend of using CNN-based approaches for training and testing on these tasks separately with a large amount of data. Conventional attribute models typically use a large number of region-based attribute classifiers on parts of pre-trained pose estimator without explicitly detecting the object or its parts, or considering the correlations between attributes. In contrast, our approach jointly represents both the object parts and their semantic attributes within a unified compositional hierarchy. We apply our attributed grammar model to the task of human parsing by simultaneously performing part localization and attribute recognition. We show our modeling helps performance improvements on pose-estimation task and also outperforms on other existing methods on attribute prediction task.
Real-Time Pose Estimation Piggybacked on Object Detection
Authors: Adam Herout, Marketa Dubska, Pavel Zemcik Roman Juranek
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We present an object detector coupled with pose estimation directly in a single compact and simple model, where the detector shares extracted image features with the pose estimator. The output of the classification of each candidate window consists of both object score and likelihood map of poses. This extension introduces negligible overhead during detection so that the detector is still capable of real time operation. We evaluated the proposed approach on the problem of vehicle detection. We used existing datasets with viewpoint/pose annotation (WCVP, 3D objects, KITTI).  Besides that, we collected a new traffic surveillance dataset COD20k which fills certain gaps of the existing datasets and we make it public. The experimental results show that the proposed approach is comparable with state-of-the-art approaches in terms of accuracy, but it is considerably faster - easily operating in real time (Matlab with C++ code). The source codes and the collected COD20k dataset are made public along with the paper.
Understanding and Predicting Image Memorability at a Large Scale
Authors: Akhil S. Raju, Antonio Torralba, Aude Oliva Aditya Khosla
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Progress in estimating visual memorability has been limited by the small scale and lack of variety of benchmark data. Here, we introduce a novel experimental procedure to objectively measure human memory, building the largest annotated image memorability dataset to date (with 60,000 labeled images from a diverse array of sources). Using Convolutional Neural Networks (CNNs), we show that fine-tuned deep features outperform all other features by a large margin, reaching a rank correlation of 0.64, near human consistency (0.68). Analysis of the responses of the high-level CNN layers shows which objects and regions are positively, and negatively, correlated with memorability, allowing us to create memorability maps for each image and provide a concrete method to perform image memorability manipulation. This work demonstrates that one can now robustly estimate the memorability of images from many different classes, positioning memorability and deep memorability features as prime candidates to estimate the utility of information for cognitive systems.
Multiple Granularity Descriptors for Fine-Grained Categorization
Authors: Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, Zheng Zhang Dequan Wang
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Fine-grained categorization, which aims to distinguish subordinate-level categories such as bird species or dog breeds, is an extremely challenging task. This is due to two main issues: how to localize discriminative regions for recognition and how to learn sophisticated features for representation. Neither of them is easy to handle if there is insufficient labeled data. We leverage the fact that a subordinate-level object already has other labels in its ontology tree. These "free" labels can be used to train a series of CNN-based classifiers, each specialized at one grain level. The internal representations of these networks have different region of interests, allowing the construction of multi-grained descriptors that encode informative and discriminative features covering all the grain levels. Our multiple granularity framework can be learned with the weakest supervision, requiring only image-level label and avoiding the use of labor-intensive bounding box or part annotations. Experimental results on three challenging fine-grained image datasets demonstrate that our approach outperforms state-of-the-art algorithms, including those requiring strong labels.
Guiding the Long-Short Term Memory Model for Image Caption Generation
Authors: Efstratios Gavves, Basura Fernando, Tinne Tuytelaars Xu Jia
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM)  model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.
Just Noticeable Differences in Visual Attributes
Authors: Kristen Grauman Aron Yu
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We explore the problem of predicting "just noticeable differences" in a visual attribute.  While some pairs of images have a clear ordering for an attribute (e.g., A is more sporty than B), for others the difference may be indistinguishable to human observers.  However, existing relative attribute models are unequipped to infer partial orders on novel data.   Attempting to map relative attribute ranks to equality predictions is non-trivial, particularly since the span of indistinguishable pairs in attribute space may vary in different parts of the feature space.  We develop a Bayesian local learning strategy to infer when images are indistinguishable for a given attribute.  On the UT-Zap50K shoes and LFW-10 faces datasets, we outperform a variety of alternative methods.  In addition, we show the practical impact on fine-grained visual search.
VQA: Visual Question Answering
Authors: Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh Stanislaw Antol
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing  0.25M images,  0.76M questions, and  10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines for VQA are provided and compared with human performance.
Localize Me Anywhere, Anytime: A Multi-Task Point-Retrieval Approach
Authors: Yan Yan, Li Ren, Jingkuan Song, Nicu Sebe, Chandra Kambhamettu Guoyu Lu
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Image-based localization is an essential complement to GPS localization. Current image-based localization methods are based on either 2D-to-3D or 3D-to-2D to find the correspondences, which ignore the real scene geometric attributes. The main contribution of our paper is that we use a 3D model reconstructed by a short video as the query to realize 3D-to-3D localization under a multi-task point retrieval framework. Firstly, the use of a 3D model as the query enables us to efficiently select location candidates. Furthermore, the reconstruction of 3D model exploits the correlations among different images, based on the fact that images captured from different views for SfM share information through matching features. By exploring shared information (matching features) across multiple related tasks (images of the same scene captured from different views), the visual feature's view-invariance property can be improved in order to get to a higher point retrieval accuracy. More specifically, we use multi-task point retrieval framework to explore the relationship between descriptors and the 3D points, which extracts the discriminant points for more accurate 3D-to-3D correspondences retrieval. We further apply multi-task learning (MTL) retrieval approach on thermal images to prove that our MTL retrieval framework also provides superior performance for the thermal domain. This application is exceptionally helpful to cope with the localization problem in an environment with limited light sources.
Dense Optical Flow Prediction From a Static Image
Authors: Abhinav Gupta, Martial Hebert Jacob Walker
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Given a scene, what is going to move, and in what direction will it move? Such a question could be considered a non-semantic form of action prediction. In this work, we present a convolutional neural network (CNN) based approach for motion prediction. Given a static image, this CNN predicts the future motion of each and every pixel in the image in terms of optical flow. Our CNN model leverages the data in tens of thousands of realistic videos to train our model. Our method relies on absolutely no human labeling and is able to predict motion based on the context of the scene. Because our CNN model makes no assumptions about the underlying scene, it can predict future optical flow on a diverse set of scenarios. We outperform all previous approaches by large margins.
Unsupervised Domain Adaptation for Zero-Shot Learning
Authors: Tao Xiang, Zhenyong Fu, Shaogang Gong Elyor Kodirov
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Zero-shot learning (ZSL) can be considered as a special case of transfer learning where the source and target domains have different tasks/label spaces and the target domain is unlabelled, providing little guidance for the knowledge transfer. A ZSL method typically assumes that the two domains share a common semantic representation space, where a visual feature vector extracted from an image/video can be  projected/embedded using a projection function. Existing approaches learn the projection function from the source domain and apply it without adaptation to the target domain. They are thus based on naive knowledge transfer and the learned projections are prone to the domain shift problem. In this paper a novel ZSL method is proposed based on unsupervised domain adaptation. Specifically, we formulate a novel regularised sparse coding framework which uses the target domain class labels' projections in the semantic space  to regularise the learned target domain projection thus effectively overcoming the projection domain shift problem. Extensive experiments on four object and action recognition benchmark datasets show that the proposed ZSL method significantly outperforms the state-of-the-arts.
Visual Madlibs: Fill in the Blank Description Generation and Question Answering
Authors: Eunbyung Park, Alexander C. Berg, Tamara L. Berg Licheng Yu
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images.  This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.
Actions and Attributes From Wholes and Parts
Authors: Ross Girshick, Jitendra Malik Georgia Gkioxari
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We investigate the importance of parts for the tasks of action and attribute classification. We develop a part-based approach by leveraging convolutional network features inspired by recent advances in computer vision. Our part detectors are a deep version of poselets and capture parts of the human body under a distinct set of poses. For the tasks of action and attribute classification, we train holistic convolutional neural networks and show that adding parts leads to top-performing results for both tasks. We observe that for deeper networks parts are less significant. In addition, we demonstrate the effectiveness of our approach when we replace an oracle person detector, as is the default in the current evaluation protocol for both tasks, with a state-of-the-art person detection system.
DeepBox: Learning Objectness With Convolutional Networks
Authors: Bharath Hariharan, Jitendra Malik Weicheng Kuo
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Existing object proposal approaches use primarily bottom-up cues to rank proposals, while we believe that "objectness" is in fact a high level construct. We argue for a data-driven, semantic approach for ranking object proposals. Our framework, which we call DeepBox, uses convolutional neural networks (CNNs) to rerank proposals from a bottom-up method.  We use a novel four-layer CNN architecture that is as good as much larger networks on the task of evaluating objectness while being much faster. We show that DeepBox significantly improves over the bottom-up ranking, achieving the same recall with 500 proposals as achieved by bottom-up methods with 2000. This improvement generalizes to categories the CNN has never seen before and leads to a 4.5-point gain in detection mAP. Our implementation achieves this performance while running at 260 ms per image.
Active Object Localization With Deep Reinforcement Learning
Authors: Svetlana Lazebnik Juan C. Caicedo
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We present an active detection model for localizing objects in scenes. The model is class-specific and allows an agent to focus attention on candidate regions for identifying the correct location of a target object. This agent learns to deform a bounding box using simple transformation actions, with the goal of determining the most specific location of target objects following top-down reasoning. The proposed localization agent is trained using deep reinforcement learning, and evaluated on the Pascal VOC 2007 dataset. We show that agents guided by the proposed model are able to localize a single instance of an object after analyzing only between 11 and 25 regions in an image, and obtain the best detection results among systems that do not use object proposals for object localization.
Scene-Domain Active Part Models for Object Representation
Authors: Chaohui Wang, Alan L. Yuille Zhou Ren
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper, we are interested in enhancing the expressivity and robustness of part-based models for object representation, in the common scenario where the training data are based on 2D images. To this end, we propose scene-domain active part models (SDAPM), which reconstruct and characterize the 3D geometric statistics between object's parts in 3D scene-domain by using 2D training data in the image-domain alone. And on top of this, we explicitly model and handle occlusions in SDAPM. Together with the developed learning and inference algorithms, such a model provides rich object descriptions, including 2D object and parts localization, 3D landmark shape and camera viewpoint, which offers an effective representation to various image understanding tasks, such as object and parts detection, 3D landmark shape and viewpoint estimation from images. Experiments on the above tasks show that SDAPM outperforms previous part-based models, and thus demonstrates the potential of the proposed technique.
A Unified Multiplicative Framework for Attribute Learning
Authors: Hong Chang, Shiguang Shan, Xilin Chen Kongming Liang
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Attributes are mid-level semantic properties of objects. Recent research has shown that visual attributes can benefit many traditional learning problems in computer vision community. However, attribute learning is still a challenging problem as the attributes may not always be predictable directly from input images and the variation of visual attributes is sometimes large across categories. In this paper, we propose a unified multiplicative framework for attribute learning, which tackles the key problems. Specifically, images and category information are jointly projected into a shared feature space, where the latent factors are disentangled and multiplied for attribute prediction. The resulting attribute classifier is category-specific instead of being shared by all  categories. Moreover, our method can leverage auxiliary data to enhance the predictive ability of attribute classifiers, reducing the effort of instance-level attribute annotation to some extent. Experimental results show that our method achieves superior performance on both instance-level and category-level attribute prediction. For zero-shot learning based on attributes, our method significantly improves the state-of-the-art performance on AwA dataset and achieves comparable performance on CUB dataset.
Contractive Rectifier Networks for Nonlinear Maximum Margin Classification
Authors: Munawar Hayat, Salman H. Khan, Mohammed Bennamoun, Farid Boussaid, Ferdous Sohel Senjian An
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: To find the optimal nonlinear separating boundary with maximum margin in the input data space, this paper proposes Contractive Rectifier Networks (CRNs), wherein the hidden-layer transformations are restricted to be contraction mappings. The contractive constraints ensure that the achieved separating margin in the input space is larger than or equal to the separating margin in the output layer. The training of the proposed CRNs is formulated as a linear support vector machine (SVM) in the output layer, combined with two or more contractive hidden layers. Effective algorithms have been proposed to address the optimization challenges arising from contraction constraints. Experimental results on MNIST, CIFAR-10, CIFAR-100 and MIT-67 datasets demonstrate that the proposed contractive rectifier networks consistently outperform their conventional unconstrained rectifier network counterparts.
Augmenting Strong Supervision Using Web Data for Fine-Grained Categorization
Authors: Shaoli Huang, Ya Zhang, Dacheng Tao Zhe Xu
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: We propose a new method for fine-grained object recognition that employs part-level annotations and deep convolutional neural networks (CNNs) in a unified framework. Although both schemes have been widely used to boost recognition performance, due to the difficulty in acquiring detailed part annotations, strongly supervised fine-grained datasets are usually too small to keep pace with the rapid evolution of CNN architectures. In this paper, we solve this problem by exploiting inexhaustible web data. The proposed method improves classification accuracy in two ways: more discriminative CNN feature representations are generated using a training set augmented by collecting a large number of part patches from weakly supervised web images; and more robust object classifiers are learned using a multi-instance learning algorithm jointly on the strong and weak datasets. Despite its simplicity, the proposed method delivers a remarkable performance improvement on the CUB200-2011 dataset compared to baseline part-based R-CNN methods, and achieves the highest accuracy on this dataset even in the absence of test image annotations.
Learning Like a Child: Fast Novel Visual Concept Learning From Sentence Descriptions of Images
Authors: Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, Alan L. Yuille Junhua Mao
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. Our method has an image captioning module based on the m-RNN model with several improvements. In particular, we propose a transposed weight sharing scheme, which not only improves performance on image captioning, but also makes the model more suitable for the novel concept learning task. We propose methods to prevent overfitting the new concepts. In addition, three novel concept datasets are constructed for this new task, and are publicly available on the project page. In the experiments, we show that our method effectively learns novel visual concepts from a few examples without disturbing the previously learned concepts. The project page is: http://www.stat.ucla.edu/ junhua.mao/projects/child_learning.html
Learning Common Sense Through Visual Abstraction
Authors: Xiao Lin, Tanmay Batra, C. Lawrence Zitnick, Devi Parikh Ramakrishna Vedantam
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Common sense is essential for building intelligent machines. While some commonsense knowledge is explicitly stated in human-generated text and can be learnt by mining the web, much of it is unwritten. It is often unnecessary and even unnatural to write about commonsense facts. While unwritten, this commonsense knowledge is not unseen! The visual world around us is full of structure modeled by commonsense knowledge. Can machines learn common sense simply by observing our visual world? Unfortunately, this requires automatic and accurate detection of objects, their attributes, poses, and interactions between objects, which remain challenging problems. Our key insight is that while visual common sense is depicted in visual content, it is the semantic features that are relevant and not low-level pixel information. In other words, photorealism is not necessary to learn common sense. We explore the use of human-generated abstract scenes made from clipart for learning common sense. In particular, we reason about the plausibility of an interaction or relation between a pair of nouns by measuring the similarity of the relation and nouns with other relations and nouns we have seen in abstract scenes. We show that the commonsense knowledge we learn is complementary to what can be learnt from sources of text.
Domain Generalization for Object Recognition With Multi-Task Autoencoders
Authors: W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi Muhammad Ghifary
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: The problem of domain generalization is to take knowledge acquired from a number of related domains, where training data is available, and to then successfully apply it to previously unseen domains. We propose a new feature learning algorithm, Multi-Task Autoencoder (MTAE), that provides good generalization performance for cross-domain object recognition.  The algorithm extends the standard denoising autoencoder framework by substituting artificially induced corruption with naturally occurring inter-domain variability in the appearance of objects. Instead of reconstructing images from noisy versions, MTAE learns to transform the original image into analogs in multiple related domains. It thereby learns features that are robust to variations across domains. The learnt features are then used as inputs to a classifier.  We evaluated the performance of the algorithm on benchmark image recognition datasets, where the task is to learn features from multiple datasets and to then predict the image label from unseen datasets. We found that (denoising) MTAE outperforms alternative autoencoder-based models as well as the current state-of-the-art algorithms for domain generalization.
Square Localization for Efficient and Accurate Object Detection
Authors: Yongyi Lu, Hao Chen, Chi-Keung Tang Cewu Lu
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: The key contribution of this paper is the compact square object localization, which relaxes the exhaustive sliding window from testing all windows of different combinations of aspect ratios. Square object localization is category scalable. By using a binary search strategy, the number of scales to test is further reduced empirically to only O(log(minfH;Wg)) rounds of sliding CNNs, where H and W are respectively the image height and width. In the training phase, square CNN models and object co-presence priors are learned. In the testing phase, sliding CNN models are applied which produces a set of response maps that can be effectively filtered by the learned co-presence prior to output the final bounding boxes for localizing an object. We performed extensive experimental evaluation on the VOC 2007 and 2012 datasets to demonstrate that while efficient,square localization can output precise bounding boxes to improve the final detection result.
Box Aggregation for Proposal Decimation: Last Mile of Object Detection
Authors: Cewu Lu, Jiaya Jia Shu Liu
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Regions-with-convolutional-neural-network (RCNN) is now a commonly employed object detection pipeline. Its main steps, i.e., proposal generation and convolutional neural network (CNN) feature extraction, have been intensively investigated. We focus on the last step of the system to aggregate thousands of scored box proposals into final object prediction, which we call proposal decimation. We show this step can be enhanced with a very simple box aggregation function by considering statistical properties of proposals with respect to ground truth objects. Our method is with extremely light-weight computation, while it yields an improvement of 3.7% in mAP on PASCAL VOC 2007 test. We explain why it works using some statistics in this paper.
DeepProposal: Hunting Objects by Cascading Deep Convolutional Layers
Authors: Ali Diba, Marco Pedersoli, Tinne Tuytelaars, Luc Van Gool Amir Ghodrati
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper we evaluate the quality of the activation layers of a convolutional neural network (CNN) for the generation of object proposals. We generate hypotheses in a sliding-window fashion over different activation layers and show that the final convolutional layers can find the object of interest with high recall but poor localization due to the coarseness of the feature maps. Instead, the first layers of the network can better localize the object of interest but with a reduced recall. Based on this observation we design a method for proposing object locations that is based on CNN features and that combines the best of both worlds. We build an inverse cascade that, going from the final to the initial convolutional layers of the CNN, selects the most promising object locations and refines their boxes in a coarse-to-fine manner. The method is efficient, because i) it uses the same features extracted for detection, ii) it aggregates features using integral images, and iii) it avoids a dense evaluation of the proposals due to the inverse coarse-to-fine cascade. The method is also accurate; it outperforms most of the previously proposed object proposals approaches and when plugged into a CNN-based detector produces state-of-the-art detection performance.
Semantic Segmentation With Object Clique Potential
Authors: Jianping Shi, Shu Liu, Renjie Liao, Jiaya Jia Xiaojuan Qi
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper, we propose an object clique potential for semantic segmentation. Our object clique potential addresses the misclassified object-part issues arising in solutions based on fully-connected networks. Our object clique set, compared to that yielded from segment-proposal-based approaches, is with a significantly small size, making our method consume notably less computation. Regarding system design and model formation, our object clique potential can be regarded as a functionally complement to local-appearance-based CRF models and works in synergy with these effective approaches for further performance improvement. Extensive experiments verify our method.
Automatic Concept Discovery From Parallel Text and Visual Corpora
Authors: Chuang Gan, Ram Nevatia Chen Sun
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Humans connect language and vision to perceive the world. How to build a similar connection for computers? One possible way is via visual concepts, which are text terms that relate to visually discriminative entities. We propose an automatic visual concept discovery algorithm using parallel text and visual corpora; it filters text terms based on the visual discriminative power of the associated images, and groups them into concepts using visual and semantic similarities. We illustrate the applications of the discovered concepts using bidirectional image and sentence retrieval task and image tagging task, and show that the discovered concepts not only outperform several large sets of manually selected concepts significantly, but also achieves the state-of-the-art performance in the retrieval task.
Simpler Non-Parametric Methods Provide as Good or Better Results to Multiple-Instance Learning
Authors: Parag Chandakkar, Baoxin Li Ragav Venkatesan
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Multiple-instance learning (MIL) is a unique learning problem in which training data labels are available only for collections of objects (called bags) instead of individual objects (called instances). A plethora of approaches have been developed to solve this problem in the past years. Popular methods include the diverse density, MILIS and DD-SVM. While having been widely used, these methods, particularly those in computer vision have attempted fairly sophisticated solutions to solve certain unique and particular configurations of the MIL space.      In this paper, we analyze the MIL feature space using modified versions of traditional non-parametric techniques like the Parzen window and k-nearest-neighbour, and develop a learning approach employing distances to k-nearest neighbours of a point in the feature space. We show that these methods work as well, if not better than most recently published methods on benchmark datasets. We compare and contrast our analysis with the well-established diverse-density approach and its variants in recent literature, using benchmark datasets including the Musk, Andrews' and Corel datasets, along with a diabetic retinopathy pathology diagnosis dataset. Experimental results demonstrate that, while enjoying an intuitive interpretation and supporting fast learning, these method have the potential of delivering improved performance even for complex data arising from real-world applications.
Monocular Object Instance Segmentation and Depth Ordering With CNNs
Authors: Alexander G. Schwing, Sanja Fidler, Raquel Urtasun Ziyu Zhang
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper we tackle the problem of instance-level segmentation and depth ordering from a single monocular image. Towards this goal, we take advantage of convolutional neural nets and train them to directly predict instance-level segmentations where the instance ID encodes the depth ordering within image patches. To provide a coherent single explanation of an image we develop a Markov random field which takes as input the predictions of convolutional neural nets applied at overlapping patches of different resolutions, as well as the output of a connected component algorithm. It aims to predict accurate instance-level segmentation and depth ordering. We demonstrate the effectiveness of our approach on the challenging KITTI benchmark and show good performance on both tasks.
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Authors: Zhengdong Lu, Lifeng Shang, Hang Li Lin Ma
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content and one matching CNN modeling the joint representation of image and sentence. The matching CNN composes different semantic fragments from words and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. More specifically, our proposed m-CNNs significantly outperform the state-of-the-art approaches for bidirectional image and sentence retrieval on the Flickr8K and Flickr30K datasets.
Structural Kernel Learning for Large Scale Multiclass Object Co-Detection
Authors: Xuming He, Mathieu Salzmann Zeeshan Hayder
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: Exploiting contextual relationships across images has recently proven key to improve object detection. The resulting object co-detection algorithms, however, fail to exploit the correlations between multiple classes and, for scalability reasons are limited to modeling object instance similarity with relatively low-dimensional hand-crafted features. Here, we address the problem of multiclass object co-detection for large scale datasets. To this end, we formulate co-detection as the joint multiclass labeling of object candidates obtained in a class-independent manner. To exploit the correlations between objects, we build a fully-connected CRF on the candidates, which explicitly incorporates both geometric layout relations across object classes and similarity relations across multiple images. We then introduce a structural boosting algorithm that lets us exploits rich, high-dimensional deep network features to learn object similarity within our fully-connected CRF. Our experiments on PASCAL VOC 2007 and 2012 evidences the benefits of our approach over object detection with RCNN, single-image CRF methods and state-of-the-art co-detection algorithms.
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Authors: Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, Svetlana Lazebnik Bryan A. Plummer
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains linking mentions of the same entities in images, as well as  276k manually annotated bounding boxes corresponding to each entity. Such annotation is essential for continued progress in automatic image description and grounded language understanding. We present experiments demonstrating the usefulness of our annotations for text-to-image reference resolution, or the task of localizing textual entity mentions in an image, and for bidirectional image-sentence retrieval. These experiments confirm that we can further improve the accuracy of state-of-the-art retrieval methods by training with explicit region-to-phrase correspondence, but at the same time, they show that accurately inferring this correspondence given an image and a caption remains really challenging.
Predicting Depth, Surface Normals and Semantic Labels With a Common Multi-Scale Convolutional Architecture
Authors: Rob Fergus David Eigen
Published in: IEEE International Conference on Computer Vision (ICCV), 2015 (Main Conference Track), 2015

Abstract: In this paper we address three different computer vision tasks using a single basic architecture:  depth prediction, surface normal estimation, and semantic labeling.  We use a multiscale convolutional network that is able to adapt easily to each task using only small modifications, regressing from the input image to the output map directly.  Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation.  We achieve state-of-the-art performance on benchmarks for all three tasks.