Large-scale classification of structured objects using a CRF with deep class embedding

1 INTRODUCTION . Object recognition is one of the fundamental problems in computer vision . It involves finding and identifying objects in images , and plays an important role in many real-world applications such as advanced driver assistance systems , military target detection , diagnosis with medical images , video surveillance , and identity recognition . Over the past few years deep convolutional neural networks ( CNN ) have led to remarkable progress in image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , and resulted in reliable appearance-based detectors ; e.g. , ( Ren et al. , 2015 ; Liu et al. , 2016 ; Redmon & Farhadi , 2017 ; Lin et al. , 2018 ) . Fine-grained object recognition aims to identify subcategory object classes , which includes finding subtle differences among visually similar subcategories such as dog breeds , product brands , car models , etc . The differences between classes are often small but always visually measurable , making visual recognition challenging but possible . Some of these datasets ( e.g . UT-Zap50K ( Yu & Grauman , 2014 ) ) provide each class with an in-vitro image : a catalog or studio image isolated and captured under ideal imaging conditions ; other datasets ( e.g . Caltech-UCSD Birds ( Wah et al. , 2011 ) , Stanford Dogs ( Khosla et al. , 2011 ) , FGVC-Aircraft ( Maji et al. , 2013 ) ) provide each class with several in-situ images , captured in natural real-world environments . Nonetheless , the image quality is mostly satisfactory for the task of visual classification . In fact , recent studies achieved good performance on fine-grained tasks ( Lin et al. , 2015 ; Zhang et al. , 2014 ; Peng et al. , 2018 ) . However , the problem remains extremely difficult when the dataset categories are nearly identical in terms of their visual appearance . In this case , the object categories may be virtually indistinguishable , since the discriminant features are often masked by inadequate observation or visual artifacts . Here we present an ultrafine-grained structured classification dataset ; unlike other fine-grained classification datasets , our images are in-situ low-resolution cropped patches whose classes are often virtually indistinguishable by visual inspection alone . Therefore , incorporating additional sources of information to the classifier is imperative . Since the object-patches originate from larger scenes , we can model contextual relations between the objects based on their geometric layout . This study tackles the challenge of fine-grained , large-scale structured classification , and describes a novel , state-of-the-art technique for this task . We address the problem of classifying a sequence of objects based on their visual appearance and their relative locations . Our dataset contains photos of retail store product displays , taken in varying settings and viewpoints . Our task is to identify the class of each product at the front of the shelves . The dataset is exclusively characterized by having a distinct geometric object structure made up of sequences of shelves , a large number of classes , and very subtle visual differences between groups of classes in that some classes only differ in size or minor design details . The unique challenges in this task involve ( a ) large-scale classification : handling the large number of classes in the dataset , and ( b ) ultrafine-grained structured classification : the fact that the classes are not clearly distinguishable by their appearance but rather by their context . For example , products with an identical appearance but different container volumes are considered different classes ( See Fig . 1 and 2 ) . Context has been used to improve performance for image understanding tasks in various ways ( Torralba , 2003 ; Divvala et al. , 2009 ; Felzenszwalb et al. , 2010 ) . Graphical models have been widely applied to visual and auditory analysis tasks , by jointly modeling local features , and contextual relations . The tasks addressed by these models include image segmentation and object recognition ( Chen et al. , 2018 ; Yao et al. , 2012 ; Zheng et al. , 2015 ) , as well as speech ( Wang & Wang , 2012 ) , music ( Korzeniowski & Widmer , 2016 ) , text ( Chen et al. , 2016 ) and video analysis ( Hu et al. , 2014 ) . Few studies have applied deep learning features or detection results to context models : Chen et al . ( 2015 ) explored several techniques to learn structured models jointly with deep features that form MRF potentials . Chu & Cai ( 2018 ) evaluated the performance of a joint CRF model on Faster R-CNN ( Ren et al. , 2015 ) detection results using an a-priori statistical summary for the pairwise potentials . Korzeniowski & Widmer ( 2016 ) introduced a two-stage learning model for musical chord recognition : one network learns a single-frame representation , and the other learns the potentials of a linear-chain CRF model using the frame-representations as the CRF input . These models use the vanilla CRF parametrization , which includes pairwise potentials to represent object-pair interactions . They allocate a different parameter to each class pair . This approach , which ignores class similarities , is only sufficient for small sets of distinct classes . In effect , they have solely been tested on OCR datasets , which contain 26 classes ( Chen et al. , 2015 ) , a chord-recognition dataset with 25 classes ( Korzeniowski & Widmer , 2016 ) and PASCAL VOC 2007 with 20 classes ( Chu & Cai , 2018 ) . However , this formulation is not sufficient for a large class-set that contains visually similar classes . Our dataset , which includes many visually similar categories , nearly a thousand classes and a million possible pairwise transitions overall , requires a more advanced learning mechanism . Furthermore , whereas in most previous object recognition studies the visual information was dominant , in our task , context information also makes a significant contribution . In this study we present a CRF based method that explicitly learns the embedding of classes with respect to their neighbor ’ s class and appearance . This is achieved by factorizing the CRF pairwise potential matrix to impose the structure of class embedding in a low-dimensional space . Our model learns the factorized parameters , and yields a joint contextual-visual embedding of the classes . The factorization drastically increases the learning capacity of contextual information , but also forms a multimodal likelihood function which is more challenging to optimize . To overcome this , we develop a local surrogate likelihood and apply the proper regularization required for convergence . To train the network , we introduce a pairwise softmax architecture that optimizes a local approximation of the likelihood . Since the global factorized loss function is not convex , we favor optimizing the approximate surrogate likelihood , which allows us to include batch-norm related regularization for the object samples , and achieve dramatic improvement not only in training time and model simplicity but also in terms of the overall performance of the trained model . At test time , dynamic programming techniques are used for efficient exact inference of the classes . The contributions of this work are the following : 1 . Combining deep class embedding into a CRF formulation that enables handling datasets with a huge number of classes . 2 . An approximated-likelihood training procedure that is both computational efficient and , unlike exact CRF likelihood , enables us to incorporate batch-normalization into the training procedure . 2 CRF WITH CLASS EMBEDDING . 2.1 MODEL FORMULATION . Our input data are sequences of image patches which correspond to horizontal layouts of individual product , as demonstrated in Fig . 3 . We want to predict the sequence of the target labels y = [ y1 , · · · , yn ] , given a sequence of observations x = [ x1 · · · , xn ] . Standard classification approaches use a CNN to predict each object-level observation individually , which implicitly assumes independence between object samples . In order to include context in the classification process , we model the sequences as a CRF . Linear-chain Conditional Random Field ( LC-CRF ) ( Lafferty et al. , 2001 ) is a type of discriminative undirected probabilistic graphical model , whose conditional distribution p ( y|x ) obeys a conditional Markov property . The joint probability distribution of a linear-chain CRF is : p ( y|x ) = 1 Z n∏ t=1 ϕ ( yt , xt , yt−1 ) ( 1 ) where x is the sequence of observation feature vectors , y is the corresponding sequence of the target labels , ϕ the model ’ s potential function , Z the partition function defined as the global probability normalization over all possible sequence label-assignments of length n , and y0 = 0 . Assume that the potential function is defined as : ϕ ( yt , xt , yt−1 ) = exp ( y > t−1Pyt + x > t Uyt + b > yt ) ( 2 ) where matrix P the pairwise potentials matrix , U the unary potentials , and vector b the label bias , are all model parameters , and we use a one-hot encoding for the labels . The likelihood function , therefore , is log-linear and concave . We can train a local CNN to classify individual objects , and then interpret the hidden layer activations as a non-linear representation of the input image . Similar to the concept of transfer-learning , we can now discard the CNN softmax layer , and use the convolutional layers to compute the featurevectors of the input images . For image-patch xt we define the feature vector ht = h ( xt ) as the activations of the last hidden fully-connected or global-average-pooling ( GAP ) layer ( Bengio et al. , 2013 ; Lin et al. , 2013 ) , and use it as the CRF input observation feature vector : ϕ ( yt , ht , yt−1 ) = exp ( y > t−1Pyt + h > t Uyt + b > yt ) . ( 3 ) The score function ( 3 ) is still concave , but its input is computed from a non-concave source , a deep CNN . The rationale for using a deep representation for the input images is clear : as introduced by Krizhevsky et al . ( 2012 ) , the immense complexity of the visual object recognition task requires a model with a very large learning capacity . Convolutional layers provide the structure required for learning visual features of the unary input . We aim to craft a suitable structure to learn the pairwise contextual relations as well . CRF was originally applied to language processing tasks such as Part of Speech ( POS ) tagging and Named Entity Recognition ( NER ) ( Lafferty et al. , 2001 ) . In most applications of CRF to either language or image understanding , there are no more than a few dozen different classes . In our dataset , we have hundreds of classes . The pairwise transition between two classes has nearly a million possible states , whereas the CRF function ( 3 ) has a log-linear form , and contains a single parameter per transition ordered-pair . In order to properly learn and generalize the massive variety of possible neighboring patterns , we enforce a structure on the pairwise potential matrix : the goal is to learn neighboring-class embedding in a feature vector space . For this purpose , we define a low-dimensional decomposition of the pairwise potential matrix P as the product of the left-side neighbor embedding matrix R and the class embedding matrix Q : P = R > Q . ( 4 ) The columns of Q are low-dimensional embeddings of the target classes , and the columns of R are embeddings of the classes of the left-side object . Assigning the matrix factorization ( 4 ) to the CRF potential function ( 3 ) we get : ϕ ( yt , ht , yt−1 ) = exp ( y > t−1R > Qyt + h > t Uyt + b > yt ) . ( 5 ) The objective function is no longer linear or concave with respect to the network parameters , but deep learning training techniques have been shown to yield good results for non-convex optimization tasks ( Choromanska et al. , 2015 ) . Hence , we apply deep learning approaches not only for the input image representations , but also for the neighboring transition parameters .

Summary: This paper proposed to tackle a large-scale fine-grained object classification problem by approximated CRF. The main motivation is to exploit the spatial conference of object labels to reduce noises in the instance-wise prediction. To this end, the task is formulated by sequential inference problem using CRF. To speed up training, several techniques are applied such as factorized pairwise-potential and approximation of CRF objective.