Large-scale classification of structured objects using a CRF with deep class embedding

1 INTRODUCTION . Object recognition is one of the fundamental problems in computer vision . It involves finding and identifying objects in images , and plays an important role in many real-world applications such as advanced driver assistance systems , military target detection , diagnosis with medical images , video surveillance , and identity recognition . Over the past few years deep convolutional neural networks ( CNN ) have led to remarkable progress in image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , and resulted in reliable appearance-based detectors ; e.g. , ( Ren et al. , 2015 ; Liu et al. , 2016 ; Redmon & Farhadi , 2017 ; Lin et al. , 2018 ) . Fine-grained object recognition aims to identify subcategory object classes , which includes finding subtle differences among visually similar subcategories such as dog breeds , product brands , car models , etc . The differences between classes are often small but always visually measurable , making visual recognition challenging but possible . Some of these datasets ( e.g . UT-Zap50K ( Yu & Grauman , 2014 ) ) provide each class with an in-vitro image : a catalog or studio image isolated and captured under ideal imaging conditions ; other datasets ( e.g . Caltech-UCSD Birds ( Wah et al. , 2011 ) , Stanford Dogs ( Khosla et al. , 2011 ) , FGVC-Aircraft ( Maji et al. , 2013 ) ) provide each class with several in-situ images , captured in natural real-world environments . Nonetheless , the image quality is mostly satisfactory for the task of visual classification . In fact , recent studies achieved good performance on fine-grained tasks ( Lin et al. , 2015 ; Zhang et al. , 2014 ; Peng et al. , 2018 ) . However , the problem remains extremely difficult when the dataset categories are nearly identical in terms of their visual appearance . In this case , the object categories may be virtually indistinguishable , since the discriminant features are often masked by inadequate observation or visual artifacts . Here we present an ultrafine-grained structured classification dataset ; unlike other fine-grained classification datasets , our images are in-situ low-resolution cropped patches whose classes are often virtually indistinguishable by visual inspection alone . Therefore , incorporating additional sources of information to the classifier is imperative . Since the object-patches originate from larger scenes , we can model contextual relations between the objects based on their geometric layout . This study tackles the challenge of fine-grained , large-scale structured classification , and describes a novel , state-of-the-art technique for this task . We address the problem of classifying a sequence of objects based on their visual appearance and their relative locations . Our dataset contains photos of retail store product displays , taken in varying settings and viewpoints . Our task is to identify the class of each product at the front of the shelves . The dataset is exclusively characterized by having a distinct geometric object structure made up of sequences of shelves , a large number of classes , and very subtle visual differences between groups of classes in that some classes only differ in size or minor design details . The unique challenges in this task involve ( a ) large-scale classification : handling the large number of classes in the dataset , and ( b ) ultrafine-grained structured classification : the fact that the classes are not clearly distinguishable by their appearance but rather by their context . For example , products with an identical appearance but different container volumes are considered different classes ( See Fig . 1 and 2 ) . Context has been used to improve performance for image understanding tasks in various ways ( Torralba , 2003 ; Divvala et al. , 2009 ; Felzenszwalb et al. , 2010 ) . Graphical models have been widely applied to visual and auditory analysis tasks , by jointly modeling local features , and contextual relations . The tasks addressed by these models include image segmentation and object recognition ( Chen et al. , 2018 ; Yao et al. , 2012 ; Zheng et al. , 2015 ) , as well as speech ( Wang & Wang , 2012 ) , music ( Korzeniowski & Widmer , 2016 ) , text ( Chen et al. , 2016 ) and video analysis ( Hu et al. , 2014 ) . Few studies have applied deep learning features or detection results to context models : Chen et al . ( 2015 ) explored several techniques to learn structured models jointly with deep features that form MRF potentials . Chu & Cai ( 2018 ) evaluated the performance of a joint CRF model on Faster R-CNN ( Ren et al. , 2015 ) detection results using an a-priori statistical summary for the pairwise potentials . Korzeniowski & Widmer ( 2016 ) introduced a two-stage learning model for musical chord recognition : one network learns a single-frame representation , and the other learns the potentials of a linear-chain CRF model using the frame-representations as the CRF input . These models use the vanilla CRF parametrization , which includes pairwise potentials to represent object-pair interactions . They allocate a different parameter to each class pair . This approach , which ignores class similarities , is only sufficient for small sets of distinct classes . In effect , they have solely been tested on OCR datasets , which contain 26 classes ( Chen et al. , 2015 ) , a chord-recognition dataset with 25 classes ( Korzeniowski & Widmer , 2016 ) and PASCAL VOC 2007 with 20 classes ( Chu & Cai , 2018 ) . However , this formulation is not sufficient for a large class-set that contains visually similar classes . Our dataset , which includes many visually similar categories , nearly a thousand classes and a million possible pairwise transitions overall , requires a more advanced learning mechanism . Furthermore , whereas in most previous object recognition studies the visual information was dominant , in our task , context information also makes a significant contribution . In this study we present a CRF based method that explicitly learns the embedding of classes with respect to their neighbor ’ s class and appearance . This is achieved by factorizing the CRF pairwise potential matrix to impose the structure of class embedding in a low-dimensional space . Our model learns the factorized parameters , and yields a joint contextual-visual embedding of the classes . The factorization drastically increases the learning capacity of contextual information , but also forms a multimodal likelihood function which is more challenging to optimize . To overcome this , we develop a local surrogate likelihood and apply the proper regularization required for convergence . To train the network , we introduce a pairwise softmax architecture that optimizes a local approximation of the likelihood . Since the global factorized loss function is not convex , we favor optimizing the approximate surrogate likelihood , which allows us to include batch-norm related regularization for the object samples , and achieve dramatic improvement not only in training time and model simplicity but also in terms of the overall performance of the trained model . At test time , dynamic programming techniques are used for efficient exact inference of the classes . The contributions of this work are the following : 1 . Combining deep class embedding into a CRF formulation that enables handling datasets with a huge number of classes . 2 . An approximated-likelihood training procedure that is both computational efficient and , unlike exact CRF likelihood , enables us to incorporate batch-normalization into the training procedure . 2 CRF WITH CLASS EMBEDDING . 2.1 MODEL FORMULATION . Our input data are sequences of image patches which correspond to horizontal layouts of individual product , as demonstrated in Fig . 3 . We want to predict the sequence of the target labels y = [ y1 , · · · , yn ] , given a sequence of observations x = [ x1 · · · , xn ] . Standard classification approaches use a CNN to predict each object-level observation individually , which implicitly assumes independence between object samples . In order to include context in the classification process , we model the sequences as a CRF . Linear-chain Conditional Random Field ( LC-CRF ) ( Lafferty et al. , 2001 ) is a type of discriminative undirected probabilistic graphical model , whose conditional distribution p ( y|x ) obeys a conditional Markov property . The joint probability distribution of a linear-chain CRF is : p ( y|x ) = 1 Z n∏ t=1 ϕ ( yt , xt , yt−1 ) ( 1 ) where x is the sequence of observation feature vectors , y is the corresponding sequence of the target labels , ϕ the model ’ s potential function , Z the partition function defined as the global probability normalization over all possible sequence label-assignments of length n , and y0 = 0 . Assume that the potential function is defined as : ϕ ( yt , xt , yt−1 ) = exp ( y > t−1Pyt + x > t Uyt + b > yt ) ( 2 ) where matrix P the pairwise potentials matrix , U the unary potentials , and vector b the label bias , are all model parameters , and we use a one-hot encoding for the labels . The likelihood function , therefore , is log-linear and concave . We can train a local CNN to classify individual objects , and then interpret the hidden layer activations as a non-linear representation of the input image . Similar to the concept of transfer-learning , we can now discard the CNN softmax layer , and use the convolutional layers to compute the featurevectors of the input images . For image-patch xt we define the feature vector ht = h ( xt ) as the activations of the last hidden fully-connected or global-average-pooling ( GAP ) layer ( Bengio et al. , 2013 ; Lin et al. , 2013 ) , and use it as the CRF input observation feature vector : ϕ ( yt , ht , yt−1 ) = exp ( y > t−1Pyt + h > t Uyt + b > yt ) . ( 3 ) The score function ( 3 ) is still concave , but its input is computed from a non-concave source , a deep CNN . The rationale for using a deep representation for the input images is clear : as introduced by Krizhevsky et al . ( 2012 ) , the immense complexity of the visual object recognition task requires a model with a very large learning capacity . Convolutional layers provide the structure required for learning visual features of the unary input . We aim to craft a suitable structure to learn the pairwise contextual relations as well . CRF was originally applied to language processing tasks such as Part of Speech ( POS ) tagging and Named Entity Recognition ( NER ) ( Lafferty et al. , 2001 ) . In most applications of CRF to either language or image understanding , there are no more than a few dozen different classes . In our dataset , we have hundreds of classes . The pairwise transition between two classes has nearly a million possible states , whereas the CRF function ( 3 ) has a log-linear form , and contains a single parameter per transition ordered-pair . In order to properly learn and generalize the massive variety of possible neighboring patterns , we enforce a structure on the pairwise potential matrix : the goal is to learn neighboring-class embedding in a feature vector space . For this purpose , we define a low-dimensional decomposition of the pairwise potential matrix P as the product of the left-side neighbor embedding matrix R and the class embedding matrix Q : P = R > Q . ( 4 ) The columns of Q are low-dimensional embeddings of the target classes , and the columns of R are embeddings of the classes of the left-side object . Assigning the matrix factorization ( 4 ) to the CRF potential function ( 3 ) we get : ϕ ( yt , ht , yt−1 ) = exp ( y > t−1R > Qyt + h > t Uyt + b > yt ) . ( 5 ) The objective function is no longer linear or concave with respect to the network parameters , but deep learning training techniques have been shown to yield good results for non-convex optimization tasks ( Choromanska et al. , 2015 ) . Hence , we apply deep learning approaches not only for the input image representations , but also for the neighboring transition parameters .

Summary: This paper introduces a new dataset consisting of images of various objects placed on store shelves that are labeled with object boundaries and what are described as “ultrafine-grained” class labels. The accompanying task is to predict the labels of each object given the individual images as well as their spatial layout relative to each other. To solve this task, a deep structured model is used consisting of CNN features for each image which are fed into a linear-chain CRF. To better deal with the large number of classes, pairwise potentials are represented as the multiplication of two lower-rank matrices which represent a sort of “class embedding” for each potential label. Training efficiency is improved by considering an objective based on a form of piecewise pseudolikelihood, which allows for training-time inference to be conducted with linear complexity relative to the number of labels. This objective also allows for easy use of batch normalization for the input features to the CRF model. This model/training procedure are compared against a number of models/training procedures to demonstrate its utility.