Efficient Codebook and Factorization for Second Order Representation Learning

Learning rich and compact representations is an open topic in many fields such as word embedding , visual question-answering , object recognition or image retrieval . Although deep neural networks ( convolutional or not ) have made a major breakthrough during the last few years by providing hierarchical , semantic and abstract representations for all of these tasks , these representations are not necessary as rich as needed nor as compact as expected . Models using higher order statistics , such as bilinear pooling , provide richer representations at the cost of higher dimensional features . Factorization schemes have been proposed but without being able to reach the original compactness of first order models , or at a heavy loss in performances . This paper addresses these two points by extending factorization schemes to codebook strategies , allowing compact representations with the same dimensionality as first order representations , but with second order performances . Moreover , we extend this framework with a joint codebook and factorization scheme , granting a reduction both in terms of parameters and computation cost . This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics . 1 INTRODUCTION . Learning rich and compact representations is an open topic in many fields such as word embedding ( Mikolov et al . ( 2013 ) ) , visual question-answering ( Yang et al . ( 2016 ) ) , object recognition ( Szegedy et al . ( 2015 ) ) or image retrieval ( Opitz et al . ( 2017 ) ) . The standard approach extracts features from the input data ( text , image , etc . ) and builds a representation that will be next processed for a given task ( classification , retrieval , etc. ) . These features are usually extracted with deep neural networks and the representation is trained in an end-to-end manner . Recently , representations that compute first order statistics over input data have been outperformed by improved models that compute higher order statistics such as bilinear models . This embedding strategy generates richer representations and has been applied in a wide range of tasks : word embedding ( Clinchant & Perronnin ( 2013 ) ) , VQA ( Kim et al . ( 2017 ) ) , fine grained classification ( Wei et al . ( 2018 ) ) , etc . and gets state-of-the-art results . For instance , Bilinear models perform the best for fine grained visual classification tasks by producing efficient representations that model more details within an image than classical first order statistics ( Lin et al . ( 2015 ) ) . However , even if the increase in performances is unquestionable , second order models suffer from a collection of drawbacks : Their intermediate dimension increases quadratically with respect to input features dimension , they require a projection to lower dimension that is costly both in number of parameters and in computation , they are harder to train than first order models due to the increased dimension , they lack a proper adapted pooling scheme which leads to sub-optimal representations . The two main downsides , namely the high dimensional output representations and the sub-efficient pooling scheme , have been widely studied over the last decade . On one hand , the dimensionality issue has been studied through factorization scheme , either representation oriented such as Compact Bilinear Pooling ( Gao et al . ( 2016 ) ) and Hadamard Product for Low Rank Bilinear Pooling ( Kim et al . ( 2017 ) ) , or task oriented as Low-rank Bilinear Pooling ( Kong & Fowlkes ( 2017 ) ) . While these factorization schemes are efficient in term of computation cost and number of parameters , the intermediate representation is still too large ( typically 10k dimension ) to ease the training process and using lower dimension greatly deteriorate performances . On the other hand , it is well-known that global average pooling schemes aggregate unrelated features . This problem has been tackled by the use of codebooks such as VLAD ( Arandjelovic & Zisserman ( 2013 ) ) or , in the case of second-order information , Fisher Vectors ( Perronnin et al . ( 2010 ) ) . These strategies have been enhanced to be trainable in an end-to-end manner ( Arandjelovic et al . ( 2016 ) ; Tang et al . ( 2016 ) ) . However , using a codebook on end-to-end trainable second order features leads to an unreasonably large model , since the already large second order model has to be duplicated for each entry of the codebook . This is for example the case in MFAFVNet ( Li et al . ( 2017b ) ) for which the second order layer alone ( i.e. , without the CNN part ) already costs over 25M parameters and 40 GFLOP , or about as much as an entire ResNet50 . In this paper , we tackle both of these shortcomings ( intermediate representation cost and lack of proper pooling ) by exploring joint factorization and codebook strategies . Our main results are the following : - We first show that state-of-the-art factorization schemes can already be improved by the use of a codebook pooling , albeit at a prohibitive cost . - We then propose our main contribution , a joint codebook and factorization scheme that achieves similar results at a much reduced cost . Since our approach focuses on representation learning and is task agnostic , we validate it in a retrieval context on several image datasets to show the relevance of the learned representations . We show our model achieves competitive results on these datasets at a very reasonable cost . The remaining of this paper is organized as follows : in the next section , we present the related work on second order pooling , factorization schemes and codebook strategies . In section 3 , we present our factorization with the codebook strategy and how we improve its integration . In section 4 , we show an ablation study on the Stanford Online Products dataset ( Oh Song et al . ( 2016 ) ) . Finally , we compare our approach to the state-of-the-art methods on three image retrieval datasets ( Stanford Online Products , CUB-200-2001 , Cars-196 ) . 2 RELATED WORK . In this section , we focus on methods that use representations based on second-order information and we provide a comparison in terms of computational efficiency and number of parameters for the second-order layer . These second-order methods exploit either bilinear pooling ( section 2.1 ) and factorization schemes ( section 2.2 ) or codebook strategies ( section 2.3 ) . 2.1 SECOND-ORDER POOLING . In this section , we briefly review end-to-end trainable Bilinear pooling ( Lin et al . ( 2015 ) ) . This method extracts representations from the same image with two CNNs and computes the crosscovariance as representation . This representation outperforms its first-order version and other second-order representations such as Fisher Vectors ( Perronnin et al . ( 2010 ) ) once the global architecture is fine-tuned . However , bilinear pooling leads to a small improvement compared to secondorder pooling ( i.e. , the covariance of the CNN features ) at the cost of a higher computation . Most of recent works on bilinear pooling only focus on computing covariance of the extracted features , that is : y = ∑ i∈S xix T i = XX T ∈ Rd×d ( 1 ) where X = { xi ∈ Rd|i ∈ S } ∈ Rd×hw is the matrix of concatenated CNN features , h and w are the height and the width of the extracted feature map . Another formulation is the vectorized version of y obtained by computing the Kronecker product ( ⊗ ) of xi with itself : y = ∑ i∈S xi ⊗ xi = vec ( XXT ) ∈ Rd 2 ( 2 ) Due to the very high dimension of this representation , most recent works on bilinear pooling tend to improve this representation by providing new factorization scheme to reduce the computation . 2.2 FACTORIZATION SCHEMES . Recent works on bilinear pooling proposed factorization schemes with two objectives : avoiding the direct computation of second order features and reducing the high dimensionality output representation . Gao et al . ( 2016 ) proposed Compact Bilinear Pooling that tackles the high dimensionality of second-order features by analyzing two low-rank approximations of the equivalent polynomial kernel ( equation ( 2 ) from Gao et al . ( 2016 ) ) : 〈B ( X ) ; B ( Y ) 〉 = ∑ xs∈B ( X ) ∑ yu∈B ( Y ) 〈xs ; yu〉2 ≈ 〈 ∑ xs∈B ( X ) φ ( xs ) ; ∑ yu∈B ( Y ) φ ( yu ) 〉 ( 3 ) with φ the mapping function that approximates the kernel . This compact formulation allows to keep less than 4 % of the components with nearly no loss in performances compared to the uncompressed model . Kim et al . ( 2017 ) improved Compact Bilinear Pooling using Hadamard Product and generalized it for visual question-answering tasks . Kong & Fowlkes ( 2017 ) introduced Low Rank Bilinear Pooling ( LR-BP ) that takes advantage of SVM formulation by jointly training the network and the classifier . The authors propose an efficient factorization as they never compute directly the covariance features and have slightly better results compared to the uncompressed bilinear pooling or Compact Bilinear Pooling . However , as it is , their method is limited to classification with the SVM formulation and can not be used for other tasks . Li et al . ( 2017a ) introduced Factorized Bilinear Network ( FBN ) , an improved version of Compact Bilinear Pooling . FBN never directly computes the covariance matrix . Instead , it projects the features using a hyperplan and computes a quadratic form . The matrix of this quadratic form is supposed rank deficient to reduce the number of parameters and the computation cost with negligible loss in performances . Thus , the output representation ( or directly the number of classes for classification tasks ) is generated by concatenating these scalars for all projections . Wei et al . ( 2018 ) presented Grassmann Bilinear Pooling as a new factorization scheme . The objective is to take advantage of the rank deficient covariance matrix using Singular Value Decomposition ( SVD ) and then compute the classifier over these Grassmann manifolds . This factorization is efficient in the sense that it never directly computes the second-order representation and contrary to LR-BP , this formulation allows the construction of a representation , by replacing the number of classes by the representation dimension . In practice , however , they need to greatly reduce the input feature dimension due to the SVD complexity which is cubic in the feature dimension . In this work , we start from a similar factorization as Kim et al . ( 2017 ) detailed in section 3.1 . However , this factorization is improved by the introduction of a codebook strategy that allows smaller representation dimension and improves performances . 2.3 CODEBOOK STRATEGIES . An acknowledged drawback of pooling methods is that they pool unrelated features that may decrease performances . To cope with this observation , codebook strategies have been proposed and greatly improved performances by pooling only features that belong to the same codeword . The first representations that take advantage of codebook strategies are Bag of Words ( BoW ) and in the case of second order information Fisher Vectors ( Perronnin et al . ( 2010 ) ) . Fisher Vectors ( FVs ) extend the BoW framework by replacing the hard assignment of BoW by a Gaussian Mixture Model ( GMM ) and then compute the representation as an extension of the Fisher Kernel . In practice , covariance matrices are supposed to be diagonal which leads to representations of size N ( 2d + 1 ) where d is the dimension of the features and N is the codebook size . Tang et al . ( 2016 ) proposed FisherNet , an architecture that integrates FVs as differentiable layer . The proposed layer outperforms non-trainable FVs approach but nonetheless has the high output dimension of the original FV . Li et al . ( 2017b ) introduced MFA-FV network , a deep architecture which extends the MFA-FV of Dixit & Vasconcelos ( 2016 ) by producing a second order information embedding trainable in an endto-end manner . The proposed formulation takes advantage of both worlds : MFA-FV generates an efficient representation of non-linear manifolds with a small latent space and it can be trained in an end-to-end manner . The main drawbacks of their method is the direct computation of second-order features for each codeword ( computation cost ) , the raw projection of this covariance matrix into the latent space for each codeword ( computation cost and number of parameters ) , and finally the representation dimension . In the original paper , the proposed representation reaches 500k dimension , twice the already high dimension of Bilinear Pooling . For a more compact review , computation cost , number of parameters , use of codebook and/or factorization are sumed-up in table 1 . This table shows that , to our knowledge , no efficient factorization combined with codebook strategy has been proposed to exploit the richer representation due to codebook but at a small increase in terms of number of parameters or computation cost . As is shown in this table , our proposition combine the best of both worlds by providing a joint codebook and factorization optimization scheme with a similar number of parameters and computation cost to that of methods without codebook strategies .

Summary: This paper presents a way to combine existing factorized second order representations with a codebook style hard assignment. The number of parameters required to produce this encoded representation is shown to be very low. Like other factorized representations, the number of computations as well as the size of any intermediate representations is low. The overall embedding is trained for retrieval using a triplet loss. Results are shown on Stanford online, CUB and Cars-196 datasets.