An Out-of-the-box Full-network Embedding for Convolutional Neural Networks

Abstract: Transfer learning for feature extraction can be used to exploit deep representations in contexts where there is very few training data, where there are limited computational resources, or when tuning the hyper-parameters needed for training is not an option. While previous contributions to feature extraction propose embeddings based on a single layer of the network, in this paper we propose a full-network embedding which successfully integrates convolutional and fully connected features, coming from all layers of a deep convolutional neural network. To do so, the embedding normalizes features in the context of the problem, and discretizes their values to reduce noise and regularize the embedding space. Significantly, this also reduces the computational cost of processing the resultant representations. The proposed method is shown to outperform single layer embeddings on several image classification tasks, while also being more robust to the choice of the pre-trained model used for obtaining the initial features. The performance gap in classification accuracy between thoroughly tuned solutions and the full-network embedding is also reduced, which makes of the proposed approach a competitive solution for a large set of applications.

Reviews and author responses:

This paper proposes an out-of-the-box embedding for image classification task. Instead of taking one single layer output from pre-trained network as the feature vector for new dataset, the method first extracts the activations from all the layers, then runs spatial average pooling on all convolutional layers, then normalizes the feature and uses two predefined thresholds to discretize the features to {-1, 0, 1}. Final prediction is learned through a SVM model using those embeddings. Experimental results on nine different datasets show that this embedding outperforms baseline of using one single layer. I think in general this paper lacks novelty and it shouldn't be surprising that activations from all layers should be more representative than one single layer representation. Moreover, in Table 4, it shows that discretization actually hurts the performance. It is also very heuristic to choose the two thresholds.  



The paper addresses the scenario when using a pretrained deep network as learnt feature representation for another (small) task where retraining is not an option or not desired. In this situation it proposes to use all layers of the network to extract feature from, instead of only one layer. 
Then it proposes to standardize different dimensions of the features based on their response on the original task. Finally, it discretize each dimension into {-1, 0, 1} to compress the final concatenated feature representation. 
Doing this, it shows improvements over using a single layer for 9 target image classification datasets including object, scene, texture, material, and animals.

The reviewer does not find the paper suitable for publication at ICLR due to the following reasons:
- The paper is incremental with limited novelty.
- the results are not encouraging
- the pipeline of standardization, discretization is relatively costly, the final feature vector still large. 
- combining different layers, as the only contribution of the paper, has been done in the literature before,  for instance:
“The Treasure beneath Convolutional Layers: Cross-convolutional-layer Pooling
for Image Classification” CVPR 2016


Paper claims to propose a deep transfer learning method. There are several reasons not to consider this paper for ICLR at this point.

Paper is badly written and the problem it tries to solve is not clearly stated.
Proposed feature embedding is incremental (lack of novelty and technical contribution)
Obtained results are encouraging but not good enough.
Lack of experimental validation.
I think paper can be improved significantly and is not ready for publication at this point.



Meta-review: Three reviewers recommended rejection, and there was no rebuttal.

Decision: rejected-papers