Adaptive Region Pooling for Fine-Grained Representation Learning

Abstract: Fine-grained recognition aims to discriminate the sub-categories of the images within one general category. It is fundamentally difficult due to the requirement to extract fine-grained features from subtle regions. Nonetheless, a Convolutional Neural Network typically applies strided operations to downsample the representation, which would excessively spoil the feature resolution and lead to a significant loss of fine-grained information. In this paper, we propose Adaptive Region Pooling (ARP): a novel downsampling algorithm that makes the network only focus on a smaller but more critical region, and simultaneously increase the resolution of sub-sampled feature. ARP owns a trade-off mechanism that allows users to actively balance the scale of receptive field and the granularity of feature. Also, without any learning-based parameters, ARP provides the network a stabler training process and an earlier convergence. Extensive experiments qualitatively and quantitatively validate the effectiveness and efficiency of the proposed pooling operation and show superior performance against the state-of-the-arts in both the tasks of image classification and image retrieval.

Reviews and author responses:

The paper is tackling the topic of fine-grained representation. In contrast to comparable methods, ARP focus on sub-sampled via strided operation to represent fine-grained information. 
ARP is an intuitive strategy to extract critical regions and is more concise compared with multi-stage cascade architecture. This fact facilitates the efficiency. Strong points of the submission

- Well written paper, which is easy to follow, based on clear formulation for proposed method.

- The work provides a detailed description of the implementation. 

Weak points of the submission

- Lacking extensive use cases to demonstrate that ARP can crop a more discriminative region. It would be useful to provide a visualization that compares with other methods, especially the hard and tiny critical key regions. 

- While ARP assists in fine-grained representation, this idea technically is usable for other image tasks. Given sec.2 and compared baseline in sec.4, I am concerned that there are similar approaches proposed in the field of non-fine grained recognition. If this paper is to be accepted, I believe it needs a comprehensive survey about adaptive pooling strategy. - Why the baselines in Tables 1 and 2 do not include approaches in 2021, for example [1].

- It is good and necessary that compare ARP with a functionally similar method, in sec 4.3. STN is well-known but old. According to the author's survey, is there not a better and closer method in follow-up works of STN? 

- ARP seems to be a plug-and-play module, can the original results be boosted if apply it to existing architectures? This experiment could help prove the applicability of ARP.

[1] Huynh S V. A Strong Baseline for Vehicle Re-Identification[C]//CVPR. 2021: 4147-4154.

- **Motivation**.
The paper argues that 
  - (a) there are some small but important regions of an interesting object, which are usually ignored by the previous fine-grained methods;
  - (b) excessive reduction operations on image resolution fade the discriminative features.  


- **Method**.
According to the above observations, the paper proposes a pooling algorithm called Adaptive Region Pooling (ARP). This module has two procedures:
  - (a) learn to crop image regions;
  - (b) downsampling various size regions using bilinear operation.


- **Experiments**.
The proposed method is verified in two tasks: fine-grained classification and re-identification.  Experimental results show that the improvements are promising. But I have three considerations:
- (a) The proposed method is mainly based on the assumption that there is only one target object in an image. But this is a limitation of the paper. Is there a potential limitation that the module could not work well on the images with multiple isolated objects, or the ARP is sensitive to the noise in images?
- (b) Most of the datasets used in the paper's experiments are small. Please verify the proposed method on a large-scale dataset like iNaturalist, used in competitive methods, such as [TASN]. To verify on large-scale datasets is also helpful for erasing the above limitation.  The generalization of the proposed method is the biggest concern due to the fact that images used in the experiments have a clean and single target object. It could be better to use a large-scale dataset to verify the generalization ability of the proposed method.

This paper presents an adaptive region pooling (ARP) method for fine-grained representation learning. ARP crops the features with the estimated scale. The features are further downsampled to a consistent size through bilinear downsampling. Experiments conducted on two tasks validate the effectiveness of the proposed method. Strengths 
1) Both tasks of fine-grained image classification and vehicle re-identification are evaluated in this paper.
2) The code is released.

Weaknesses
1) The novelty is limited. In recent years, many pooling-based methods have been proposed to explore the fine-grained classification task, such as adaptive bilinear pooling, multi-domain pooling and so on. The ideas of ARP are similar to these methods. However, the details are not discussed in this paper. 

2) In the experiments, it seems unfair to compare the proposed method with the state-of-the-art methods. ARP is a general pooling operation for fine-grained representation learning. It is necessary to verify whether ARP helps improve these methods.

3) Many symbols in the paper are confusing, although they are case-sensitive or bold.

4) Many related references were not mentioned in this paper.
 See above comments. 

Based on weaknesses and strengths, I have concerns about the contributions and experimental evaluations. 

This paper proposes an adaptive region pooling (ARP) module that adaptively estimates the scale factors and crops the discriminative regions based on the estimated scale factors. It aims to focus on the most discriminative region and simultaneously contain more fine-grained information. The authors apply the ARP module to fine-grained image recognition and Vehicle Re-Id tasks. Indeed, the ARP module is an attentional mechanism to adaptively locate discriminative regions to facilitate learning fine-grained representation. It first crops the feature from the most critical region and then downsampled to a consistent size through bilinear downsampling. It is quite similar to current work [1], which proposes a context-aware attentional feature pooling (CAP) module that learns to locate discriminative regions with different scales and map to fixed-size feature maps. The advantage of the ARP compared with CAP should be presented and discussed. Besides, this work achieves quite well performance, e.g., 91.8% on CUB. The performance comparisons with this work should also be provided.

Figure 3 presents the network architecture that shows how to integrate the ARP module. However, why using such an architecture is not discussed. Besides, are there other ways to integrate the module? I think more discussion and experiments on how to integrate the APR module should be provided.

To compute the confidence map, the authors utilize instance normalization to normalize the feature maps. The motivation of this point is not well explained and its contribution is not verified.

[1] Behera et al., Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification, in AAAI, 2021. I think the advantage and performance comparisons with current works are not well clarified. The discussion and analysis about some crucial steps and components are not presented.

Meta-review: Four reviewers have reviewed this submission. Three of them recommended to reject the paper and one was marginally above the acceptance threshold. The authors have not responded to the criticisms or questions of reviewers. Among many concerns were the issues with the use of '`lean and single target object' images, lack of discussions on related models such as adaptive bilinear pooling and multi-domain pooling, lack of evaluations on datasets such as  large-scale iNaturalist. Given the above criticisms and the lack of authors' response, this submission falls below the acceptance bar.

Decision: Reject