Adaptive Region Pooling for Fine-Grained Representation Learning

1 INTRODUCTION . The main goal of fine-grained recognition is to discriminate the sub-categories of the images that belong to a same general class . Its practical applications include fine-grained image classification ( Welinder et al. , 2010 ; Krause et al. , 2013 ; Maji et al. , 2013 ) , re-identification ( Zheng et al. , 2017b ; Liu et al. , 2016b ; a ) , etc . These tasks are fundamentally challenging due to considerable inter-class similarity . Consequently , an ideal feature extractor should be capable of extracting discriminative fine-grained features from subtle image regions , such as the beak to classify the bird species or the stickers on the windshield to identify the vehicle . While the general design of Convolutional Neural Network ( CNN ) ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ) applies either strided convolution or strided pooling as downsampling operations to reduce the computational requirement of the network and also increase the receptive field of subsequent convolutions , these operations , however , have been substantiated to be harmful to fine-grained recognition ( Sun et al. , 2018 ; Luo et al. , 2019a ) . Its disadvantage is two-fold : 1 ) false assumption on spatial equality which would overemphasize the background region while neglecting the smaller but more discriminative regions and 2 ) excessive reduction of resolution which would spoil the granularity of feature . To cope with the problem , existing approaches ( Zhang et al. , 2014 ; Fu et al. , 2017 ; Chen & Deng , 2019 ; Zhang et al. , 2019 ) introduce cropping operation that guides the network to only focus on the most critical region while preserving more details in the sampled part . However , to maintain computational complexity , full conservation of feature resolution would instead restrict a limited scale , or area , of the cropped region . Comprehensively , as shown in Figure 1 ( a ) ( c ) , the problems of two commonly used downsamplings can be summarized into a trade-off problem between the scale of receptive field and the granularity of representation . To tackle the problem , in this work , we introduce a novel pooling algorithm , named Adaptive Region Pooling ( ARP ) , to bridge strided and cropping operations . It consists of two steps . First , ARP automatically crops the feature from the most critical region with a well-estimated and adaptive cropping scale . Second , after being cropped into different sizes , the feature is further downsampled to a consistent size through bilinear downsampling . As illustrated in Figure 1 ( b ) , with the meticulous evaluation of cropping scale , ARP can better handle the trade-off issue and leverage the beneficial properties of both strided and cropping operations . Besides , ARP also features other three advantages . First , it is incorporated with a controllable mechanism that supports users to actively reach a more satisfying balance in the trade-off problem , as in two cases in Figure 1 ( b ) . Second , compared to the previous work ( Lin et al. , 2015a ; Huang et al. , 2016 ; Kim et al. , 2018 ; Zheng et al. , 2019 ) which implement the cropping operations by an additional neural network , ARP utilizes a more efficient yet effective operation without any learning-based parameters . Hence , it makes the network achieve lower computational overhead , experience a stabler training process , and converge more quickly . Finally , ARP can be simply plugged into any CNN layer and the network can be optimized in a one-stage end-to-end manner , which is completely identical to the training of original CNN backbone . At last , we demonstrate that ARP could facilitate fine-grained recognition through our Multiple Scale and Granularity Network ( MSGN ) . In MSGN , ARP serves as a downsampling alternative to extract the feature of finer details complementary to the coarse feature provided by the typical downsampling method . The contributions of this paper can be summarized as follows : • We introduce Adaptive Region Pooling ( ARP ) : a novel downsampling that smoothly bridges strided pooling and cropping operations . The sub-sampled representation focuses on the most discriminative region and simultaneously contains more fine-grained information . • To our knowledge , ARP is the first pooling algorithm with an adjustable trade-off mechanism that allows users to apply the human domain knowledge of the target task to strike a more desirable balance between the scale of receptive field and the granularity of downsampled feature . • With the usage of ARP , MSGN outperforms the existing frameworks in terms of both effectiveness and efficiency on multiple benchmarks for both the tasks of image classification and retrieval . 2 RELATED WORK . Fine-grained recognition focuses on learning discriminative representations to classify or identify the images within a same general category . There are two common applications : re-identification and fine-grained image classification . Re-identification ( re-ID ) studies the problem of identifying the individuals of persons ( Zheng et al. , 2015 ; 2017b ; Ristani et al. , 2016 ) or vehicles ( Liu et al. , 2016b ; c ; a ) in different camera views . To distinguish two similar targets with only slight differences , most of the existing methods ( Yao et al. , 2019 ; Kalayeh et al. , 2018 ; Liu et al. , 2021 ; Wang et al. , 2017 ; Zhou & Shao , 2018 ; Chen et al. , 2020b ) adopt spatial attentive mechanism to emphasize the features extracted from critical parts , such as the head for person re-ID or the tires for vehicle re-ID . While previous work get significant improvement , they generally suffer from massive loss of fine-grained details when CNN backbone downsamples the feature through strided operations . Sun et al . ( 2018 ) and Luo et al . ( 2019a ) both find a huge performance gain in the task of person re-ID , numerically 1.4 % for mAP on Market1501 ( Zheng et al. , 2015 ) , by simply removing the pooling operation in the last CNN layer . Fine-grained image classification ( FGIC ) aims to classify the sub-classes of images , such as the species of birds ( Welinder et al. , 2010 ) or the models of vehicles ( Krause et al. , 2013 ) . Compared to re-ID , the datasets of FGIC have a severe problem of image misalignment which impels lots of work to apply another sampling operation , cropping , to align the image or representation and also guide the network to focus on the sampled region . Several prior work ( Welinder et al. , 2010 ; Zhang et al. , 2013 ; 2014 ; Lin et al. , 2015a ; Huang et al. , 2016 ; Kim et al. , 2018 ; He et al. , 2019 ) leverage extra object detection datasets or human-labeled bounding box annotations to learn the localization of discriminative regions . However , such auxiliary labels are hard to acquire in the real-world and large-scale scenarios . Recently , there have been emerging work with a more general setting by proposing unsupervised approaches . Jaderberg et al . ( 2015 ) introduce a differentiable STN that can learn to well align the images or representations by themselves . While it can perform well on the digit datasets ( LeCun et al. , 1998 ; Netzer et al. , 2011 ) , lack of explicit guidance leads to the unstable training process and being hard to converge on more sophisticated tasks ( see Section 4.3 for more details and discussions ) . As the following work , Fu et al . ( 2017 ) propose RACNN which is a cascade coarse-to-fine CNN model . The similar architecture is also adopted by Chen & Deng ( 2019 ) and Zhang et al . ( 2019 ) . Although they show the great capability of cropping the discriminative region , the cascade architecture requires multi-stage training for better optimization and struggles with the requirement of heavy computation . In this work , we also introduce a sampling operation with an unsupervised approach ; however , Adaptive Region Pooling ( ARP ) contains neither additional neural network nor extra learning-based parameters and makes the network using ARP more computationefficient and able to be optimized in a simple one-stage end-to-end manner . 3 PROPOSED METHODOLOGY . In this section , we introduce the proposed pooling algorithm in three parts . First , in Section 3.1 , Region Pooling ( RP ) considers a simplified task which aims to crop the hidden representation from the most critical region with a fixed cropping scale . Then , in Section 3.2 , Adaptive Region Pooling ( ARP ) estimates the adaptive cropping scale for each representation to better capture the interesting object . Finally , we introduce Multiple Scale and Granularity Network ( MSGN ) as an example of network architecture with the usage of ARP in Section 3.3 . 3.1 REGION POOLING . Given a CNN feature and a fixed downsampling rate , the main goal of Region Pooling ( RP ) is to crop the feature from the most critical region into a consistent scale . An intuitive approach is to use an extra network to predict the cropping center which nevertheless , often suffers from the requirement of expensive bounding box annotations or complicated multi-stage learning strategy to train the network . In contrast , RP uses a more efficient yet effective operation with two steps . In the first step , as shown in Figure 2 ( a ) , RP would evaluate the significance score for each location and generate the keypoint confidence map C based on the C-dim feature vectors , where C denotes the number of channels . Here , we define “ significance ” in an engineering sense as follows : a location , whose feature vector includes more highly-activated neurons , is more significant . The physical meaning is that a channel-wise feature map is actually a detector of a certain semantic object ( Chen et al. , 2017 ; 2020a ) , such as a bird ’ s beak or wings . For a well-trained CNN , most of the channels would detect the objects which are more beneficial to the target task ; thus , the locations activated by more channels typically cover critical semantic objects and are more significant . Based on the definition , we then give a formal formulation as follows . For the input feature F ∈ RC×H×W with width W , height H and C channels and its element ( or activated value ) denoted as x ( i , j , k ) at ith channel , jth row and kth column , in order to balance the influence of each channel in the subsequent operations , we first standardize each channel-wise feature map as : x̂ ( i , j , k ) = x ( i , j , k ) − µ ( i ) σ ( i ) , where µ ( i ) = H∑ j W∑ k x ( i , j , k ) H ·W , σ2 ( i ) = H∑ j W∑ k ( x ( i , j , k ) − µ ( i ) ) 2 H ·W , ( 1 ) and x̂ represents the element of the standardized feature . Such standardization can be simply implemented by Instance Normalization ( Ulyanov et al. , 2016 ) . Afterwards , we evaluate the significance score of each location based on the average of their feature vectors with the form as : s ( j , k ) = C∑ i=1 x̂ ( i , j , k ) C ( 2 ) where s ( j , k ) represents the significance score of the location ( j , k ) . Complied with our definition , Equation 2 assigns larger s for the locations that are highly activated in more channels . Next , we estimate the confidence of a location to be a “ keypoint ” , or a critical location , by Sigmoid function σ ( · ) . Specifically , we generate a keypoint confidence map C ∈ RH×W by : C = [ c ( 1,1 ) , ... , c ( H , W ) ] = [ σ ( s ( 1,1 ) ) , ... , σ ( s ( H , W ) ) ] ( 3 ) where c ( j , k ) denotes the keypoint confidence at the location ( j , k ) . We visualize the generated keypoint confidence maps C in Figure 5 and 6 . If we project them to the original images , the locations with larger keypoint confidences effectively represent the more discriminative regions , such as the beak and colorful feather of a bird or the bumper and lamp of a vehicle . In the second step , as in Figure 2 ( c ) , given a fixed downsampling rate DSR = [ DSRH , DSRW ] , RP would crop the most critical region with height ( H · DSRH ) and width ( W · DSRW ) . To this end , we apply a convolution operation on the keypoint confidence map C with a kernel K ∈ R ( H·DSRH ) × ( W ·DSRW ) . The region with the largest weighted summation of keypoint confidences would be selected as the cropped region , which can be written as : ( j∗ , k∗ ) = arg max ( j , k ) ( K C ( j , k ) ) ( 4 ) where represents inner product , C ( j , k ) is the cropped keypoint confidence map centered at the location ( j , k ) , and ( j∗ , k∗ ) indicates the center location of the selected cropped region . Discussion of kernel K. Empirically , we find that using Gaussian filter as kernel K yields the best performance . A potential explanation is that it tends to select the region with more keypoints locating near the center of cropped region and hence , also solves the problem of feature misalignment .

Summary: This paper proposes an adaptive region pooling (ARP) module that adaptively estimates the scale factors and crops the discriminative regions based on the estimated scale factors. It aims to focus on the most discriminative region and simultaneously contain more fine-grained information. The authors apply the ARP module to fine-grained image recognition and Vehicle Re-Id tasks.