Adaptive Region Pooling for Fine-Grained Representation Learning

1 INTRODUCTION . The main goal of fine-grained recognition is to discriminate the sub-categories of the images that belong to a same general class . Its practical applications include fine-grained image classification ( Welinder et al. , 2010 ; Krause et al. , 2013 ; Maji et al. , 2013 ) , re-identification ( Zheng et al. , 2017b ; Liu et al. , 2016b ; a ) , etc . These tasks are fundamentally challenging due to considerable inter-class similarity . Consequently , an ideal feature extractor should be capable of extracting discriminative fine-grained features from subtle image regions , such as the beak to classify the bird species or the stickers on the windshield to identify the vehicle . While the general design of Convolutional Neural Network ( CNN ) ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ) applies either strided convolution or strided pooling as downsampling operations to reduce the computational requirement of the network and also increase the receptive field of subsequent convolutions , these operations , however , have been substantiated to be harmful to fine-grained recognition ( Sun et al. , 2018 ; Luo et al. , 2019a ) . Its disadvantage is two-fold : 1 ) false assumption on spatial equality which would overemphasize the background region while neglecting the smaller but more discriminative regions and 2 ) excessive reduction of resolution which would spoil the granularity of feature . To cope with the problem , existing approaches ( Zhang et al. , 2014 ; Fu et al. , 2017 ; Chen & Deng , 2019 ; Zhang et al. , 2019 ) introduce cropping operation that guides the network to only focus on the most critical region while preserving more details in the sampled part . However , to maintain computational complexity , full conservation of feature resolution would instead restrict a limited scale , or area , of the cropped region . Comprehensively , as shown in Figure 1 ( a ) ( c ) , the problems of two commonly used downsamplings can be summarized into a trade-off problem between the scale of receptive field and the granularity of representation . To tackle the problem , in this work , we introduce a novel pooling algorithm , named Adaptive Region Pooling ( ARP ) , to bridge strided and cropping operations . It consists of two steps . First , ARP automatically crops the feature from the most critical region with a well-estimated and adaptive cropping scale . Second , after being cropped into different sizes , the feature is further downsampled to a consistent size through bilinear downsampling . As illustrated in Figure 1 ( b ) , with the meticulous evaluation of cropping scale , ARP can better handle the trade-off issue and leverage the beneficial properties of both strided and cropping operations . Besides , ARP also features other three advantages . First , it is incorporated with a controllable mechanism that supports users to actively reach a more satisfying balance in the trade-off problem , as in two cases in Figure 1 ( b ) . Second , compared to the previous work ( Lin et al. , 2015a ; Huang et al. , 2016 ; Kim et al. , 2018 ; Zheng et al. , 2019 ) which implement the cropping operations by an additional neural network , ARP utilizes a more efficient yet effective operation without any learning-based parameters . Hence , it makes the network achieve lower computational overhead , experience a stabler training process , and converge more quickly . Finally , ARP can be simply plugged into any CNN layer and the network can be optimized in a one-stage end-to-end manner , which is completely identical to the training of original CNN backbone . At last , we demonstrate that ARP could facilitate fine-grained recognition through our Multiple Scale and Granularity Network ( MSGN ) . In MSGN , ARP serves as a downsampling alternative to extract the feature of finer details complementary to the coarse feature provided by the typical downsampling method . The contributions of this paper can be summarized as follows : • We introduce Adaptive Region Pooling ( ARP ) : a novel downsampling that smoothly bridges strided pooling and cropping operations . The sub-sampled representation focuses on the most discriminative region and simultaneously contains more fine-grained information . • To our knowledge , ARP is the first pooling algorithm with an adjustable trade-off mechanism that allows users to apply the human domain knowledge of the target task to strike a more desirable balance between the scale of receptive field and the granularity of downsampled feature . • With the usage of ARP , MSGN outperforms the existing frameworks in terms of both effectiveness and efficiency on multiple benchmarks for both the tasks of image classification and retrieval . 2 RELATED WORK . Fine-grained recognition focuses on learning discriminative representations to classify or identify the images within a same general category . There are two common applications : re-identification and fine-grained image classification . Re-identification ( re-ID ) studies the problem of identifying the individuals of persons ( Zheng et al. , 2015 ; 2017b ; Ristani et al. , 2016 ) or vehicles ( Liu et al. , 2016b ; c ; a ) in different camera views . To distinguish two similar targets with only slight differences , most of the existing methods ( Yao et al. , 2019 ; Kalayeh et al. , 2018 ; Liu et al. , 2021 ; Wang et al. , 2017 ; Zhou & Shao , 2018 ; Chen et al. , 2020b ) adopt spatial attentive mechanism to emphasize the features extracted from critical parts , such as the head for person re-ID or the tires for vehicle re-ID . While previous work get significant improvement , they generally suffer from massive loss of fine-grained details when CNN backbone downsamples the feature through strided operations . Sun et al . ( 2018 ) and Luo et al . ( 2019a ) both find a huge performance gain in the task of person re-ID , numerically 1.4 % for mAP on Market1501 ( Zheng et al. , 2015 ) , by simply removing the pooling operation in the last CNN layer . Fine-grained image classification ( FGIC ) aims to classify the sub-classes of images , such as the species of birds ( Welinder et al. , 2010 ) or the models of vehicles ( Krause et al. , 2013 ) . Compared to re-ID , the datasets of FGIC have a severe problem of image misalignment which impels lots of work to apply another sampling operation , cropping , to align the image or representation and also guide the network to focus on the sampled region . Several prior work ( Welinder et al. , 2010 ; Zhang et al. , 2013 ; 2014 ; Lin et al. , 2015a ; Huang et al. , 2016 ; Kim et al. , 2018 ; He et al. , 2019 ) leverage extra object detection datasets or human-labeled bounding box annotations to learn the localization of discriminative regions . However , such auxiliary labels are hard to acquire in the real-world and large-scale scenarios . Recently , there have been emerging work with a more general setting by proposing unsupervised approaches . Jaderberg et al . ( 2015 ) introduce a differentiable STN that can learn to well align the images or representations by themselves . While it can perform well on the digit datasets ( LeCun et al. , 1998 ; Netzer et al. , 2011 ) , lack of explicit guidance leads to the unstable training process and being hard to converge on more sophisticated tasks ( see Section 4.3 for more details and discussions ) . As the following work , Fu et al . ( 2017 ) propose RACNN which is a cascade coarse-to-fine CNN model . The similar architecture is also adopted by Chen & Deng ( 2019 ) and Zhang et al . ( 2019 ) . Although they show the great capability of cropping the discriminative region , the cascade architecture requires multi-stage training for better optimization and struggles with the requirement of heavy computation . In this work , we also introduce a sampling operation with an unsupervised approach ; however , Adaptive Region Pooling ( ARP ) contains neither additional neural network nor extra learning-based parameters and makes the network using ARP more computationefficient and able to be optimized in a simple one-stage end-to-end manner . 3 PROPOSED METHODOLOGY . In this section , we introduce the proposed pooling algorithm in three parts . First , in Section 3.1 , Region Pooling ( RP ) considers a simplified task which aims to crop the hidden representation from the most critical region with a fixed cropping scale . Then , in Section 3.2 , Adaptive Region Pooling ( ARP ) estimates the adaptive cropping scale for each representation to better capture the interesting object . Finally , we introduce Multiple Scale and Granularity Network ( MSGN ) as an example of network architecture with the usage of ARP in Section 3.3 . 3.1 REGION POOLING . Given a CNN feature and a fixed downsampling rate , the main goal of Region Pooling ( RP ) is to crop the feature from the most critical region into a consistent scale . An intuitive approach is to use an extra network to predict the cropping center which nevertheless , often suffers from the requirement of expensive bounding box annotations or complicated multi-stage learning strategy to train the network . In contrast , RP uses a more efficient yet effective operation with two steps . In the first step , as shown in Figure 2 ( a ) , RP would evaluate the significance score for each location and generate the keypoint confidence map C based on the C-dim feature vectors , where C denotes the number of channels . Here , we define “ significance ” in an engineering sense as follows : a location , whose feature vector includes more highly-activated neurons , is more significant . The physical meaning is that a channel-wise feature map is actually a detector of a certain semantic object ( Chen et al. , 2017 ; 2020a ) , such as a bird ’ s beak or wings . For a well-trained CNN , most of the channels would detect the objects which are more beneficial to the target task ; thus , the locations activated by more channels typically cover critical semantic objects and are more significant . Based on the definition , we then give a formal formulation as follows . For the input feature F ∈ RC×H×W with width W , height H and C channels and its element ( or activated value ) denoted as x ( i , j , k ) at ith channel , jth row and kth column , in order to balance the influence of each channel in the subsequent operations , we first standardize each channel-wise feature map as : x̂ ( i , j , k ) = x ( i , j , k ) − µ ( i ) σ ( i ) , where µ ( i ) = H∑ j W∑ k x ( i , j , k ) H ·W , σ2 ( i ) = H∑ j W∑ k ( x ( i , j , k ) − µ ( i ) ) 2 H ·W , ( 1 ) and x̂ represents the element of the standardized feature . Such standardization can be simply implemented by Instance Normalization ( Ulyanov et al. , 2016 ) . Afterwards , we evaluate the significance score of each location based on the average of their feature vectors with the form as : s ( j , k ) = C∑ i=1 x̂ ( i , j , k ) C ( 2 ) where s ( j , k ) represents the significance score of the location ( j , k ) . Complied with our definition , Equation 2 assigns larger s for the locations that are highly activated in more channels . Next , we estimate the confidence of a location to be a “ keypoint ” , or a critical location , by Sigmoid function σ ( · ) . Specifically , we generate a keypoint confidence map C ∈ RH×W by : C = [ c ( 1,1 ) , ... , c ( H , W ) ] = [ σ ( s ( 1,1 ) ) , ... , σ ( s ( H , W ) ) ] ( 3 ) where c ( j , k ) denotes the keypoint confidence at the location ( j , k ) . We visualize the generated keypoint confidence maps C in Figure 5 and 6 . If we project them to the original images , the locations with larger keypoint confidences effectively represent the more discriminative regions , such as the beak and colorful feather of a bird or the bumper and lamp of a vehicle . In the second step , as in Figure 2 ( c ) , given a fixed downsampling rate DSR = [ DSRH , DSRW ] , RP would crop the most critical region with height ( H · DSRH ) and width ( W · DSRW ) . To this end , we apply a convolution operation on the keypoint confidence map C with a kernel K ∈ R ( H·DSRH ) × ( W ·DSRW ) . The region with the largest weighted summation of keypoint confidences would be selected as the cropped region , which can be written as : ( j∗ , k∗ ) = arg max ( j , k ) ( K C ( j , k ) ) ( 4 ) where represents inner product , C ( j , k ) is the cropped keypoint confidence map centered at the location ( j , k ) , and ( j∗ , k∗ ) indicates the center location of the selected cropped region . Discussion of kernel K. Empirically , we find that using Gaussian filter as kernel K yields the best performance . A potential explanation is that it tends to select the region with more keypoints locating near the center of cropped region and hence , also solves the problem of feature misalignment .

Summary: This paper presents an adaptive region pooling (ARP) method for fine-grained representation learning. ARP crops the features with the estimated scale. The features are further downsampled to a consistent size through bilinear downsampling. Experiments conducted on two tasks validate the effectiveness of the proposed method.