# M2 Former: Multiscale Patch Selection for Fine-Grained Visual Recognition
## 1 Introduction  
Despite recent rapid advances, fine-grained visual recognition (FGVR) is still one of the nontrivial tasks in the computer vision community. Unlike conventional recognition tasks, FGVR aims to predict subordinate categories of a given object, e.g., subcategories of birds [1], flowers [2; 3], and cars [4; 5]. It is a highly challenging task due to inherently subtle inter-class differences caused by similar subordinate categories and large intraclass variations caused by object pose, scale, or deformation. The most common solution for FGVR is to decompose the target object into multiple local parts [6; 7; 8; 9; 10]. Due to subtle differences between fine-grained categories mostly residing in the unique properties of object parts [11], decomposed local parts provide more discriminative clues of the target object. For example, a given bird object can be decomposed into its weak, wing, and head parts. At this time, 'Glaucous Winged Gull' and 'California Gull' can be distinguished by comparing their corresponding object parts. Early approaches of these part-based methods find discriminative local parts using manual part annotations [6; 7; 11]. However, curating manual annotations for all possible object parts is labor-intensive and carries the risk of human error [12]. Therefore, the research focus has consequently shifted to a weakly supervised manner [8; 9; 10; 13; 14; 15]. Researchers use additional tricks such as attention mechanisms [9; 14; 16] or region proposal networks (RPNs) [13; 15; 17] to estimate local parts with only category-level labels. However, the part proposal process greatly increases the overall computational cost. Additionally, they tend not to deeply consider the interactions between estimated local parts that are essential for accurate recognition [18]. Recently, Vision Transformers (ViTs) [19] are being actively applied to FGVR [18; 20; 21; 22; 23; 24; 25; 26; 27]. Relying exclusively on the Transformer [28] architecture, ViTs have shown competitive image classification performance on a large scale. Similar to token sequences in NLP, ViTs embed the input images into fixed-size image patches, and the patches pass through multiple Transformer encoder blocks. Patch-by-patch processing is highly suitable for FGVR because each image patch can be considered as a local part. This means that the cumbersome part proposal is no longer necessary. Additionally, the self-attention mechanism [28] inherent in each encoder block facilitates the modeling of global interactions between patch-divided local parts. ViT-based FGVR methods use patch selection to further boost performance [18; 20; 21; 22]. Because ViTs deal with all patch-divided image regions equally, many irrelevant patches may lead to inaccurate recognition. Similar to part proposals, patch selection selects the most salient patches from a set of generated image patches based on the computed importance ranking, i.e., accumulated attention weights [21; 29]. As a result, redundant patch information is filtered out, and only selected salient patches are considered for the final decision. However, the existing ViT-based FGVR methods suffer from their single-scale limitations. ViTs use fixed-size image patches throughout the entire network, ensuring that the receptive fields remain the same across all layers and preventing ViTs from obtaining multiscale feature representations [30; 31; 32]. On the other hand, Convolutional Neural Networks (CNNs) are suitable for multiscale feature representations thanks to their staged architecture, where feature resolution decreases as layer depth increases [33; 34; 35; 36; 37]. In the early stages, spatial details of an object are encoded on high-resolution feature maps, and as the stages deeper, the receptive field expands with decreasing feature resolution, and higher-order semantic patterns are encoded into low-resolution feature maps.