Hypergraph-guided Intra- and Inter-category Relation Modeling for Fine-grained Visual Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 03:29:12

Abstract: Fine-grained Visual Recognition (FGVR) aims to distinguish objects within similar subcategories. Humans adeptly perform this challenging task by leveraging both intra-category distinctiveness and inter-category similarity. However, previous methods failed to combine these two complementary dimensions and mine the intrinsic interrelationship among various semantic features. To address the above limitations, we propose HI2R, a Hypergraph-guided Intra- and Inter-category Relation Modeling approach, which simultaneously extracts the intra-category structural information and inter-category relation information for more precise reasoning. Specifically, we exploit a Hypergraph-guided Structure Learning (HSL) module, which employs hypergraphs to capture high-order structural relations, transcending traditional graph-based methods that are limited to pairwise linkages. This advancement allows the model to adapt to significant intra-category variations. Additionally, we propose an Inter-category Relation Perception (IRP) module to improve feature discrimination across categories by extracting and analyzing semantic relations among them. Our objective is to alleviate the robustness issue associated with exclusive reliance on intra-category discriminative features. Furthermore, a random semantic consistency loss is introduced to direct the model's attention to commonly overlooked yet distinctive regions, which indirectly enhances the representation ability of both HSL and IRP modules. Both qualitative and quantitative results demonstrate the effectiveness and usefulness of our proposed HI2R model.

Main paper: 
 introduction: 
 Fine-grained visual recognition (FGVR) [39] has been a fundamental yet challenging task in computer vision, intending to distinguish subordinate categories within a broader category. Unlike general object recognition, FGVR faces increased difficulty due to large intracategory variations and small inter-category differences. These challenges are further compounded when objects from different Unpublished working draft. Not for distribution.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. categories share similar textural characteristics and appear in the same visual context, making it difficult to differentiate local features. Furthermore, even within a single category, significant variation in appearance, shape, and pose complicates the recognition process.Preceding works in FGVR have successfully employed attention mechanisms in CNNs to highlight and utilize salient regions [8, 9, 27, 28, 49, 50] . However, the conventional CNN-based approach of enhancing receptive fields through layer stacking, while capturing more contextual information, inadvertently blurs fine-grained details, compromising both robustness and discriminative ability in subcategory recognition. In response, recent research has shifted towards capturing intricate local features to deal with the challenge of large intra-category variations. Vision Transformers (ViT) [10] and its variants leverage the self-attention mechanism that naturally emphasizes local features by processing images in a patchby-patch manner. This allows ViT to capture fine-grained details, effectively overcoming the limitations of traditional CNNs in FGVR tasks [19, 21, 35, 40, 45, 46] . Despite these advancements, many methods still primarily concentrate on significant patch token selection (see in Fig. 1a ), while overlooking the intricate high-order relations among various local semantic features. Recognizing these structural relations is crucial for accurately identifying object foreground parts, which further helps reduce background distractions and enhances classification stability [5, 30] . For example, distinguishing a bird's gray legs from dense foliage requires a deep understanding of the structural relations among different bird parts.To address these challenges, some research has explored the utilization of graphs to capture relational information [30, 36, 48] . Although graph structures provide flexibility and effectiveness in relation modeling, a fundamental limitation [18, 23] is their restriction to exclusively connecting two nodes, accommodating only pairwise relations. This is inadequate for fine-grained objects whose parts (like a bird's head in Fig. 1a ) may be represented across multiple patch tokens with complex dependencies. Confining these complex multivariate relations to binary connections may lead to redundant calculations or the loss of crucial information [2] . Inspired by this, our research focuses on effectively modeling complex higher-order structural relations to better discern intra-category differences.In addition, for the challenge of small inter-category differences, previous works mainly guide the model to extract more discriminative features [3] . Nevertheless, these approaches primarily focus on extracting global features or those at a single level of granularity. The limited and uniform nature of the information encoded in these features often leads to pixel misidentification. For example, as illustrated in Fig. 1b , discerning the category of the central image based solely on one single local feature is challenging due to the similarity of its local features to those in other categories. Therefore, another focus of this paper is on leveraging inter-category relations to provide supplementary guidance for reasoning, tackling the challenge of small inter-category differences.Considering the above factors, we propose a Hypergraph-guided Intra-and Inter-category Relation Modeling (HI2R) for FGVR. It utilizes both intra-category high-order relations and inter-category similarity relations to enhance the discriminative power of feature representations. Firstly, to cope with significant intra-category variance, we introduce a hypergraph-guided structural learning (HSL) module that constructs patch tokens into a hypergraph. This structured modeling approach leverages hypergraphs to connect an arbitrary number of nodes within one hyperedge, thus facilitating the effective representation of complex spatial relations among semantic features. Additionally, we implement spatial background suppression to reduce noise introduced into the hypergraph. Secondly, the Inter-category Relation Perception (IRP) module extracts discriminative semantic relation information across categories to address the challenge posed by small inter-category differences. By modeling the similarity distribution within each category, this module facilitates a more nuanced understanding of inter-category relations. It enhances the original discriminative features through integrating cross-category shared semantics, thereby improving the model's ability to distinguish among similar categories. Finally, to tackle the potential semantic alignment challenges within the backbone network ViT [15] and further bolster the representational capabilities of the HSL and IRP modules, we introduce the Random Semantic Consistency (RSC) loss. Visualization experiments clearly demonstrate (Fig. 5 ) that this loss formulation encourages the model to delve deeper into diverse fine-grained representations, further improving the robustness of the overall system against various distortions and perturbations.Our main contributions are summarized as follows:• We propose a Hypergraph-guided Intra-and Inter-category Relation Modeling method, which emphasizes both intracategory structural features and inter-category semantic features. By collaboratively leveraging these two complementary dimensions, our method cooperates to promote model performance improvement.  
related work 2.1 vision transformer in fgvr: 
 By virtue of its patch-based processing and powerful self-attention mechanism, ViT [10] exhibits superior capacities over traditional CNNs, resulting in enhanced performance. Following this advancement, several methods [19, 21, 35, 40, 45, 46, 51] have been proposed to improve the performance of FGVR. Among these, RAMS-Trans [21] proposes a recursive approach leveraging self-attention weights to learn discriminative parts in a multi-scale way. To capture subtle visual differences between different subordinate categories in ViT, DCAL [51] introduces a dual cross-attention learning algorithm to complement self-attention learning for capturing visual differences. Meanwhile, MpT-Trans [35] extends the class token to multiple tokens representing various parts, improving the extraction of discriminative information. However, while these methods offer significant benefits, they primarily focus on learning discriminative features without considering the relations among various semantic features. To bridge this gap, our proposed HI2R innovatively leverages rich semantic relations across different categories, utilizing the IRP module for enhanced reasoning.Graph neural networks (GNNs) have gained popularity for their ability to represent unstructured data as graphs, with diverse applications ranging from social networks [17] to recommendation systems [37] , and other fields. This versatility extends to FGVR, where recent efforts have explored graph structures to represent image elements, enhancing the analysis of complex visual data. For instance, GaRD [48] introduces a graph grouping module to map features onto a low-dimensional manifold, preserving rich Fuzzy C-Means semantic relationships for fine-grained recognition. SIM-Trans [30] constructs a graph to represent object structures, which are then extracted and integrated into the backbone. Additionally, SRGN [36] deduces structure embedding by correlating position information with visual features along the axial direction. Despite these advancements, the traditional graph structure, primarily reliant on pairwise connections, faces limitations in expressing multifaceted relationships inherent in complex data.To overcome this, hypergraphs are employed for high-order feature representation, with Hypergraph neural networks (HGNNs) extending the capabilities of GNNs to encompass more intricate data structures. DHGNN [23] employs K-Means and KNN algorithms to construct a dynamic hypergraph structure, enhancing data representation precision. Similarly, ViHGG [18] innovates by establishing and updating the hypergraph structure with the Fuzzy C-Means method, improving computational efficiency. Inspired by ViHGG [18] , HI2R utilizes hypergraphs to capture high-order structural relations among semantic features. To specifically cater to FGVR challenges, we introduce a spatial-based background suppression operation to mitigate the impact of background noise, thereby enhancing discriminative representation. 
approach: 
 As illustrated in Fig. 2 , our proposed HI2R incorporates the HSL module, which processes patch tokens derived from the L-th transformer layer and leverages hypergraph structures to effectively capture spatial information ( §3.2). Additionally, the IRP module employs cross-attention mechanisms to extract and utilize inter-category semantic relations, enhancing reasoning through additional guidance ( §3.3). Furthermore, the integration of RSC loss facilitates the learning of diverse discriminative features, thereby indirectly boosting the representational power of both the HSL and IRP modules ( §3.4).Following previous methods, ViT is utilized as the backbone network for feature extraction. This section briefly revisits the key components of ViT, including tokenization, position embeddings, and encoder blocks, which play a crucial role in this research.An image I is first split into 𝑁 = 𝐻 × 𝑊 patches 𝑥 𝑖 ∈ R 𝑃 ×𝑃 ×𝐶 , 𝑖 ∈ {1, . . . , 𝑁 }, where 𝑃 is the size of each patch, 𝐶 is the channel number of the image, and 𝑁 is the number of patches. Subsequently, a linear embedding layer 𝐸 ∈ R (𝑃 2 •𝐶) ×𝐷 is employed to map each patch into a token, with 𝐷 being the dimension of tokens. In addition, a learnable class token 𝑥 𝑐𝑙𝑠 ∈ R 𝐷 is introduced for classification and a trainable positional encoding 𝐸 𝑝𝑜𝑠 ∈ R (𝑁 +1) ×𝐷 is incorporated. Consequently, the input to the first transformer layer can be represented as follows: There are 𝐿 transformer layers and each layer is composed of a Multi-Head Self-Attention (MSA) block and a Multi-Layer Perception (MLP) block. Both blocks are preceded by a Layer Normalization (LN) operation. The output of each layer encapsulates the transformations applied by these blocks, and can be summarized as follows:EQUATIONEQUATIONIn the standard ViT model, the class token of the last layer 𝑧 𝑐𝑙𝑠 𝐿 is input into a classifier to derive the final classification results.ViT is adept at encoding region-wise dependencies but tends to neglect high-order structural relations within fine-grained objects. To overcome this limitation, the HSL module is introduced to provide high-order spatial enhancement for further feature representations. However, the direct application of hypergraphs in this context inadvertently introduces background noise. To mitigate this issue, a spatial-based background suppression operation is employed prior to the modeling process. Spatial-based Background Suppression. Constructing a hypergraph directly from all patch tokens obtained from the final transformer layer risks incorporating background noise into our node representations. This noise can detract from the model's performance by obscuring critical features. Consequently, a preliminary step of spatial-based background suppression is proposed to suppress background noise, facilitating the model's ability to learn more meaningful node representations. Assuming there are 𝑁 ℎ heads in the MSA block, the attention weights of the 𝑘-th layer can be represented as follows:EQUATIONwhere 𝑎 𝑖 𝑘 is the attention weight in the 𝑖-th head of the 𝑘-th layer and 𝑎 𝑖 0 𝑘 is the corresponding weight of the class token. Thus, the attention weights of all layers are organized as:EQUATIONTo further focus attention, we follow attention rollout [1] to integrate attention weights from all preceding layers. Specifically, the aggregated attention is iteratively computed through the matrix multiplication in the following manner:EQUATIONwhere Ā = 1 2 𝐴 𝑘 + 1 2 𝐸 illustrates the re-normalized attention weights by using an identity matrix 𝐸 to account for residual connections, preserving identity information through the layers. Here 𝐴 𝑠𝑐𝑜𝑟𝑒 signifies the comprehensive relationships between patch tokens. The weights associated with the class token 𝑎 𝑖 0 𝑠𝑐𝑜𝑟𝑒 are then extracted from each head in 𝐴 𝑠𝑐𝑜𝑟𝑒 . In multi-head self-attention, different heads concentrate on distinct aspects. To leverage the unique insights each head offers, weights from various heads are combined. This process can be formalized as follows:EQUATIONwhere𝑇𝑜𝑝 𝑠 retains the top 𝑠 = 𝜆×𝑁 relevant patch tokens 𝑠 ∈ 𝑁 + , and 𝐹 represents the selected patch tokens. The term 𝜆 serves as a hyperparameter that influences the selection ratio. The hypergraph construction is carried out utilizing the subset 𝐹 . Hypergraph Construction. A hypergraph G = (V, E, W) is constructed with hyperedges and nodes, where V denotes the node set, E represents the set of hyperedges, and W signifies the weight matrix of the hyperedge set. Unlike vanilla graphs with pairwise connections, each hyperedge e can connect an arbitrary number of nodes. In this paper, each patch token derived from spatial-based background suppression is considered a node. The 𝑛-th node is represented by 𝑓 𝑛 ∈ R 1×𝐷 , with all nodes collectively represented as𝐹 = [𝑓 1 , 𝑓 2 , . . . , 𝑓 𝑠 ] ∈ R 𝑠 ×𝐷 .To construct the hypergraph structure, the Fuzzy C-Means is employed to generate 𝐸 nodes clusters and a membership matrix U, where each cluster 𝑖 corresponds to a hyperedge 𝑒 𝑖 (𝑖 = 1, 2, . . . , 𝐸). The set of hyperedges is defined as E = {𝑒 1 , 𝑒 2 , . . . , 𝑒 𝐸 }. Utilizing membership matrix U, the hypergraph G can be represented by an incidence matrix H ∈ R 𝑠 ×𝐸 , where 𝐻 𝑖 𝑗 = 𝑈 𝑖 𝑗 indicates the membership degree of the corresponding node belonging to the corresponding hyperedge. Following this, the hypergraph convolutional layer then facilitates information exchange between nodes based on the hypergraph G.Hypergraph Convolution. By leveraging the learned H , the model employs a hypergraph convolutional operation to aggregate high-order structural information, thereby enhancing feature representations. This operation facilitates information exchange among nodes via a two-stage node-hyperedge-node message passing scheme. Initially, it aggregates node features onto hyperedges, followed by aggregating hyperedge features back onto nodes. And the relation-enhanced feature R ∈ R 𝑠 ×𝐷 can be obtained as:EQUATIONwhere Θ 1 and Θ 2 are the learnable parameters, and 𝜎 is an activation function. The multiplication of H 𝑇 facilitates information aggregation from nodes to hyperedges. Conversely, multiplying H can be considered as information aggregation from hyperedges to nodes. Intermediately, D ∈ R 𝑠 ×𝑠 and B ∈ R 𝐸×𝐸 denote the node degree matrix and the hyperedge degree matrix obtained through the broadcast operation, respectively. To further enhance the feature transformation capacity and mitigate the over-smoothing phenomenon, a Multilayer Perceptron (MLP) is applied to each node:EQUATIONwhere Z ∈ R 𝑠 ×𝐷 , 𝑊 1 and 𝑊 2 are the weights of fully-connected layers, with the bias term omitted for simplicity. In both the Hypergraph Convolution and MLP modules, batch normalization is applied following each fully connected and hypergraph convolution layer, which is not shown for brevity. The updated feature embeddings Z are considered as the structural feature, which is subsequently flattened and concatenated with (represented in Fig. 2 as ) the class token 𝑧 𝑐𝑙𝑠 𝐿 to form an aggregated feature. This feature introduces structural information into the transformer backbone. The aggregated feature is then input into a fully connected layer for feature fusion:EQUATIONwhere 𝜑 denotes the flatten operation, 𝑊 3 and 𝑏 represent the weight and bias in the fully connected layer, respectively. And X signifies the fusion feature. Through end-to-end training, the model effectively understands the composition of fine-grained objects and places emphasis on spatial-based feature embeddings. Consequently, the transformer network is equipped to capture both the appearance and structural information of the objects, facilitating accurate recognition.Relying solely on intra-category discriminative features without considering semantic relations across categories may lead models to overemphasize specific details. This tendency often results in confusion, especially when processing images with similar local features. Therefore, it is equally essential to focus on inter-category semantic relations, which improves model robustness by discerning subtle differences across categories. The proposed IRP module capitalizes on these relations to refine the fusion feature, thereby boosting overall accuracy and performance. Specifically, a concept cache is maintained, denoted by P = {𝑝 𝑐 | 𝑖 = 1, 2, . . . , 𝐶}, where 𝐶 represents the count of categories in the dataset, and each row 𝑝 𝑐 ∈ R 𝐷 denotes a basis concept for the fine-grained category 𝑐.Update of Concept Cache. To derive the specific concept feature 𝑝 𝑐 for each instance from category 𝑐, the fusion feature X ∈ R 𝐷 , obtained from the HSL module, serves as the extracted concept. Before training, the concept cache 𝑃 is uniformly initialized. Subsequently, the concept prompt 𝑝 𝑐 is then dynamically updated using a moving average fashion:EQUATIONwhere 𝛼 ∈ (0, 1) is a coefficient that controls the update rate of the category concept 𝑝 𝑐 . Fusion Feature Refinement. Our goal is to enrich the fusion feature by incorporating knowledge from other categories, thus achieving more informative and comprehensive representations.Fig. 2 illustrates the utilization of a cross-attention mechanism to model affinities among different categories, facilitating the propagation of similarities and distinctions. Finally, we obtain refined feature representations abundant with invariant matching clues. With the fusion feature X serving as a query and the concept cache P as both key and value, cross attention refines X as follows:EQUATIONwhere 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(•) first normalizes the input, then projects X to 𝑄 ∈ R 1×𝐷 ℎ , and P to 𝐾 ∈ R 𝐶 ×𝐷 ℎ and 𝑉 ∈ R 𝐶 ×𝐷 ℎ . The affinity matrices 𝐴 𝑐𝑟𝑜𝑠𝑠 is computed as:EQUATION)The process involves matching 𝑄 against 𝐶 key vectors (𝐾 = [𝑘 1 ; 𝑘 2 ; ...; 𝑘 𝐶 ], 𝑘 𝑖 ∈ R 1×𝐷 ℎ ) using inner product, scaled and normalized by softmax. The fusion feature updates with affinity weights, effectuating semantic relation information propagation:EQUATION)To some extent, the information propagation procedure can be deduced as a weighted sum of 𝐶 value vectors (𝑉 = [𝑣 1 ; 𝑣 2 ; ...; 𝑣 𝐶 ],𝑣 𝑖 ∈ R 1×𝐷 ℎ ). This process maintains moderate computational complexity without introducing additional parameters. Notably, The residual features provide stability in the early training stages.Following this, the refined feature X 𝑟 is forwarded to a classification head, producing the prediction vector 𝑝𝑟𝑒𝑑 (I) for the input image I. Subsequently, the classification loss is defined as:EQUATIONwhere 𝑆 (I) is the training set and 𝑦 is the one-hot label for I.To further improve the representational capabilities of the HSL and the IRP modules, the RSC loss is introduced. This loss encourages the model to delve deeper into diverse label-relevant regions. As shown in Fig. 2 , the architecture incorporates a dual-branch module, each starting with a ViT backbone and culminating in a classification header. The parameters of the two branches are shared. Giving a batch of fine-grained images I, we first randomly erase the input images and then flip these images to get their flipped counterparts I ′ = 𝑇 (I). This mimics large inter-class variations due to varying poses and obscured discriminative features. Each branch processes I and the transformed image I ′ respectively, aggregating attentions as outlined in Eq.5. In the transformer layer, attention weights between patch tokens and the class token indicate their significance in classification. The extracted category-related attention score 𝐴 𝑐𝑙𝑠 𝑠𝑐𝑜𝑟𝑒 ∈ R 𝑁 ×1 is reshaped into 𝑀 ∈ R 𝐻 ×𝑊 , where 𝐻 and 𝑊 is the number of tokens along image height and width, respectively (See §3.1).Based on the definition of attention consistency, the attention weights of the transformed image I ′ need to be flipped for equivariance. The RSC consistency loss is employed as the disparity between attention weight to minimize the distance between the attention map 𝑀 and 𝐹𝑙𝑖𝑝 (𝑀 ′ ):EQUATION)where ∥•∥ 2 represents the Euclidean norm. And 𝑀 𝑖,𝑗 and 𝐹𝑙𝑖𝑝 (𝑀 ′ ) 𝑖,𝑗 are the values at the (𝑖, 𝑗)-th position of the corresponding attention maps 𝑀 and 𝑀 ′ , respectively. By minimizing the L 𝑅𝑆𝐶 between 𝑀 and 𝑀 ′ , the model is encouraged to learn diverse and viewpointrobust features from a single image. In summary, the vision transformer backbone and the proposed modules are jointly trained end to end, with the total objective:EQUATION)where 𝛽 is used for the numerical balance of various losses. Training our method only needs the ground-truth label. 
experiments 4.1 datasets and evaluation metric: 
 The effectiveness of our proposed HI2R is evaluated through experiments conducted on three benchmarks for fine-grained visual recognition:• CUB-200-2011 [34] contains 11,788 images across 200 subcategories, with 5,994 images for training and 5,794 for testing. It is considered one of the most challenging datasets due to each species having only 30 images for training. • NAbirds [32] is a collection of 48,000 annotated photographs of the 400 species of birds that are commonly observed in North America. Each species is represented by more than 100 photographs, including separate annotations for males, females, and juveniles, resulting in 700 visual categories. • iNaturalist2017 [33] is one of the largest FGVR datasets.It features cross-species images with a biased distribution across 5,089 categories. The dataset's large scale and challenging nature provide a robust benchmark for evaluating fine-grained visual recognition performance.Top-1 accuracy is adopted as the evaluation metric. The model is trained using only image-level labels, without any additional annotations for supervised training.Consistent with the settings in previous methods, we emolpy the ViT-Base [10] pre-trained on ImageNet-21k as the backbone network, comprising 12 encoder blocks. The patch size is set to 16, with a stride factor of 12. To ensure a fair comparison with other methods, input images in the CUB-200-2011 and NAbirds datasets are first resized to 600 × 600 and subsequently randomly cropped to 448 × 448 during training. For the iNaturalist 2017 dataset, the image size is set to 304 × 304 following [30] . Random horizontal flipping is applied for training, while center cropping is employed during testing. The model is trained using the stochastic gradient descent (SGD) optimizer with a momentum of 0.9, and is regulated by a cosine annealing scheduler. The initial learning rate is set to 4e-2 for iNaturalist 2017 and 2e-2 for the other two datasets. The model is trained for 10,000 steps, with the first 500 steps as warmup, and the batch size is set to 16 for all datasets. Therefore, the comparison experiments with state-of-the-art transformer-based methods are conducted under fair and convincing conditions.In the HSL module, the ratio of selected patch tokens 𝜆 in spatialbased background suppression and the number of node cluster 𝐸 for hypergraph construction are set to 0.3 and 6, respectively. In the IRP module, the update rate of category concepts 𝛼 is set to 1e-2. And 𝛽 is set to 1e-2 when calculating the loss. The experiments are performed using PyTorch on NVIDIA GeForce GTX 3090 Ti GPUs.Our proposed HI2R is compared with existing state-of-the-art works, including CNN-based and ViT-based methods on the fine-grained datasets mentioned above.CUB-200-2011: Table 1 illustrates the superior performance of the proposed HI2R over current state-of-the-art approaches. HI2R achieves the best classification accuracy at 92.5%, marking a 1.9% enhancement compared to the ViT baseline [10] . The effectiveness of HI2R is validated through a comparison with various state-of-theart methods, categorized into two groups. The first group consists of methods using CNNs for feature extraction, with SR-GNN [5] achieving the highest accuracy of 91.9% by leveraging Xception to aggregate context-aware features from relevant image regions. Venue Backbone Acc(%) RA-CNN [14] CVPR 2017 VGGNet19 85.3 MA-CNN [49] ICCV 2017 86.5 FDL [38] CVPR 2018 VGGNet16 86.7 HBP [42] ECCV 2018 87.1 NTS-Net [41] ECCV 2018 ResNet50 87.5 Cross-X [26] ICCV 2019 87.7 DCL [6] CVPR 2019 87.8 PMG [11] ECCV 2020 89.6 MCEN [25] ACM MM 2021 89.3 GaRD [48] CVPR 2021 89.6 CMN [8] TIP 2022 88.2 MA-ASN [43] TMM 2022 89.5 SRGN [36] IJCV 2024 91.4 API-Net [52] AAAI 2020 ResNet101 88.6 PART [47] TIP 2021 90.1 CAL [28] ICCV 2021 90.6 CAP [4] AAAI 2021 Xception 91.8 SR-GNN [5] TIP 2022 91.9 ViT [10] ICLR 2020ViT-Base 90.6 RAMS-Trans [21] ACM MM 2021 91.3 AF-Trans [45] ICASSP 2022 91.6 TransFG [19] AAAI 2022 91.7 DCAL [51] CVPR 2022 92.0 SIM-Trans [30] ACM MM 2022 91.8 IELT [40] TMM 2023 91.8 MpT-Trans [35] ACM MM 2023 92.0 MP-FGVC [24] AAAI 2024 91.8 HI2R (Ours) -ViT-Base 92.5 Venue Backbone Acc(%) MaxEnt [13] NeurIPS 2018 DenseNet-161 83.0 API-Net [52] AAAI 2020 88.1 DSTL [7] CVPR 2018 Inception-v3 87.9 Cross-X [26] ICCV 2019 ResNet50 86.4 PAIRS [16] WACV 2019 87.9 GaRD [48] CVPR 2021 88.0 CMN [8] TIP 2022 87.8 PMG-v2 [12] TPAMI 2022 ResNet101 88.4 MGE-CNN [44] ICCV 2019 88.6 FixSENet [31] NeurIPS 2019 SENet-154 89.2 CAP [4] AAAI 2021 Xception 91.0 SR-GNN [5] TIP 2022 91.2 ViT [10] ICLR 2020ViT-Base 89.9 TransFG [19] AAAI 2022 90.8 IELT [40] TMM 2023 90.8 MpT-Trans [35] ACM MM 2023 91.3 MP-FGVC [24] AAAI 2024 91.0 HI2R (Ours) -ViT-Base 91.5 Despite the accomplishments of CNN-based methods, ViT-based methods perform better in FGVR. Among ViT-based methods, HI2R demonstrates superior performance, surpassing MpT-Trans [35] by a margin of 0.5%. Furthermore, HI2R exceeds SIM-Trans [30] by an absolute gain of 0.7% in performance. These findings indicate that although the integration of graphs with ViT yields remarkable performance, hypergraphs exhibit an enhanced capability in representing complex high-order structural information in FGVR. NABirds: Table 2 presents the performance of the proposed HI2R in comparison with state-of-the-art methods based on CNNs and ViT. The proposed approach surpasses the ViT baseline [10] by 1.6%. Additionally, it achieves improvements of 0.3% and 0.5% over the leading CNN-based method SR-GNN [5] and ViT-based method MpT-Trans [35] , respectively. These methods depended exclusively on the distinctiveness of features within categories for reasoning and neglected the potential benefits of leveraging similarities across categories. This limited and uniform nature of the information encoded in these features further led to pixel misidentification. Our HI2R enhances model performance by concurrently emphasizing intra-category high-order structural features and inter-category discriminative semantic features. Experimental results support the effectiveness of integrating these two complementary dimensions. Venue Backbone Acc(%) ResNet152 [20] CVPR 2016 ResNet152 59.0 SRGN [36] IJCV 2024 ResNet50 73.6 TASN [50] CVPR 2019 68.2 SSN [29] ECCV 2018 ResNet101 65.2 IARG [22] CVPR 2020 66.8 ViT [10] ICLR 2020ViT-Base 67.0 RAMS-Trans [21] ACM MM 2021 68.5 TransFG [19] AAAI 2022 71.7 AF-Trans [45] ICASSP 2022 68.9 SIM-Trans [30] ACM MM 2022 69.9 HI2R (Ours) -ViT-Base 73.9 iNaturalist2017: Table 3 summarizes a comparison between the proposed HI2R and existing state-of-the-art methods. The pure ViT baseline [10] demonstrates a significant advantage over the pure ResNet152 baseline [20] , with an absolute improvement of 8.0%, thereby highlighting the effectiveness of the Transformer structure. It is noteworthy that the accuracy of SGRN [36] reaches 73.6%, markedly outperforming all ViT-based methods, which highlights the significance of learning structural information to a certain degree. Nonetheless, compared with SRGN and the optimal ViT-based method TransFG [19] , HI2R still realizes additional enhancements of 0.3% and over 2.2%, respectively. This achievement emphasizes the effectiveness of HI2R in leveraging both intra-category structural information and inter-category relation information in FGVR, particularly in the context of large-scale datasets.We conduct ablation studies to evaluate the contributions of different modules in HI2R and to examine the impact of various settings category-related representations and significantly enhances recognition performance. Influence of the ratio of selected patch tokens. We evaluate the impact of adjusting the ratio of selected patch tokens in spatial-based background suppression, which will further influence the hypergraph construction. As illustrated in Fig. 3a , setting the ratio to 0.3 yields the highest accuracy performance. Notably, an excessive number of tokens can introduce undesired background noise, negatively affecting classification precision. Conversely, insufficient patch tokens may lead to inadequate representation of local features, thereby reducing model accuracy.Influence of the number of node clusters. We assess the influence of varying node cluster quantities FGVR accuracy. According to Fig. 3b , a node cluster count of 6 constitutes the optimal setting for our method. The number of node clusters directly impacts the hypergraph's dimensions by dictating the number of hyperedges. Insufficient hyperedges can lead to reduced performance due to an inadequate representation of data complexities. Conversely, an excess of hyperedges may compromise the model's ability to effectively discriminate local regions, thereby impairing its comprehensive understanding. Influence of the update rate of category concepts. We analyze the impact of the update rate of category concepts on the performance of the proposed HI2R. As shown in Fig. 4 , an update rate of 0.01 results in optimal outcomes. In FGVR, it is crucial to distinguish subtle features, requiring rapid adaptation to new samples. A higher update rate for category concepts (larger 𝛼 values) allows the model to rapidly integrate new category features, improving classification accuracy. However, fine-grained datasets are often small, and inter-category differences can be slight. An excessively high update rate may render the model overly sensitive to noise, increasing the risk of overfitting. Therefore, selecting an optimal update rate is essential for balancing rapid adaptation to new samples with the prevention of overfitting. This balance enhances both the accuracy and the stability of the model.Visualization. We present attention maps of different approaches for comparison. As illustrated in Fig. 5 , compared with ViT which primarily highlights the most discriminative regions of an image, HI2R can more accurately isolate the object foreground by comprehending both intra-and inter-category relations, effectively eliminating irrelevant background details. To demonstrate the influence of the RSC loss, the third and fourth rows of Fig. 5 depict the focus regions of HI2R without and with the RSC loss, respectively. These visualizations confirm that the RSC loss directs the network towards identifying a broader range of discriminative features, thus enhancing its effectiveness in recognition. 
conclusion: 
 In this paper, a novel framework has been proposed for FGVR, termed Hypergraph-guided Intra-and Inter-category Relation Modeling (HI2R). This framework is inspired by human visual perception and leverages both intra-category structural relations and inter-category similarity relations to improve FGVR task accuracy. Specifically, the HSL module has employed hypergraphs to encapsulate complex high-order structural relations among local features. Simultaneously, the IRP module has been designed to discern and utilize inter-category relations, serving as additional discriminative signals for more effective reasoning. To mitigate the model's over-reliance on specific semantic features, the RSC loss has been introduced to encourage the exploration of previously neglected discriminative regions, thereby indirectly enhancing the representational quality of both the HSL and the IRP modules. Qualitative and quantitative experiments have consistently demonstrated the superiority of our HI2R framework in the FGVR task.  
 
 Appendix: ACM MM, 2024, Melbourne, Australia © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. 


Decision: The reviewers have some concerns on the motivations, novelties, experimental validation, and computational cost of the proposed method. After rebuttal, three reviewers agree to accept this paper while one reviewer insists of weakly rejecting the paper. Though the performance gain of the proposed method on some datasets is limited, the paper is well written with clear motiviations, and the novel components are shown to be effective in experiments. I hence decide to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a Hypergraph-guided Intra and Inter-category Relation Modeling method for fine-grained visual categorization, which leverages both the intra-category structural relations and inter-category similarity relations for robust feature extraction and classification. Experiments are conducted on three fine-grained datasets.
strength_weakness: strength_weakness: 1.	The proposed method is sound and the procedures are easy to understand. 
2.	This paper is well-written and easy to follow.
limitations: limitations: 1.	The motivation of the proposed method should be further clarified. Why does the method cluster the significant image patches into different hyperedges in the proposed HSL module and what is its physical meaning? Besides, the high-order hyperedge in Figure 1 should be carefully drawn to emphasize its difference from the previous pairwise linkages.
2.	The novelty should be further clarified. The concept cache in the proposed IRP module seems similar to [a], which keeps the information of different categories. The differences should be highlighted.
3.	The effectiveness of the proposed method should be further validated. In the large-scale fine-grained dataset, i.e., iNaturalist 2017, the proposed HI2R method only surpasses the sub-optimal SRGN [b] by 0.3% with a stronger ViT backbone. More explanations should be added.
4.	More visualization results should be presented to show the image patches in the proposed hypergraph for readability.
[a] Fine-Grained Classification via Categorical Memory Networks. TIP 2022
[b] Accurate fine-grained object recognition with structure-driven relation graph networks. IJCV 2024
paper_summary: paper_summary: This paper addresses Fine-grained Visual Recognition (FGVR). The authors propose HI2R, a Hypergraph-guided Intra- and Inter-category Relation Modeling approach, to combine intra-category distinctiveness and inter-category similarity effectively. HI2R includes a Hypergraph-guided Structure Learning (HSL) module to capture high-order structural relations and an Inter-category Relation Perception (IRP) module to improve feature discrimination across categories. Additionally, a random semantic consistency (RSC) loss is introduced to enhance the model's attention to distinctive regions. The experiments demonstrate the superior performance of the proposed model.
strength_weakness: strength_weakness: The paper is well-written and easy to follow. 

All the modules of the model proposed by the author have proved to be effective.
limitations: limitations: 1 The application of GNN technology appears to be a common practice. For example, [1] and [2] have applied this technology. The authors have also discussed related methods in the related work section. However, it is necessary to compare this work with those methods, especially the relevant ViHGG, and point out the differences. This comparison is crucial for assessing the contributions of this paper.

[1] Region Graph Embedding Network for Zero-Shot Learning. ECCV 2020.

[2] Dual Part Discovery Network for Zero-Shot Learning. MM 2022.

2 The impact of $\beta$ has not been studied.

3 In some places, such as the abstract, "Inter-category Relation Perception" is used, but Figure 2 mentions "Intra-category Relation Perception".

4 The accuracy improvement still seems limited. 

Overall, the modules designed in this paper are effective, and the motivation is clear. However, some issues still need further attention.
paper_summary: paper_summary: The authors propose HI2R, a Hypergraph-guided Intra- and Inter-category Relation Modeling approach, which simultaneously extracts intra-category structural information and inter-category relation information for more precise reasoning in fine-grained visual recognition. They introduce a Hypergraph-guided Structure Learning (HSL) module to capture high-order structural relations and an Inter-category Relation Perception (IRP) module to improve feature discrimination across categories, addressing limitations of previous methods. Additionally, a random semantic consistency (RSC) loss is implemented to enhance the model's focus on distinctive regions, improving the representation ability of both HSL and IRP modules.
strength_weakness: strength_weakness: 1. The motivation of this paper is clear. Leveraging both intra- and inter-category relation for FGVR is interesting and reasonable.
2. This paper proposes two novel module including a hypergraph-guided structural learning module and an inter-category relation perception module.
3. Compared with previous methods, significant performance improvement is observed in this paper.
limitations: limitations: 1. Model parameter and computational cost should be provided to for more thorough comparison.
2. In order to prevent background noise, a spatial-based background suppression operation is designed in HSL. What is the performance of the spatial-based background suppression being removed from HSL?
3. What is the connection and difference of the proposed IRP module with prototype learning? I suppose more visualization could be done to show what is learned in the concept cache.
paper_summary: paper_summary: Fine-Grained Visual Recognition (FGVR) necessitates distinguishing between similar subcategories, a skill humans excel at by balancing intra-category differences and inter-category similarities. 
HI2R innovatively integrates these dimensions using Hypergraph-guided Intra- and Inter-category Relation Modeling, capturing intricate intra-category structures and inter-category relations through Hypergraph-guided Structure Learning (HSL) and Inter-category Relation Perception (IRP) modules. 
HSL overcomes pairwise limitations of graph-based methods with high-order hypergraph connections, enhancing intra-category variability understanding, while IRP bolsters cross-category distinction. 
Complemented by Random Semantic Consistency (RSC) loss focusing on overlooked yet critical regions, HI2R significantly improves recognition accuracy, as evidenced by empirical results.
strength_weakness: strength_weakness: The author's ideas are simple and easy to understand The writing is clear. 

The experiments are relatively sufficient and prove the effectiveness of the method.
limitations: limitations: Several issues require attention:

1. Clarification is needed on the distinction between Concept Cache and category prototypes, as relevant work is currently lacking.

2. Semantic consistency is a common theme in contrastive learning. Could you elaborate on the specifics of the random erasing technique employed? Why were other image augmentation strategies not considered?

3. The paper would benefit from additional comparative studies. For instance: 
i) What is the actual gain from background suppression? The spatial background suppression strategy designed by the authors is complex. Could a simpler approach, such as using fully connected layers to predict weights [1], produce comparable performance? Further analysis drawing from related work is missing. 

ii) Why was Fuzzy C-Means clustering chosen over other methodologies?

4. The code has yet to be made publicly available. 

[1] Progressive Spatio-Temporal Prototype Matching for Text-Video Retrieval