It Takes Two: Accurate Gait Recognition in the Wild via Cross-granularity Alignment

Authors:   -𝛾
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 05:36:49

Abstract: Existing studies for gait recognition primarily utilized sequences of either binary silhouette or human parsing to encode the shapes and dynamics of persons during walking. Silhouettes exhibit accurate segmentation quality and robustness to environmental variations, but their low information entropy may result in sub-optimal performance. In contrast, human parsing provides fine-grained part segmentation with higher information entropy, but the segmentation quality may deteriorate due to the complex environments. To discover the advantages of silhouette and parsing and overcome their limitations, this paper proposes a novel cross-granularity alignment gait recognition method, named XGait, to unleash the power of gait representations of different granularity. To achieve this goal, the XGait first contains two branches of backbone encoders to map the silhouette sequences and the parsing sequences into two latent spaces, respectively. Moreover, to explore the complementary knowledge across the features of two representations, we design the Global Cross-granularity Module (GCM) and the Part Cross-granularity Module (PCM) after the two encoders. In particular, the GCM aims to enhance the quality of parsing features by leveraging global features from silhouettes, while the PCM aligns the dynamics of human parts between silhouette and parsing features using the high information entropy in parsing sequences. In addition, to effectively guide the alignment of two representations with different granularity at the part level, an elaborate-designed learnable division mechanism is proposed for the parsing features. Finally, comprehensive experiments on two large-scale gait datasets not only show the superior performance of XGait with the Rank-1 accuracy of 81.0% on Gait3D and 88.3% CCPG but also reflect the robustness of the learned features even under challenging conditions like occlusions and cloth changes.

Main paper: 
 introduction: 
 Gait, as a distinctive biometric feature, can be used to identify individuals based on their walking patterns. Unlike face and fingerprint, gait has lots of advantages, such as difficulty to disguise, remote Figure 1 : Comparisons of different gait recognition methods, i.e., GaitSet [4] , MTSGait [50] , GaitBase [7] , DyGait [39] , and ParsingGait [52] on the Gait3D dataset in terms of Rank-1 accuracy. (Best viewed in color.) accessibility, and non-contact nature. Consequently, gait recognition holds significant utility in intelligent monitoring and social security [24, 34] .In recent years, the studies for gait recognition have begun to transfer from laboratory environments to real-world scenarios [51, 53] . Due to the complex background, occlusion, and clothchanging in real-world scenarios, the traditional gait recognition methods often perform poorly [16, 51] . According to the recent research progress, we find that the appearance representations, i.e., silhouette and parsing sequences, gradually show certain advantages in this challenging task. For example, HSTL [37] utilizes the hierarchical clustering analysis to obtain multi-scale spatialtemporal gait features from silhouette sequences, which achieves Rank-1 accuracy over 60% on the Gait3D dataset. DyGait [39] proposes a dynamic augmentation module to extract dynamic features of gait from silhouette sequences. ParsingGait [52] for the first time uses parsing sequences as the gait representation and extracts dynamic gait features from human body parts, which obtains a 17.5% Rank-1 increase compared with the State-Of-The-Art (SOTA) gait recognition methods. However, although researchers have made great efforts in these two promising appearance representations, i.e., silhouette and parsing, their performance is still far from practical application.Silhouette sequences have been dominating the gait recognition community for a long time. Because they are easy to extract from the video frames, and compared to the sparse skeleton sequences, they also contain more useful gait information like appearance and belongings. However, the information entropy of the binary silhouette is still very low, resulting in unsatisfactory performance. Most recently, an emerging gait representation, i.e., gait parsing sequence, is proposed [52] . It can encode the shapes and dynamics of fine-grained human parts during walking, which can provide higher information entropy. Therefore, more and more researchers gradually pay attention to using parsing sequences for gait recognition. However, extracting parsing requires a more powerful human parsing model. In fact, the complex background, occlusion, and cloth-changing in real-world scenarios make it extremely difficult to train such a robust human parsing model. Consequently, the quality of the parsing is generally lower than that of the silhouette, which hinders it from achieving optimal performance.To this end, we propose a novel cross-granularity alignment gait recognition framework called XGait. It is the first framework to integrate the two most promising appearance representations, i.e., silhouette and parsing sequences, to achieve accurate gait recognition in the wild. The XGait framework comprises three primary stages. First, the silhouette sequence and gait parsing sequence are derived from the original RGB frames through segmentation and human parsing models. Next, we input these two appearance representations into separate CNN-based backbones to extract distinctive gait features. At last, we develop the Global Cross-granularity Module (GCM) and Part Cross-granularity Module (PCM) to achieve the granular alignment across global and part levels. The GCM is designed to enhance the quality of parsing features through the global appearance features extracted by silhouette sequences. The PCM is devised to align the dynamics of human body parts between silhouette and parsing features based on the high information entropy in parsing sequences. Moreover, to effectively handle the occlusion situation that is easy to appears in real-world scenarios, we propose a learnable division mechanism to guide the alignment of two granularity features at the part level. Finally, our XGait can leverage the superior segmentation quality of silhouette along with the high information entropy provided by parsing. Comprehensive experimental results on Gait3D and CCPG datasets demonstrate the effectiveness of our proposed method.In summary, the contributions of this paper are as follows:• To our knowledge, this study is the first work to investigate the integration of two most effective appearance representations, i.e., silhouette sequences and parsing sequences.Following the insightful analysis, we notice and explore their distinct advantages to achieve complementarity. • We propose a novel cross-granularity alignment gait recognition framework called XGait to unleash the power of silhouette and parsing. In XGait, the Global Cross-granularity Module (GCM) and Part Cross-granularity Module (PCM) are designed to achieve efficient granular alignment at the global and part levels, respectively. • We achieve the state-of-the-art performance on two largescale gait datasets, i.e., Gait3D and CCPG. Our XGait achieves the Rank-1 accuracy of 81.0% on Gait3D and 88.3% on CCPG, respectively. The experiments reflect that our method is robust under challenging conditions like occlusions and cloth changes. 
related work: 
 In this section, the gait representations are first surveyed. Next, we introduce the current gait recognition methods.Early gait representations primarily consisted of hand-designed models, such as 3D cylinders [2] and Chrono-Gait Image [35] . However, these hand-designed representations lack critical gait information. Subsequently, researchers began employing the frame subtraction method to extract pedestrian silhouettes and utilized them as the gait representations [45] . Building on this foundation, Han et al. [11] introduced the Gait Energy Image (GEI), a compressed representation derived from the silhouette sequence for gait feature extraction. However, the frame subtraction method severely constrains the scope of gait representations that can be extracted. Fortunately, due to the advancements in semantic segmentation and pose estimation algorithms [3, 21, 36] , an increasing number of gait datasets utilizing silhouette and skeleton data have emerged. Representative datasets can be categorized into the CASIA series [26, 28, 38, 45] and the OU-ISIR series [1, 12, 15, 23, 31, 32, 41] . Generally, due to its sparsity, skeletal information is not as comprehensive as that provided by silhouettes. The silhouette sequence has long been the predominant choice in the field of gait recognition. However, the aforementioned datasets are all gathered from controlled laboratory environments, posing challenges to the applicability of gait recognition technology. Recently, researchers have started constructing gait datasets from real-world environments, such as GREW [53] and Gait3D [51] . Challenges such as occlusion, 3D viewpoint changes, irregular walking routes, and various walking speeds in real-world environments cause difficulties for silhouette sequences to satisfy the requirements of the gait recognition task. Consequently, existing gait recognition methods often demonstrate poor performance on in-the-wild datasets [51, 53] , despite their success on in-the-lab datasets [15, 27, 28] .Most recently, Zheng et al. [52] proposed a novel gait representation called Gait Parsing Sequence (GPS), which exhibits significantly higher information entropy compared to the silhouette sequence. Substituting the silhouette sequence with GPS results in notable performance enhancements in contemporary appearance-based gait recognition methods. However, achieving more refined parsing results necessitates more strict requirements for human parsing models. This will result in a reduction in segmentation quality. Conversely, despite the low information entropy of silhouettes, their quality is ensured due to their straightforward characteristics. In general, silhouette and parsing, as promising appearance representations, each has its own strengths and limitations. We believe that integrating them will effectively improve gait recognition performance, which has not been deeply explored in the gait recognition community.In this section, we divide the existing gait recognition methods into single-representation and multi-representation approaches. Single-representation methods occupy the majority of the community and can also be divided into two main categories: modelbased methods and appearance-based approaches [34] . In earlier years, model-based approaches dominated the field of gait recognition, including the 2D/3D skeletons [17] , a structural human body model [2, 42] , etc. For example, Urtasun et al. [33] introduced a gait analysis technique that leverages articulated skeletons and three-dimensional temporal motion models. Zhao et al. [49] employed a local optimization algorithm to pursue motion tracking for gait recognition. Yamauchi et al. [43] proposed an interesting approach for human walking recognition by utilizing 3D pose estimation derived from RGB frames. Nevertheless, model-based techniques are less potent than appearance-based methods in gait recognition due to their inability to capture relevant gait features such as shape and appearance. Appearance-based approaches, also known as model-free methods, often utilize the gait silhouette sequence, gait parsing sequence and Gait Energy Image (GEI) as input [4, 9, 11, 13, 14, 18, 19, 25, 40, 47, 48, 52] . For example, Han et al. suggested the consolidation of a series of silhouettes into a concise Gait Energy Image (GEI) [11] and extracted gait knowledge from it. Chao et al. [4] treated the gait sequence as an unordered set and employed Convolutional Neural Networks (CNNs) to extract gait features from the frame-level and sequence-level. Fan et al. [9] proposed the GaitPart framework to horizontally split silhouettes and extract fine-grained features from each part. However, despite their impressive performance in controlled laboratory environments, these methods often falter when applied to real-world scenarios [51, 53] . In fact, silhouettes exhibit low information entropy, which renders them incapable of fully capturing the complexity of gait information in real-world scenarios.Most recently, the Gait Parsing Sequence (GPS) has been introduced as a novel gait representation that not only captures the body shape and appearance but also encompasses the dynamics of finegrained human body parts [52] . By replacing the input from the silhouette sequence with the GPS, existing appearance-based methods achieve a significant performance improvement, i.e., 12.5% ∼ 19.2% improvements in Rank-1 accuracy. Furthermore, Zheng et al. [52] proposed the ParsingGait network to extract the discriminative gait features from the global appearance information and the relations among human body parts. However, compared to silhouette, parsing necessitates a higher requirement for semantic segmentation models, i.e., human parsing models. Especially, through experiments on another challenging cloth-changing dataset, i.e., CCPG, we observe that the performance of parsing is inferior to the silhouette due to its low quality. Morte details can be found in Section 5.In recent years, multi-representation methods have been studied in the gait recognition community. For instance, Cui et al. [5] introduced the MMGaitFormer framework to realize the effective spatial-temporal feature fusion from silhouette and skeleton sequences. Fan et al. [8] proposed the SkeletonGait++ model to integrate the skeleton map and silhouette features. However, the fusion between silhouette and parsing sequences, the two most promising appearance representations, has not been studied by gait recognition researchers. This paper aims to explore their complementarity to achieve accurate gait recognition in the wild. 
method: 
 This section provides a detailed introduction to the proposed XGait framework. First, we provide an overview of our method in Section 3.1. Next, we discuss the silhouette and parsing encoding modules in Section 3.2. In Section 3.3, we explain how our Global Crossgranularity Module (GCM) utilizes the global appearance features extracted from silhouette sequences to improve the quality of parsing features. In Section 3.4, we introduce our Part Cross-granularity Module (PCM) and the designed learnable division mechanism, and explain how it establishes the fine-grained alignment between the silhouette and parsing features. Finally, we present the details of training and inference.The architecture of the proposed gait recognition framework, i.e., XGait, is illustrated in Figure 2 . The framework consists of three main stages: the preprocessing stage, the encoding stage, and the cross-granularity alignment stage.In the preprocessing stage, two appearance representations will be extracted offline from the original RGB sequence. One is the silhouette sequence 𝑆 ∈ R 𝑇 ×𝐶 ×𝐻 ×𝑊 obtained using a semantic segmentation model. Here, 𝑇 represents the sequence length, while 𝐶, 𝐻 , and 𝑊 denote the channel, height, and width of the frame, respectively. The other representation is the gait parsing sequence 𝑃 ∈ R 𝑇 ×𝐶 ×𝐻 ×𝑊 , extracted using a human parsing model.In the encoding stage, we utilize two separate CNN-based backbone networks to extract the distinctive characteristics of each appearance representation. Specifically, we obtain the feature maps F 𝑠 and F 𝑝 from the silhouette sequence 𝑆 and gait parsing sequence 𝑃, respectively. Here, F 𝑠 and F 𝑝 belong to R 𝑇 ×𝑐 ×ℎ×𝑤 , with 𝑐, ℎ, and 𝑤 representing the channel, height, and width of the feature maps.In the cross-granularity stage, the above two feature maps are fed into two branches: (1) the Global Cross-granularity Module (GCM) integrates these two granularity features at the global level, aiming to optimize the parsing features with relatively low segmentation quality through the global silhouette features. This global-level integration results in the feature map 𝐹 𝑔𝑎 . (2) the Part Cross-granularity Module (PCM) aligns the dynamics of human parts between silhouette and parsing features using the high information entropy in parsing, yielding the feature map 𝐹 𝑝𝑎 .Then, by merging F 𝑠 and F 𝑝 , we employ four separate Feature Mapping Heads (FMHs) to further extract more discriminative features, resulting in features F 𝑠 , F 𝑝 , F 𝑔𝑎 , and F 𝑝𝑎 . Subsequently, concatenating these features along the channel dimension yields F 𝑜𝑢𝑡 , as the final output feature.Finally, the cross-entropy loss 𝐿 𝑐𝑒 and the triplet loss 𝐿 𝑡𝑟𝑖 are employed to train the model.While both the silhouette sequence and parsing sequence are appearance representations, they possess distinct characteristics: 1) Semantic information: Parsing provides richer semantic details compared to the binary silhouette, including delineation of the head, left/right hand, etc. 2) Segmentation quality: The silhouette exhibits more accurate segmentation quality due to its simplicity and more robustness to environmental variations.Based on the above insights, we utilize two separate encoders to extract distinctive features from these two appearance representations. In particular, the silhouette encoding module 𝐹 𝑆 (•) and the parsing encoding module 𝐹 𝑃 (•) are established with ResNetlike structures [7] . The silhouette feature maps F 𝑠 and the parsing feature maps fF 𝑝 are then obtained from silhouette sequences and parsing sequences, respectively. it is worth noting that the backbone within the Silhouette and Parsing Encoding Modules can be substituted with any other gait recognition network. Although a stronger backbone may yield performance enhancements, this is not the focus of our method.The Global Cross-granularity Module (GCM) aims to utilize the global appearance features of the silhouette to improve the quality of parsing features. As illustrated in Figure 2 (c), a light-weighted module, i.e., Cross-granularity Alignment (CA) module, is designed to align F 𝑠 and F 𝑝 at the global level. To preserve the distinct semantic information, we concatenate F 𝑠 and F 𝑝 along the channel dimension. Before that, we employ Global Average Pooling (GAP) to compress the spatial dimension for computational consideration. The process is formulated as follows:EQUATIONwhere [•, •] represents the concatenation operation. Subsequently, we apply two Fully Connected (FC) layers, each with 𝑐/𝑟 and 2𝑐 neurons, respectively, where 𝑟 denotes the reduction ratio. Following the first FC layer, we employ the ReLU activation function. Then, a sigmoid layer is utilized after the final FC layer to derive element-wise weights for the silhouette feature 𝐹 𝑠 and the parsing feature 𝐹 𝑝 . Finally, the output feature F 𝑔𝑎 is obtained through element-wise weighted summation. These operations facilitate interaction between silhouette and parsing features at a global level. The described operations can be expressed as:EQUATIONwhere 𝑓 1 𝑐𝑎 (•) and 𝑓 2 𝑐𝑎 (•) represent the outputs of 𝑓 𝑐𝑎 (•) splitting along the channel dimension, with 𝑓 1 𝑐𝑎 (•) containing the first half and 𝑓 2 𝑐𝑎 (•) containing the second half. In addition, the detailed operation of the CA module can be expressed as follows:EQUATIONwhere W 1 and W 2 are the weights of the two FC layers, 𝜎 denotes the ReLU activation function.The Part Cross-granularity Module (PCM) is designed to take advantage of the high information entropy in parsing to align human body parts between silhouette and parsing features, as is shown in Figure 2 (d) . In general, the human body can generally be divided into three main parts: 1) the upper body, which includes the head and shoulders; 2) the middle body, comprising the torso, arms, and hands; 3) the lower body, encompassing the legs and feet. Meanwhile, previous studies [5, 9] have shown that different parts of the human body display distinct movement patterns during walking. Therefore, we constrain the silhouette and parsing features to capture mutual information associated with the relevant body parts. As illustrated in Figure 4 , we establish a part-level relationship between the silhouette and parsing features. The silhouette feature F 𝑠 is horizontally divided into three segments: the top quarter (0- 1 4 ), the middle half ( 14 -3 4 ), and the bottom quarter ( 3 4 -1). Similarly, following the categories of human body parts in [52] , we segment the parsing feature F 𝑝 into three regions: the upper body (head), the middle body (torso, left arm, right arm, left hand, right hand, dress), and the lower body (left leg, right leg, left foot, right foot, dress). Acknowledging that certain human body parts, such as the head, hands, and feet, are prone to occlusion due to their small size, we propose a learnable division mechanism to segment the parsing regional features, as depicted in Figure 3 . The mathematical expression for the learnable division is as follows:EQUATIONwhere 𝑖 = 1, 2, 3 is the index of the upper body, the middle body, and the lower body, respectively. 𝑀 𝑖 denotes the mask corresponding to the i-th human body part. 𝛾 𝑖 represents a learnable parameter used to modulate the weight of the feature associated with the i-th body part, and ⊙ denotes the element-wise multiplication.As is shown in Figure 4 , after obtaining distinct part-level silhouette and parsing features, we apply average pooling and max pooling to reduce the spatial dimension. Next, three independent CA modules, namely CA-Upper, CA-Middle, and CA-Down, are employed to extract the Cross-granularity features F 𝑖 𝑝𝑎 . Lastly, we concatenate the F 𝑖 𝑝𝑎 horizontally to obtain the output feature F 𝑝𝑎 of PCM. In this way, our XGait can fully leverage the complementary advantages of these two appearance representations, resulting in more robust and discriminative features for gait recognition.Four Feature Mapping Heads (FMHs) are implemented to enhance the efficiency of training and inference. These heads, applied to the discriminative gait features F 𝑠 , F 𝑝 , F 𝑔𝑎 , and F 𝑝𝑎 , utilize Set Pooling (SP) and Horizontal Pyramid Mapping (HPM) techniques [4] . The SP compresses temporal knowledge using max pooling, significantly reducing computational complexity. The HPM consists of Horizontal Pyramid Pooling (HPP) and Separate Mapping (SM). The HPP performs fine-grained sampling and horizontally reduces the dimension using max pooling and mean pooling. The SM utilizes separate fully connected layers for each pooled feature to aggregate more discriminative information. Taking F 𝑠 as an example, the above operations can be expressed as:EQUATIONWhere 𝑃 (•) represents the SP operation, 𝐻 (•) denotes the HPM process. At last, we concatenate F 𝑠 , F 𝑝 , F 𝑔𝑎 , and F 𝑝𝑎 to obtain the final feature vector F 𝑜𝑢𝑡 , which is used for sequence-to-sequence matching during both training and inference. For a comprehensive understanding of the SP and HPM, please refer to [4] .Our XGait framework is trained in an end-to-end manner. We optimize the network through a loss function comprising two components:EQUATIONwhere 𝐿 𝑡𝑟𝑖 is the triplet loss, 𝐿 𝑐𝑒 is the cross entropy loss. 𝛼 and 𝛽 are the weighting parameters.In the inference phase, the similarity between a query-gallery pair is assessed by computing the Euclidean distance. 
experiments: 
 We first introduce the datasets used in this work in Section 4.1. Furthermore, implementation details are outlined in Section 4.2. Section 4.3 presents a comprehensive comparison of our proposed XGait with existing gait recognition methods. Then, ablation studies on our framework are conducted in Section 4.4. At last, the visualization of heatmaps is provided in Section 4.5.Gait3D. The Gait3D dataset, proposed in [51] , is a challenging gait recognition dataset derived from real-world scenarios. It comprises 4,000 subjects, 25,309 sequences, and 3,279,239 frame images captured by 39 cameras in an unconstrained indoor environment, i.e., a large supermarket. To comply with the official train/test strategy outlined in [51] , 3,000 subjects are chosen for training, while another 1,000 subjects are reserved for testing. Moreover, each subject in the testing set has one sequence designated as the query, while the remaining sequences are used as the gallery. The Rank-1, Rank-5, mean Average Precision (mAP), and mean Inverse Negative Penalty (mINP) [44] are employed to evaluate the performance of the Gait3D dataset.CCPG. The CCPG dataset [16] is a recently introduced dataset designed for cloth-changing gait recognition. It consists of 200 subjects and 16,566 sequences captured using two outdoor cameras and eight indoor cameras. It provides a diverse array of outfits, encompassing 13 tops, 8 bottoms, and 5 distinct bags. Moreover, it includes a variety of walking routes, such as turning, occlusion, and background change routes. The above factors also make it an extremely challenging gait dataset. There are 100 subjects for training and another 100 for testing. In the testing phase, three types of cloth-changing scenarios are provided: cloth-changing (CL-FULL), ups-changing (CL-UP), and pants-changing (CL-DN). Rank-1 accuracy and mean Average Precision (mAP) are adopted as evaluation metrics to assess the performance of the CCPG dataset.Input. We utilize the officially provided silhouette and parsing data directly as inputs to our XGait model for the Gait3D dataset. For the CCPG dataset, where parsing is unavailable, we utilize CDGNet [20] to extract the parsing information. To ensure basic quality standards, we randomly sample and label 1,400 RGB images from the CCPG dataset. Labeling follows the guidelines outlined in [52] . The CDGNet model is then fine-tuned using the parameters published in [52] . Finally, we utilize the optimized CDGNet model to extract the parsing data from the CCPG dataset.Setting. Following the same preprocessing in [4] , silhouette and parsing images are normalized to 64 × 44 for both Gait3D and CCPG datasets. The silhouette and parsing encoding module employ the GaitBase [7] as the backbone. For the Gait3D dataset, the batch size is 32 × 2 × 30, representing 32 subjects, 2 sequences per subject, and 30 frames per sequence. The reduction ratio 𝑟 in the GCM is 1. The batch size for the CCPG dataset is 8 × 8 × 30. The reduction ratio 𝑟 is set to 16. Training the model for both Gait3D and CCPG datasets involves 1,200K iterations. The Learning Rate (LR) starts at 0.1 and is subsequently multiplied by 0.1 at the 400K, 800K, and 1,000K iterations. The SGD optimizer is adopted with a weight decay of 5e-4. The unordered sampling strategy is implemented in both datasets. The margin in the triplet loss is 0.2. In Equation 6, the weighting parameters 𝛼 and 𝛽 are set to 1.0. During the inference phase, all frames of each gait sequence are utilized, with a maximum of 720 frames per sequence for memory consideration.In this section, we compare the proposed XGait with several popular state-of-the-art (SOTA) gait recognition methods.Evaluation on Gait3D. The experimental results on the Gait3D dataset are listed in Table 1 . It is evident that model-based approaches, such as PoseGait and GaitGraph, typically underperform in comparison to appearance-based methods. This is because the utilization of sparse keypoints of the human body, leads to a deficiency in comprehensive gait information, further exacerbated by the complexity of real-world scenarios. In contrast, appearancebased methods tend to exhibit significantly superior performance. However, GEINet's performance is also poor due to significant information loss during the compression of the gait silhouette sequence into a gait energy image. Moreover, the approach utilizing parsing as input, namely ParsingGait, outperforms methods employing silhouettes such as GaitSet, GaitBase, and DyGait. This demonstrates the value of the high information entropy contained in the gait parsing sequence, offering more useful gait knowledge. Finally, our XGait achieves optimal performance, reaching a Rank-1 accuracy of 81%, showcasing the effectiveness of combining silhouette and parsing sequences harmoniously. Meanwhile, this illustrates the significant potential of our method for practical application in real-world scenarios. Evaluation on CCPG. Table 2 presents the experimental results obtained from the CCPG dataset, also known as the cloth-changing gait dataset. We can first observe that model-based approaches, such as GaitGraph2 [29] , GaitTR [46] , and SkeletonGait [8] , exhibit comparatively low performance. This once more demonstrates that sparse skeletons make it difficult for the model to acquire enough gait knowledge due to information loss. In contrast, appearancebased methods achieve better results, while our XGait method exhibits superior performance, confirming its effectiveness in tackling challenging scenarios like cloth-changing. This section presents ablation studies on key components. First, we analyze the impact of silhouette and parsing on gait recognition. Next, we examine the influence of different fusion modes. Subsequently, we demonstrate the significant performance enhancement achieved by the proposed GCM and PCM modules. Then, we investigate the shareability of the backbone and the mapping head. At last, different part division strategies are evaluated.Analysis of Silhouette and Parsing. As shown in Table 3 , using only silhouette as input results in a Rank-1 accuracy of 58.7%. However, replacing silhouette with parsing leads to a notable performance improvement, with Rank-1 accuracy increasing by 12.5%. This improvement can be attributed to parsing sequences having higher information entropy, offering more useful knowledge for gait recognition. In addition, our approach effectively integrates these two appearance representations, resulting in a significant performance enhancement. Specifically, compared to only using silhouette and parsing, the Rank-1 accuracy improved by 22.3% and 9.8%, respectively. For a detailed discussion on silhouette and parsing, please refer to Section 5.Fusion mode. We also assess the impact of various fusion modes, as listed in Table 4 . We first implement the distance fusion experiment, where the distance matrices of silhouette and parsing are directly combined and subsequently utilized for metric evaluation. This fusion technique is commonly employed in challenge competitions. When combined with Table 3 , it becomes evident that the distance fusion method also leads to performance improvements. In addition, we perform a feature fusion experiment by directly concatenating the two types of gait features after the backbone in the channel dimension. In other words, this is equivalent to removing the GCM and PCM modules from our XGait method. Surprisingly, we observe a significant performance improvement even with such a simple concatenation operation, resulting in Rank-1 and mAP increasing to 77.0% and 69.7%, respectively. This suggests a strong complementarity between silhouette and parsing representations. Furthermore, the fusion mode presented in our method enhances the exploration of mutual information between them, resulting in further improvements in Rank-1 and mAP accuracy, reaching 81.0% and 73.3%, respectively. Impact of GCM and PCM. Subsequently, we analyze the impact of the GCM and PCM modules in our XGait framework. The results are presented in Table 5 . Both GCM and PCM effectively enhance various evaluation metrics. Additionally, incorporating both modules results in further performance improvement. This highlights the validity of our proposed feature interaction approach from both global and part levels.Analysis of the shareability of backbone and mapping head. We also investigate the shareability of the backbone and the Feature Mapping Head (FMH). As is shown in Table 6 , we observe that performance is worst when backbone parameters are shared while FMH parameters are independent. This is because the differing semantics of silhouette and parsing. Sharing backbone parameters can interfere with model learning. Furthermore, independent FMH parameters exacerbate subsequent feature learning. When backbone parameters are independent, even if subsequent FMH parameters are shared, the model's performance is almost doubled. Finally, when both the backbone and FMH parameters are independent, the model achieves its best performance, with a Rank-1 accuracy of 81.0%.Analysis of the division strategy. Additionally, we perform experiments to validate the effectiveness of the proposed learnable division mechanism described in Section 3.4. Keeping other settings constant, we explore three distinct part division strategies: simple division, fixed division, and learnable division. We utilize Equation 4and Figure 3 to depict the distinctions among the three division strategies. Simple division and fixed division correspond to setting the learnable parameters 𝛾 to 1 and 0.75, respectively. The best results, as shown in Table 7 , are obtained when employing learnable division in the PCM. We also examine the specific values of the three learnable parameters in Equation 4 once the model converges. The final values of 𝛾 1 , 𝛾 2 , and 𝛾 3 are 1.1, 2.0, and 1.4, respectively. This indicates that the model automatically intensifies the search in the target area. Comparatively, the upper body receives the least attention, while the middle body and lower body receive more attention. In this sectoin, we visualize the heatmaps of F 𝑠 , F 𝑝 , F 𝑔𝑎 , and F 𝑝𝑎 in the cloth-changing and occlusion scenarios. As shown in Figure 5 , the feature map F 𝑔𝑎 focuses on the global region, indicating the GCM module's capability to enhance granularity features at the global level. The F 𝑝𝑎 concentrates on the lower body, which indicates that the movement of the legs and feet can effectively reflect useful gait information. 
discussion: 
 In this section, we aim to thoroughly analyze the influence of silhouette and parsing on gait recognition. As indicated in Table 8 , we are surprised to find that parsing shows a lower Rank-1 accuracy compared to silhouette on the CCPG dataset, with 81.4% accuracy compared to 83.9% accuracy with silhouette. Our analysts attribute this to the lower segmentation quality of parsing compared to the silhouette on the CCPG dataset. To verify this hypothesis, we conduct an operation where we intersect the silhouette and parsing, as depicted in Figure 6 . This operation ensures that the silhouette and parsing edges are identical, which effectively controls the segmentation quality variable. Subsequently, we conduct experiments utilizing the intersecting silhouette (𝑆𝑖𝑙 . * ) and the intersecting parsing (𝑃𝑎𝑟 . * ), finding that parsing yields superior results compared to the silhouette. Specifically, after the contour edges are aligned, the Rank-1 with the parsing is 4.1% higher than with the silhouette. This demonstrates that parsing's high information entropy significantly benefits gait recognition, even in cloth-changing scenarios. Finally, we replace the (𝑆𝑖𝑙 ., 𝑃𝑎𝑟 .) with (𝑆𝑖𝑙 . * , 𝑃𝑎𝑟 . * ) as inputs to our XGait framework. The experimental results indicate a notable performance reduction. In particular, the Rank-1 accuracy decreased from 88.3% to 82.5%. This is because forcing segmentation edge alignment results in the loss of the advantage of segmentation quality in silhouette sequences. In contrast, the XGait with original silhouette and parsing sequences effectively leverages the complementary information of these two appearance representations, realizing the best gait recognition performance. 
conclusion: 
 In this paper, we propose a novel cross-granularity gait recognition framework, named XGait, to unleash the power of different appearance representations, i.e., silhouette and parsing sequences. The Global Cross-granularity Module (GCM) and Part Cross-granularity Module (PCM) are developed to explore the complementary knowledge across the features of these two distinctive representations. Extensive experiments on two large-scale datasets, Gait3D and CCPG, validate the effectiveness of our method under challenging conditions like occlusions and cloth changes. In the future, further exploration of information interaction across more gait representations, such as silhouettes, parsing maps, 3D meshes, skeletons, etc., is expected to promote the development of gait recognition.  
 
 Appendix:  


Decision: All reviewers agree that this paper has somewhat novelty and vote for WA or BA. Although reviewers provided some critiques/suggestions to improve clarity in some parts, all reviewers agreed to accept it.

Reviews: review: paper_summary: paper_summary: The paper introduces XGait, a novel gait recognition framework that integrates silhouette and parsing sequences to enhance recognition accuracy in real-world scenarios. It proposes the use of Global and Part Cross-granularity Modules to align features at different levels, demonstrating improved performance on the Gait3D and CCPG datasets.
strength_weakness: strength_weakness: The strengths of the paper are:

Innovation: It introduces the first gait recognition framework, XGait, that combines silhouette and parsing sequences.
Theoretical Rationale: Provides a clear theoretical basis for the fusion of features of different granularities.

Technical Implementation: Detailed description of the model architecture, including key GCM and PCM modules.
Comprehensive Evaluation: Experiments on two large-scale datasets validate the effectiveness of the approach.

Clarity: Well-structured, with figures and tables that aid understanding.

Application Potential: Implicit in discussions of robustness under challenging conditions, though not explicitly stated.
limitations: limitations: The paper's weakness is specifically in the insufficient evaluation of the model's robustness to tracking errors. There is no detailed discussion or experimental data provided on how inaccuracies in the initial frame alignment or tracking stability throughout the gait sequence affect the final recognition accuracy. Given that gait recognition in the wild often relies on video sequences with potential motion blur or occlusions, understanding the model's sensitivity to such tracking errors is essential for assessing its practical viability.
paper_summary: paper_summary: The paper aims to improve gait recognition on in-the-wild datasets by combining silhouette and human parsing sequences, the two most promising gait representations.
A cross-granularity framework is proposed to exchange and fuse the complementary knowledge from these two granularity (i.e., global shape and human parts) features.
Experimental results on two large-scale in-the-wild datasets demonstrate that the proposed method achieves state-of-the-art performance, outperforming previous well-known gait recognition systems such as GaitSet, GaitBase and ParsingGait.
strength_weakness: strength_weakness: **Novelty**:
According to the authors, this paper is the first study to explore the complementarity of silhouette sequence and human parsing sequence for gait recognition, based on the observation that they have different granularity, information entropy and annotation quality.
The proposed GCM and PCM modules are straightforward but still somewhat novel.

**Significant results**:
The proposed method successfully achieves state-of-the-art performances on two challenging gait datasets.
Specifically, the two core designs in the paper, namely GCM and PCM, effectively improve the fusion between silhouette and parsing maps, according to Table 4 & 5.
limitations: limitations: **Additional cost**:
The paper does not clarify the computational costs (at training / inference) to run the proposed model.
According to the paper, it utilizes additional human parsing model to get parsing results, two independent GaitBase backbones during the encoding stage, and the subsequent alignment modules (GCM & PCM).

Therefore, the overall computational cost should be greater than 2 x GaitBase costs, and the comparison in Table 1 & 2 may be not fair considering this doubled overhead --- what if the baseline methods use 2x model parameters / flops?

**Some details**:
The reduction ratio $r$ for the bottleneck in the alignment MLP modules are varying on different datasets ($1$ on Gait3D, $16$ on CCPG).
As a hyper-parameter that controls network architecture (instead of some dataset-dependent one like loss weighting or image resolution), intuitively, this $r$ value should not be so sensitive to dataset choices.
The paper could be improved if the hyper-parameter searching and the analysis are presented.
paper_summary: paper_summary: This paper proposes a framework for gait recognition using silhouette sequences and parsing sequences, XGait. This framework extracts global information through silhouette sequences. The partial information is extracted through parsing sequences, and the Part Cross-granularity Module and the Global Cross-granularity Module are designed to use the two kinds of information to complement each other. State-of-the-art results are obtained on two large datasets.
strength_weakness: strength_weakness: + State-of-the-art performance is achieved on two large-scale gait datasets.
+ The first work combines silhouette sequence and parsing sequences for gait recognition, and obtains global information and local information through these two representations. 
+ A neural network framework, XGait, is put forward to integrate the information of different granularity, so as to complement each other's advantages.
+ The conclusion is proved by complete experimental data and visualization results.
limitations: limitations: - The usage scenario mentioned in the title is in the wild, but Gait3D is an indoor dataset, while CCPG contains more indoor cameras than outdoor cameras, and the characteristics in the wild are not reflected in the experiment.
- An important evaluation criterion for gait recognition methods is how effective they are in cross-view. This paper has not conducted cross-view experiments on datasets such as CASIA-B.
- In Part Cross-granularity Module, how to determine the value of   is not explained clearly.