Multi-granularity Feature Fusion Network for Cross-domain Person Re-identification

Authors: shaoqi  hou
zebang  qin
jiajie  wang
junqi  an
yusong  zhang
xinzhong  wang
zhiguo  wang
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-15 10:30:35

Abstract: In recent years, a lot of encouraging work has emerged in the field of domain generalization as one of the core tools for solving the cross-domain Person Re-identification (Re-ID) task. However, on the one hand, most of the domain generalization methods are devoted to the extraction and aggregation of global or local features, and lack the modeling of structured features of the human body with rich fine-grained information; on the other hand, the existing domain generalization methods emphasize the representation of a single feature, and do not take into full consideration the association and effective fusion of many features. To this end, we propose a Multi-granularity Feature Fusion Network (MFFNet) for cross-domain Re-ID, which utilizes the designed four branching features to construct a complete representation of pedestrians with rich fine-grained and associative information, by drawing on the design idea of graph convolution. Specifically, we effectively construct the intrinsic correlation between local features by introducing the Local Feature Comparison Module (LFCM). In addition, we design a Graph Convolution Module (GCM) to generate pose relational features and topological features with strong discriminative structured information. A series of ablation and comparison experiments on authoritative benchmarks show that the proposed MFFNet achieves competitive performance among similar algorithms.

Main paper: 
 i. introduction: 
 Cross-domain Person Re-identification (Re-ID) aims to test the model trained on known source domain data directly on unknown target domain data and expects to achieve good results. In recent years, with the rapid development of deep learning technology and the urgent need to deal with massive surveillance data, cross-domain Re-ID technology has become a core element of intelligent security construction. Although cross-domain Re-ID technology has made some good progress, its actual performance is still difficult to reach the point of ground application, occlusion, lighting, viewing angle, especially background differences, and other complex factors are still the key challenges affecting the performance of crossdomain Re-ID technology.Due to the significant distributional differences between source and target domain data, how to minimize the "domain difference" has become the core starting point for solving the cross-domain Re-ID task. Since then, many scholars at home and abroad have carried out a lot of substantive exploration around this goal.The aim of the domain generalization approach is to design a new model that has the ability to mine domain-invariant pedestrian features with strong robustness and discriminative information. Since it does not require the aid of an additional model, this approach is becoming one of the mainstream directions for "domain difference reduction" due to its convenience and efficiency.At the beginning, a number of algorithms used CNNs and other operations (e.g., HSV histograms, directional gradient histograms, etc.) to extract global features consisting of physical appearance (i.e., color, shape, texture features) [1] . However, extracting only global features inevitably misses some detailed features of pedestrians, so some scholars have focused on the extraction of local features that contain rich fine-grained information. Specifically, Sun et al [2] proposed a PCB network, which divides the pedestrian image into six horizontal chunks to extract different local features. At the same time, they designed an RPP method to enhance the consistency of local features. Guo et al [3] utilized parsing model and self-attention mechanism to extract the features of the human body part and the potential part, respectively, and combined their complementary strengths to enhance the characterization of each pixel. Zheng et al [4] proposed a global-to-local matching model to capture the spatial layout information of the pedestrian images. He et al [5] reconstructed feature maps for local queries from overall pedestrians, and the literature [6] further improved the foreground-background mask to avoid the effect of background clutter. Sun et al [7] proposed a visibility-aware pocalization model (VPM), which extracts region-level features and compares the shared visible regions of two images through selfsupervised learning.In addition, when extracting local features, the local region may contain invalid occluded parts. The re-identification task of occluded pedestrians is more challenging due to incomplete information and spatial misalignment. Zhuo et al [8] proposed a new human attention framework which uses an occlusion simulator to automatically generate artificial occlusions and uses multi-task loss to force the neural network to differentiate between occlusions and full body. Miao et al [9] proposed a new pose-guided feature alignment method that utilizes pose flags to separate useful information from occlusion noise. Fan et al [10] proposed a spatial-channel parallel network (SCPNet), where the features of each group of channels provide information about a particular spatial region of the human body, and used spatialchannel correlation to train the network. Luo et al [11] used a spatial transformation module to transform the overall image in order to align the local images and then calculated the distance of each set of matched pairs after alignment, in addition to attempting spatial alignment for a local Re-ID task.However, although the above methods have achieved certain results, they mainly focus on extracting global features or local features, and lack the modeling of structured features of the human body with rich fine-grained information. In addition, the above methods emphasize the expression of individual features and do not fully consider the correlation and effective fusion among many features.To address the above problems, we design a Multigranularity Feature Fusion Network (MFFNet), which consists of four branches, including global features, local relational features, gesture-relational features and topological features. The fusion of these branches together constitutes a complete representation of pedestrians with rich fine-grained and correlation information, which effectively improves the robustness and generalization ability of the model on the cross-domain Re-ID task. In particular, on the one hand, we enable the network to effectively construct the intrinsic correlations among local features by introducing the Local Feature Comparison Module (LFCM) from [12] . On the other hand, we design a Graph Convolution Module (GCM) for generating pose relational features and topological features with strong discriminative structured information by borrowing the idea of higher-order relational and topological modules from [13] .In order to fully demonstrate the effectiveness of the proposed scheme, we conduct complete ablation and comparison experiments on two authoritative benchmarks. The results show that our scheme still achieves competitive performance among similar algorithms even without the aid of additional complex models.The overall framework of our MFFNet is shown in Fig. 1 , which uses the classical ResNet-50 [14] as the backbone network. It can be seen that MFFNet extracts four kinds of features of pedestrians, including global features, local relational features, gesture relational features and topological features, and finally obtains a complete representation of pedestrians with rich granularity. Specifically, global feature generalizes the pedestrian image as a whole and pay more attention to the overall feature distribution of pedestrians, but they are easily affected by background clutter and occlusion; local features can well avoid the above problems, and they tend to pay more attention to the local information of pedestrians, such as the upper and lower body. However, supervised training of each local feature individually seems to have a better identification of the query using different local information, but it tends to confuse the local information of the same parts of different pedestrians. Therefore, we introduce the Local Feature Comparison Module (LFCM) from [12] to enable the network to learn the intrinsic correlation between a local feature and other local features, which makes the feature representation of pedestrians more discriminative, as shown in Fig. 2 .In addition, in order to further alleviate the occlusion problem as well as extract pedestrian features with strong robustness and discrimination, we introduce the higher-order relation module and topology module from [13] and combine them into the proposed Graph Convolution Module (GCM) (shown in Fig. 3 ), and the final outputs are obtained as the gesture relation features and topology features. The graph convolution technique can make full use of the intrinsic correlations between points and points, edges and edges to enable the network to learn effective and robust pedestrian feature representations, which on the one hand can better cope with the occlusion problem, and on the other hand enhances the final and complete representation of pedestrian features. Specifically, since we need to obtain the pedestrian's pose features before performing graph convolution, we use a good human pose estimator, LIP-JPPNet [15] , which can predict the positions of 16 key points of the human body, namely, the head, neck, spine, and pelvis individually as well as the shoulders, elbows, wrists, hips, knees, and ankles in pairs. In MFFNet, considering that a large number of local features (e.g., horizontal division and human posture) have been extracted and correlation learning has been carried out, in order to prevent network redundancy and learning conflicts, we do not carry out correlation processing on global features, and we obtain the pedestrian global features only by performing the global average pooling and fully connected operation on the features output from the backbone network. Finally, the global feature, local relational features, gesture relational features and topological features are fused to obtain the complete pedestrian feature representation of the proposed framework, which significantly improves the domain generalization ability of the model.Conv Conv Cat GAP f l2 f l3 f l4 f a1 Conv f l1 f ac1 f lc1 f lr1 Fig. 2. Structure of LFCM. f p1 f p2 f p15 f p16 A f pr GM f pr1 f pr2 U Cat Cat f t1 f t2 f g ...As shown in Fig. 1 , the processing flow of MFFNet is as follows:Step 1: The input image is processed by ResNet-50 to get the original feature map fo with dimension C×H×W.Step 2: Global feature acquisition. Global average pooling (which pays more attention to the overall style of the image) and fully connected operations are performed sequentially on the original feature map fo to obtain the global feature fg with dimension 1×1×C.Step 3: Acquisition of local relational features. First, the original feature map fo is partitioned into 4 regions in the horizontal direction on average to obtain 4 feature maps with dimensions C×H/4×W. Then after global maximum pooling (which pays more attention to the saliency information of the whole region) is performed for each block region to obtain 4 local features fl1, fl2, fl3 and fl4 with dimensions 1×1×C respectively.Then, the four local features are inputted into the LFCM for processing to obtain the local relational features fl1, fl2, fl3 and fl4, respectively. Fig. 2 illustrates the computation process of fl1, and the other local relational features are computed in the same way.As we know, in the past, local features were usually processed by stitching them together in a specific order and representing them as a structured form with a certain meaning. This approach is robust to the variation of pedestrian data, however, in real scenarios the data may be more diverse and difficult to learn, only limited information about pedestrians can be extracted using these features alone. Moreover, this approach does not exploit the rich correlation information hidden among many local features, and does not fully utilize the identification performance of the network. Therefore, we introduce LFCM to effectively alleviate this problem, so that each local feature contains both its own information and the information of other local features.Specifically, taking the computation of the local relational feature fl1 as an example, Global Average Pooling (GAP) is first performed on the other local features fl2, fl3 and fl4 to obtain the aggregated feature fa1: 1 1 1 3 a lj j ff     Then, 1×1 convolution is performed on fl1 and fa1 respectively to compress the number of channels to c, obtaining feature maps flc1 and fac1, both with dimensions 1×1×c. Next, flc1 and fac1 are channel spliced (the number of channels becomes 2c), and then processed by a 1×1 convolution unit (with the addition of the batch normalization and the ReLU function) to reduce the dimension from 2c to c. Finally, the the feature after channel number reduction is summed with flc1, the local relational feature flr1 containing both fl1 itself and other local information is obtained: 1 1 1 1Conv(Cat( , ))lr lc lc ac f f f f   where Cat(• ) denotes the splicing operation and Conv(• ) denotes the 1×1 convolutional unit. In particular, the other local relational features flr2, flr3 and flr4 can also be computed from Eq.(1) and Eq. ( 2).It should be emphasized that the loss functions used in the training of both the local relational features here and the two branches of the global relational features described above are the standard cross entropy loss and the triplet loss.Step 4: Acquisition of pose relational features and topological features. First, the input image is processed by the human posture estimator to obtain a feature map fk containing 16 keypoint coordinates, and fk is multiplied with the original feature map fo to obtain 16 posture features fp1, fp2…fp15 and fp16 with dimensions 1×1×C. Then, the posture features and the above global features fg are inputted into the GCM for processing, which is used to learn the pose relationship and topological relationship among different pedestrians.As shown in Fig. 3 , the GCM operates as follows:1) The difference between the global feature fg and a set of pose features fp (i.e., fp1, fp2 … fp16) is computed to obtain the weights of the edges between the pose points (which are used to characterize the correlation between the points), denoted as A.2) A is multiplied by fp, and then the obtained result is summed with fp to obtain the pose relational feature fpr: adp pr p p f A f f     The loss function for fpr still uses a combination of cross entropy loss (denoted by Lcls) and triplet loss (denoted by Ltri) with the following formula: ,, 1 1 1 [] 1 [ log max{ , 0}] a p a n K pr k cls tri k K k pr k L L L K p d d K              where K=16 and k  is the confidence level of the k-th key point (range is the interval [0,1]); ppr denotes the probability of predicting that fpr belongs to the real identity;  is the boundary value, (a,p) denotes the positive sample feature pairs from the same identity, and (a,n) denotes the negative sample feature pairs from different identities.3) Given two input images i1 and i2, follow the above steps to compute the two sets of pose relational features fpr1 and fpr2, respectively. the goal of Graph Matching (GM) is to learn the matching matrix U (of dimension K×K and values between 0 and 1) between fpr1 and fpr2, where the element Uij denotes the degree of match between the i-th point in fpr1 and the j-th point in fpr2.4) Define a symmetric matrix M (of dimension K 2 ×K 2 ), where the elements Mac;bd are used to measure the match between the edge (a,b) in fpr1 and the edge (c,d) in fpr2, and the pairs of points that do not form an edge are set 0 in the matrix. Therefore, the optimal matching U * is defined as Eq. ( 5), and we use the characteristics of the pairs of points to initialize M, and optimize U by a stochastic gradient descent algorithm. * T U U ar gm axU MU    5)After obtaining the matching matrix U, a set of topological features ft1 and ft2 are finally obtained by multiplication, splicing, convolution, and summation operations shown in Fig. 3 : 1 1 1 2 2 2 2 1Conv(Cat( , ))Conv(Cat( , ))t pr pr pr t pr pr pr f f f f U f f f f U        The loss function of the resulting topological features is shown in Eq. ( 7), y=1 when i1 and i2 are from the same pedestrian, otherwise y=0. 1 2 1 2 log | | (1 ) log(1 | |) t t t t t L y f f y f f        A. Implementation Details Datasets. Our experiments use two of the most commonly used public datasets in the Re-ID field, Market1501 [16] and DukeMTMC-reID [17] , whose details are shown in TABLE I . Specifically, M→D denotes training on Market1501 and testing on DukeMTMC-reID, while D→M does the opposite.All experiments were evaluated using the Cumulative Match Characteristic (CMC) and mean Average Precision (mAP) for Rank-1, Rank-5, and Rank-10, which are described in detail in the literature [18] .Experimental setup. The experimental environment we used is Python3.6+Torch1.4+Torchvision0.5+CUDA11.2, and the other experimental settings are the same as in the literature [12, 13] . For training, all input images are Resize to 256×128 (C and c mentioned above are set to 2048 and 256, respectively), and random horizontal flipping, random cropping, and random erasure are performed to enhance the diversity of the data. The Batch for each iteration is set to 64 (16 identities in total, with 4 images selected for each identity). In addition, the gradient update is performed using the Adam optimizer, with the initial learning rate set to 0.001 and reduced to 0.1 times the previous one after every 20 epochs of training, stopping after a total of 100 epochs of training. In particular, the total loss of this model is the weighted sum of the losses of each feature branch, and the weight parameter is set to 1 by default. The partitioning of different horizontal regions has an important impact on the computation of local relational features and their fusion with global features, pose relational features, and topological features. In order to explore the optimal number of chunks that can maximize the effect of GCM, we compute the local relational features and fuse them with other features on 2, 3, 4, 5 and 6 horizontal regions (i.e., average segmentation according to H/2, H/3, H/4, H/5 and H/6 in H dimension, respectively) and supervise the training of each region individually in each experiment, and the experimental results are shown in TABLE II . As can be seen from TABLE II , the performance of the proposed algorithm shows an increasing and then decreasing trend with the increase of the number of chunks. When the number of horizontal chunks is small, the performance of MFFNet is poor, which may be due to the fact that the local relational features derived at this time do not have more local characteristics, and even conflict with the global features, which can easily make the learning of the network misguided. When the number of horizontal chunks is higher, the metrics Rank-1 and mAP also undergo a significant decline, for example, when cross-domain identification is performed on M→D, the Rank-1 and mAP of the division of 6 chunks are 2.2% and 2.8% lower than those of the division of 5 chunks, respectively. It is sufficient to show that more chunks are not better, although more chunks may make the network more capable of distinguishing different pedestrians based on delicate local features, but it may lead to the conflict of multiple learning tasks as numerous pose features and topological features are also introduced in this paper. In addition, the more chunks are divided, the greater the number of parameters and computational effort of the overall model. Therefore, the 4-chunks horizontal chunks division is more suitable for the proposed network structure and enables the model to perform the best crossdomain identification performance, which will be consistent in subsequent experiments.Our MFFNet extracts global features, local relational features, pose relational features and topological features of pedestrian images and uses these features to obtain a complete feature representation of pedestrians with rich fine-grain and strong discriminative power, which gives the model strong robustness and generalization ability. In order to verify the effect of the fusion of different features mentioned above on the performance of cross-domain Re-ID, we have done a large number of ablation experiments, and the results are shown in TABLE III .From TABLE III, it can be seen that the worst reidentification performance is obtained by using only global features for training and testing. After adding local relational and topological features to the global features, respectively, the Rank-1 and mAP of the proposed model went up by about 4 percentage points. Further, the introduction of both local relational and topological features brings about a cumulative performance gain (more than 8%), which is enough to show that modeling the intrinsic associations between local features and aligned matching of human postures are necessary for the reidentification task. In addition, after adding the pose relational features to the global features, the model's re-identification effect is remarkable, with the metrics Rank-1 and mAP improving by about 12% for both the M→D task and the D→M task, which reflects sideways the robustness of the pose relational features to the most common occlusion problems in pedestrian images. Undoubtedly, the model after fusing the four features establishes a richer and more complete pedestrian feature representation with the addition of multi-granularity information, and also achieves the best re-identification performance. In order to demonstrate the superiority of the proposed MFFNet in terms of overall modeling, we have selected some state-of-the-art algorithms in recent years for comparison, including PUL [19] , TJ-AIDL [20] , and MMFA [21] , and a total of seven cutting-edge works.As can be seen from TABLE IV, the metrics of MFFNet have significant advantages over most of the selected algorithms, especially the most critical Rank-1 and mAP achieve the best performance on both M → D and D → M tasks. Specifically, compared with the best UDAP, our MFFNet exceeds the Rank-1 metrics by 0.7% and 6.2%, and the mAP metrics by 6.9% and 8.1% for the two cross-domain tasks, respectively. It is fully demonstrated that our mining, construction and fusion of global features, local relational features, pose relational features and topological features have accomplished an accurate description of the discriminative representation of pedestrian images in cross-domain scenes. We propose a multi-granularity feature fusion network (MFFNet) to enhance the robustness and generalization ability of the model on cross-domain Re-ID tasks. Different from the previous focus on processing a single feature descriptor, MFFNet effectively fuses multiple descriptors such as global features, local relational features, pose relational features, and topological features, which together form a complete representation of pedestrians with rich fine-grained and correlation information. To deal with the most challenging pedestrian occlusion problem, we draw on the principles of graph convolution operation and design a novel graph convolution module, GCM.Encouragingly, GCM dynamically controls the direction and degree of discriminative message passing by modeling the relationships between the pedestrian skeleton features point-to-point and edge-to-edge, and guiding the network to continuously learn the pose characteristics and topological features of the pedestrians in the image.The effectiveness of the proposed method is demonstrated by testing the two cross-domain tasks M→D and D→M. In future work, we will try to introduce the image style transformation method based on MFFNet, with the aim of designing a better crossdomain Re-ID scheme from both the model construction and data processing perspectives at the same time. 
 
 Appendix:  



Reviews: review: title: title: Current domain generalization techniques often focus on extracting and combining either global or local features but neglect the detailed modeling of structured human body features with fine-grained information. They also tend to prioritize individual feature representation without considering feature associations and effective fusion. To address these limitations, we introduce the Multi-granularity Feature Fusion Network (MFFNet) for cross-domain Re-ID tasks. MFFNet employs four distinct branching features to create a comprehensive pedestrian representation that includes fine-grained details and associative data. It incorporates the Local Feature Comparison Module (LFCM) for establishing relationships among local features and the Graph Convolution Module (GCM) for generating pose relational and topological features. Rigorous ablation studies and benchmark comparisons show that MFFNet performs competitively with other state-of-the-art algorithms.But they needExplain what assumptions and lemma are used, and explain the rationale. Originality needs to be further emphasized.
main_review: main_review: In order to solve most domain generalization techniques currently focus on extracting and combining either global or local features, overlooking the modeling of structured human body features that contain rich, fine-grained information. Furthermore, existing methods tend to concentrate on the representation of individual features, neglecting the importance of feature associations and effective fusion. To tackle these issues, we introduce the Multi-granularity Feature Fusion Network (MFFNet) tailored for cross-domain Re-ID tasks. This novel network harnesses four distinct branching features to craft a comprehensive pedestrian representation enriched with fine-grained details and associative data, inspired by graph convolution principles. Specifically, we’ve incorporated the Local Feature Comparison Module (LFCM) to efficiently establish the inherent relationships among local features. Additionally, we’ve devised a Graph Convolution Module (GCM) aimed at generating pose relational and topological features, packed with highly discriminatory structured information. Rigorous ablation studies and comparative experiments on established benchmarks demonstrate that our proposed MFFNet offers competitive performance among similar algorithms.
1.However,Originality needs to be further emphasized.
2.Explain what assumptions and lemma are used, and explain the rationale.
title: title: The paper is written clearly, exceptionally excellent.
main_review: main_review: This paper excels in terms of quality, clarity, originality, and significance, but I would still like to offer some suggestions.
1. Emphasize the original work of the paper.
2. Further emphasize the research motivation.
title: title: accept
main_review: main_review: This paper proposes a Multi-granularity Feature Fusion Network (MFFNet) for cross-domain Re-ID, which utilizes the designed four branching features to construct a complete representation of pedestrians with rich fine-grained and associative information, by drawing on the design idea of graph convolution. A series of ablation and comparison experiments on authoritative benchmarks show that the proposed MFFNet achieves competitive performance among similar algorithms. 
1). In abstract, about the proposed method, the statement is unclear. Authors need to rewrite abstract and to focus on the proposed method and to stress both the specific application and the generic aspects of the work.
2). The paper needs to clearly indicate its contributions to show the superiority of the proposed fuzzy approach on the previous approach.