# Impostor Resilient Multimodal Metric Learning for Person Reidentification

Authors: Muhamamd Adnan Syed; Zhenjun Han; Zhaoju Li; Jianbin Jiao
Journal: Advances in Multimedia
 (2018)

---

## Abstract

In person reidentification distance metric learning suffers a great challenge from impostor persons. Mostly, distance metrics are learned by maximizing the similarity between positive pair against impostors that lie on different transform modals. In addition, these impostors are obtained fromGallery view for query sample only, while the Gallery sample is totally ignored. In real world, a given pair of query and Gallery experience different changes in pose, viewpoint, and lighting. Thus, impostors only fromGallery view can not optimally maximize their similarity. Therefore, to resolve these issues we have proposed an impostor resilient multimodal metric (IRM3). IRM3 is learned for each modal transform in the image space and uses impostors from bothProbe andGallery views to effectively restrict large number of impostors. Learned IRM3 is then evaluated on three benchmark datasets, VIPeR, CUHK01, and CUHK03, and shows significant improvement in performance compared to many previous approaches.


---

## Body

## 1. Introduction

Person reidentification (Re-ID) matches a given person across a large network of nonoverlapping cameras [1], and is fundamentally used for person tracking in camera networks. Despite years of research, reidentification is still a challenging problem as the data space in Re-ID is multimodal (modal in our work is defined as the space which is formed by the joint combination of different changes a given pair images of the same person undergo in different camera views) and the observed images in different views undergo various different changes in poses [2], viewpoints [3], lighting [4], background clutter, and also experience occlusion.Most approaches in Re-ID can mainly be divided into two categories: robust features extraction [5–13] for representation and globally learning distance metric for matching [14, 15]. These global metrics [16–19] project features into low dimension subspace where they tend to maximize the discrimination among different persons; however, these metrics still suffer a great challenge from impostor (an impostor is a person that belongs to the other person and, however, possess higher similarity with the given query than the right Gallery sample) samples [20, 21]. Though, in past some attempts are made to eliminate impostors [14, 20–22], however, all these attempts have not given due consideration of different transform modals on which the reidentification images lie [23]. This situation is illustrated in Figure 1, where we have shown three transform modals M1, M2, and M3 in the image space. M1 contains a positive pair (query and Gallery) enclosed in green rectangles for which a metric is learned, while there are two more pairs lying in modals M2 and M3, respectively. View b images (enclosed in red rectangles) in M2 and M3 are similar to query in M1 and, thus, are impostors for query sample. In conventional approaches [14, 20–22], the metric between query and Gallery samples in M1 is learned using the impostor sample from M2 (Metric DM1) or M3 (Metric DM2) as a constraint. Therefore, when the similarity for positive pair is learned under the constraint of an impostor person lying on a different transform modal other than the positive pair, then the learned similarity metric would not be the optimal matching function, which can be proved from poor retrieval results inRanklist 1 andRanklist 2 in Figure 1.Figure 1
Three ModalsM1, M2, and M3 in Image Space. Query and Gallery lie in modal M1, while one impostor for query lies in modal M2 and the other in modal M3. Metric DM1 is learned using the impostor from modal M2, and DM2 is learned using the impostor from modal M3. Then, the obtained retrieval results of DM1 and DM2 are shown inRanklist 1 andRanklist 2, respectively. Correct Match is in green rectangle.Further, in Figure1 previous approaches [14, 20–22] have used impostor samples for query sample only from theGallery view, while totally ignoring the Gallery sample. Therefore, to resolve the above shortcomings in [14, 20–22] we have proposed an impostor resilient multimodal metric, referred as IRM3, which eliminates the impostors largely and attains an optimal matching between positive pair. The objective of IRM3 is to maximize the matching of a positive pair against both the negative gallery samples (NGS) (samples which are not impostors and belong to different persons), as well as against impostors by taking into account the modal a given pair, its negative gallery samples, and its impostors reside. Further, in contrast to [14, 20–22], it also takes into consideration impostor samples for both the query and its respective Gallery sample. This pair of impostors are referred to asCross views impostors (CVI) which are obtained for query and Gallery samples from their opposite views and help in further maximizing the similarity between given query and Gallery samples. The contributions of our impostor resilient multimodal metric IMR3 are as follows:(i)
Improving impostors resistance by jointly exploiting the transform modals [23], as well as impostor samples from bothProbe andGallery views;(ii)
With our IMR3 approach a significant gain in performance is obtained in Multikernel Local Fisher Discriminant Analysis (MK-LFDA) [44].


## 2. Methodology

Figure2 shows the framework of our IRM3. In Figure 2, first color and texture features are extracted from each training sample; then, different modals are discovered in the image space. These modals are discovered by using sum of squares clustering which is explained in Section 2.2. Finally, for each modal cross views impostors (CVI) (explained in Section 2.3) and negative gallery samples (NGS) (explained in Section 2.4) are generated to train the modal metric Mk for each transform modal k. In our work, the modal metric Mk is learned using MK-LFDA [44], and the learning procedure is explained in Section 2.6. Finally, in Section 2.7 we have explained how we have performed matching between test query and Gallery.Figure 2
Impostor resilient multimodal metric learning (IRM3) for person reidentification.


### 2.1. Feature Extraction

RGB, HSV, LAB, YCbCr, and SCNCD histograms are extracted according to similar settings in [45] using 32 bins per channel, and settings in [12], respectively. Then, all five features are concatenated together. Similarly, DenseSIFT, SILTP, and HOG are extracted according to the settings in [46], [11], and [47], respectively, and are concatenated together. Dimension of color and texture features after concatenation become large, and since Re-ID data is multiview we have used CCA [48] to reduce dimension. However, to keep the local discriminative information of each type of feature we have applied CCA to color and texture features individually. By cross validation on VIPeR and CUHK03 we obtained optimal dimension for color feature to be 900, and texture feature to be 700. Finally, the reduced color and texture features are concatenated to form a feature vector F of size 1600.


### 2.2. Partition Image Space

LetX be the image space of a camera view; then X is(1)X=xii=1,…,n,where xi is the feature representation Fi of person i and n are the number of persons in X. Since images in X lie on different transform modals, therefore, there exist distinct clusters of different modals in X. Each of these modal clusters has its own unique transformation and visual patterns; thus, all the persons belonging to a modal k can be obtained using sum of squares clustering as(2)Sw=1n∑k=1K∑i=1nzk,ixi-mkxi-mkT,where K are the number of modals in X, Sw is scatter matrix of within transform modals, zk,i is the association of xi with transform modal k, and mk is the center of the kth transform modal.In (2), each modal center mk is critical in discovering distinct, stable, and nonempty modals in X. Thus, choosing any sample (xia,xib) as center mk of any given modal k, it is necessary to make sure it is a right choice. In order to make sure a chosen modal center is right it has to fulfill two conditions: First, (i) if the chosen sample (xia,xib) is a center of modal cluster k, then, all the persons in modal k will be its neighbors, and it has the highest number of nearest neighbors. Second, (ii) center mk and all its nearest neighbors lie on the same modal; therefore, these neighbors will share similar patterns with the center mk in bothProbe andGallery views.Now, we compute the number of nearest neighbors for each person in training set by taking into consideration the above two conditions. For this purpose, we have used both Probe(xia) and Gallery (xib) samples of each person to obtain four lists of neighbors, which are computed from both camera views. To acquire most reliable neighbors we then select only top@40 (here, the reason to choose top@40 neighbors is to maintain maximum reliability with minimum time and memory cost in large datasets. For instance, when we have k=16 modals in CUHK03, then, in each modal there will be at least 78 training persons. Now, to obtain a center sample xi of any modal it must have at least 51% neighbors in that modal, and thus, we take top@40 neighbors which is in actual 52% proportion of the training persons in a modal to find out whether xi is a center or not) (top@20 for VIPeR) neighbors from each list and then perform an intersection operation among all the four lists to obtain the cardinality value, as well as IDs of the neighbors which are common in bothProbe andGallery views of a given person. This cardinality value and the IDs of the obtained neighbors are then stored in a matrix. Further, this procedure is repeated for the rest of the remaining n-1 persons in the training set, and then their cardinality values, as well as IDs of the neighbors, are also stored in the same matrix.Using this matrix we will now obtain ourK initial centers for K modal transforms. These K centers are chosen as the K top persons with highest number of neighbors. However, it could be possible that two or more persons can have the same cardinality value, as well as share the same nearest neighbors IDs. In that condition simply choosing top K persons will not be the best solution; instead, we chose only those top K persons that do not have any person IDs common in their neighbors lists. In addition, for situations where more than two persons have the same cardinality and share same neighbors IDs, we randomly chose any one person from them to represent that modal center. Finally, getting the K modal centers the optimal partitioning of the image space X is obtained by minimizing the trace of within transform modals scatter matrix as(3)arg⁡min⁡trSw.Though the image space is partitioned into K modals, however, to ensure the obtained modals are distinct and stable (in our work a stable modal is formed when it contains at least 15% training persons) we have updated the modal centers and repartitioned the space for further t=3 times. The modal centers are updated as(4)mk=1Nk′∑i=1nzk,i∗xi,where Nk′ is the number of persons in modal k and given as(5)Nk′=∑i=1nzk,i.Computing the initial modal centers is computationally tedious in our work, however, it has still moderate computational burden. For the training size of n persons the complexity is about O(t×K×n), where t is the number of iterations, and K is the number of modals.


### 2.3. Cross Views Impostors (CVI)

After getting the distinct modals in the image spaceX, we can now obtain the set of CVI for each positive pair (xia,xib) lying in modal k from both of itsProbe andGallery views. We believe in real world situation (open set) where a positive pair has always limited or few samples; these CVI can be exploited to deliver subtle and differentiating information in metric learning that can differentiate a given pair more efficiently against large number of diverse real world impostors, as well as negative gallery samples. These impostors are obtained by comparing the similarity value of a given person pair against the other persons inGallery andProbe views. First, the similarity values for a Probe sample xia are computed with the wholeGallery view using metric Mini and CCA reduced feature F as(6)Sprobei=xia-xjbTMinixia-xjb,where xi and xj are CCA reduced feature F of person i and j, while Mini is a globally learned metric with feature F using K-LFDA [45]. We have usedlinear kernel to save memory and computational time. Similarly, the similarity values for Gallery person xib are obtained with the wholeProbe view as(7)Sgalleryi=xib-xjaTMinixib-xja.These obtained values Sprobei and Sgalleryi for person (xia,xib) in modal k are then stored into two sets as(8)Simxia=Sprobe,i′ii′=1,…,Nk′,Simxib=Sgallery,i′ii′=1,…,Nk′,where Nk′ refers to the number of persons in a modal k. Now, we compare each similarity value in these sets with the reference similarity value S(xia,xib)ref of a given pair (xia,xib) to obtain its CVI set Set(xia,xib)C.V.I. as(9)Setxia,xibCVI=xp,herexp=Sprobe,pi<Sxia,xibref,orxp=Sgallery,pi<Sxia,xibref,and p is the index of impostor person, and S(xia,xib)ref is computed as(10)Sxia,xibref=xia-xibTMinixia-xib.Further, using (6)–(10), CVI set Set(xia,xib)C.V.I. for all the Nk′ persons in the modal k are computed. The computational cost of generating cross views impostors for a modal k is about O(3×Nk′), where Nk′≪n.


### 2.4. Negative Gallery Samples (NGS)

We have also used negative gallery samples (NGS) to learn metricMk. Set of NGS, denoted as Set(xia,xib)Ng, for person pair (xia,xib) are obtained from Gallery view only as(11)Setxia,xibNg=xqb,whereq≠pinSetxia,xibCVI,q≠ifor  probei,where q is the index of NGS Further, the set of NGS Set(xia,xib)Ng for all Nk′ persons in modal k is then obtained using (11).


### 2.5. Triplet Formation

Getting the set of CVISet(xia,xib)C.V.I. and NGS Set(xia,xib)Ng for all Nk′ persons in modal k we will now generate triplet samples to learn metric Mk. Since the positive samples for each person xi are too scarce compared to the number of negative samples, therefore, following the protocol of data augmentation in [49] we augment each person pair five times. Similarly, following the protocol in [39] we generate 20 triplets for each positive pair. Now, the triplet samples Tiimp and TiNg for person xi using impostor p and negative Gallery q are given as(12)Tiimp=xia,xib,p,TiNg=xia,xib,q,where p and q are taken from respective sets Set(xia,xib)C.V.I. and Set(xia,xib)Ng of person xi.


### 2.6. Impostor Resilient Multimodal Metric (IRM3)

Taking triplets fromTiimp and TiNg, metric IRM3 for modal k is learned using MK-LFDA [44]; however, to save both the computational time and memory requirements we adapted [44] and use three RBF kernels and one χ2 kernel. The weights for these kernels are learned globally for once for each dataset in our work using the similar method in [44]. The reason to learn weights globally is to save both time and computational burden. Further, there is considerably minor effect on kernel weights; even the weights are learned globally. This is due to the fact that the global space is comprised of all the existing modals, and thus, all the modals contribute in learning the global weights. For learning weights of kernels all the extracted features are used individually, and the dimensions of these features are also individually reduced to 450 by CCA before learning weights. In all our experiments the obtained weights for VIPeR are 0.3, 0.22, and 0.22 for RBF kernels, while weight for χ2 kernel is 0.26. For CUHK01 and CUHK03 the obtained weights for RBF kernels are 0.28, 0.24, and 0.24, while weight for χ2 kernel is 0.24. The σ values in all the datasets for the three RBF kernels are set to the mean value of modal k, as well as (mean value + mean/2) and (mean value-mean/2). These values for σ are chosen to model all the different variations in the modal k, while the σ value for χ2 kernel is also set to mean value of modal k. The mean value in our work is the similarity value between Probe and Gallery samples of center mk. Finally, the metric Mk is learned as(13)maxMk⁡trMkTSBMkMkTSWMk,where matrices SB and SW are obtained with similar method in [44]. Now, (13) is then solved using generalized eigenvalue problem [50] in (14) to obtain first r′= 300 eigenvectors corresponding to eigenvalues with largest magnitude as(14)SBφ=λSWφ.


### 2.7. Reidentification

From Figure2, reidentification between test pair (xia,xjb) is performed by first determining the transform modal the test pair belongs to usingK-NN classifier. InK-NN classifier, the parameterK is set to the number of modals in the image space; that is, in VIPeR the value ofK is set to the number of modals k=7. Then, the features of (xia,xjb) are projected into the weighted multikernel space of the respective modal, followed by the respective modal metric Mk to perform matching as(15)dxia,xjb=xia-xjbTMkxia-xjbT.


## 2.1. Feature Extraction

RGB, HSV, LAB, YCbCr, and SCNCD histograms are extracted according to similar settings in [45] using 32 bins per channel, and settings in [12], respectively. Then, all five features are concatenated together. Similarly, DenseSIFT, SILTP, and HOG are extracted according to the settings in [46], [11], and [47], respectively, and are concatenated together. Dimension of color and texture features after concatenation become large, and since Re-ID data is multiview we have used CCA [48] to reduce dimension. However, to keep the local discriminative information of each type of feature we have applied CCA to color and texture features individually. By cross validation on VIPeR and CUHK03 we obtained optimal dimension for color feature to be 900, and texture feature to be 700. Finally, the reduced color and texture features are concatenated to form a feature vector F of size 1600.


## 2.2. Partition Image Space

LetX be the image space of a camera view; then X is(1)X=xii=1,…,n,where xi is the feature representation Fi of person i and n are the number of persons in X. Since images in X lie on different transform modals, therefore, there exist distinct clusters of different modals in X. Each of these modal clusters has its own unique transformation and visual patterns; thus, all the persons belonging to a modal k can be obtained using sum of squares clustering as(2)Sw=1n∑k=1K∑i=1nzk,ixi-mkxi-mkT,where K are the number of modals in X, Sw is scatter matrix of within transform modals, zk,i is the association of xi with transform modal k, and mk is the center of the kth transform modal.In (2), each modal center mk is critical in discovering distinct, stable, and nonempty modals in X. Thus, choosing any sample (xia,xib) as center mk of any given modal k, it is necessary to make sure it is a right choice. In order to make sure a chosen modal center is right it has to fulfill two conditions: First, (i) if the chosen sample (xia,xib) is a center of modal cluster k, then, all the persons in modal k will be its neighbors, and it has the highest number of nearest neighbors. Second, (ii) center mk and all its nearest neighbors lie on the same modal; therefore, these neighbors will share similar patterns with the center mk in bothProbe andGallery views.Now, we compute the number of nearest neighbors for each person in training set by taking into consideration the above two conditions. For this purpose, we have used both Probe(xia) and Gallery (xib) samples of each person to obtain four lists of neighbors, which are computed from both camera views. To acquire most reliable neighbors we then select only top@40 (here, the reason to choose top@40 neighbors is to maintain maximum reliability with minimum time and memory cost in large datasets. For instance, when we have k=16 modals in CUHK03, then, in each modal there will be at least 78 training persons. Now, to obtain a center sample xi of any modal it must have at least 51% neighbors in that modal, and thus, we take top@40 neighbors which is in actual 52% proportion of the training persons in a modal to find out whether xi is a center or not) (top@20 for VIPeR) neighbors from each list and then perform an intersection operation among all the four lists to obtain the cardinality value, as well as IDs of the neighbors which are common in bothProbe andGallery views of a given person. This cardinality value and the IDs of the obtained neighbors are then stored in a matrix. Further, this procedure is repeated for the rest of the remaining n-1 persons in the training set, and then their cardinality values, as well as IDs of the neighbors, are also stored in the same matrix.Using this matrix we will now obtain ourK initial centers for K modal transforms. These K centers are chosen as the K top persons with highest number of neighbors. However, it could be possible that two or more persons can have the same cardinality value, as well as share the same nearest neighbors IDs. In that condition simply choosing top K persons will not be the best solution; instead, we chose only those top K persons that do not have any person IDs common in their neighbors lists. In addition, for situations where more than two persons have the same cardinality and share same neighbors IDs, we randomly chose any one person from them to represent that modal center. Finally, getting the K modal centers the optimal partitioning of the image space X is obtained by minimizing the trace of within transform modals scatter matrix as(3)arg⁡min⁡trSw.Though the image space is partitioned into K modals, however, to ensure the obtained modals are distinct and stable (in our work a stable modal is formed when it contains at least 15% training persons) we have updated the modal centers and repartitioned the space for further t=3 times. The modal centers are updated as(4)mk=1Nk′∑i=1nzk,i∗xi,where Nk′ is the number of persons in modal k and given as(5)Nk′=∑i=1nzk,i.Computing the initial modal centers is computationally tedious in our work, however, it has still moderate computational burden. For the training size of n persons the complexity is about O(t×K×n), where t is the number of iterations, and K is the number of modals.


## 2.3. Cross Views Impostors (CVI)

After getting the distinct modals in the image spaceX, we can now obtain the set of CVI for each positive pair (xia,xib) lying in modal k from both of itsProbe andGallery views. We believe in real world situation (open set) where a positive pair has always limited or few samples; these CVI can be exploited to deliver subtle and differentiating information in metric learning that can differentiate a given pair more efficiently against large number of diverse real world impostors, as well as negative gallery samples. These impostors are obtained by comparing the similarity value of a given person pair against the other persons inGallery andProbe views. First, the similarity values for a Probe sample xia are computed with the wholeGallery view using metric Mini and CCA reduced feature F as(6)Sprobei=xia-xjbTMinixia-xjb,where xi and xj are CCA reduced feature F of person i and j, while Mini is a globally learned metric with feature F using K-LFDA [45]. We have usedlinear kernel to save memory and computational time. Similarly, the similarity values for Gallery person xib are obtained with the wholeProbe view as(7)Sgalleryi=xib-xjaTMinixib-xja.These obtained values Sprobei and Sgalleryi for person (xia,xib) in modal k are then stored into two sets as(8)Simxia=Sprobe,i′ii′=1,…,Nk′,Simxib=Sgallery,i′ii′=1,…,Nk′,where Nk′ refers to the number of persons in a modal k. Now, we compare each similarity value in these sets with the reference similarity value S(xia,xib)ref of a given pair (xia,xib) to obtain its CVI set Set(xia,xib)C.V.I. as(9)Setxia,xibCVI=xp,herexp=Sprobe,pi<Sxia,xibref,orxp=Sgallery,pi<Sxia,xibref,and p is the index of impostor person, and S(xia,xib)ref is computed as(10)Sxia,xibref=xia-xibTMinixia-xib.Further, using (6)–(10), CVI set Set(xia,xib)C.V.I. for all the Nk′ persons in the modal k are computed. The computational cost of generating cross views impostors for a modal k is about O(3×Nk′), where Nk′≪n.


## 2.4. Negative Gallery Samples (NGS)

We have also used negative gallery samples (NGS) to learn metricMk. Set of NGS, denoted as Set(xia,xib)Ng, for person pair (xia,xib) are obtained from Gallery view only as(11)Setxia,xibNg=xqb,whereq≠pinSetxia,xibCVI,q≠ifor  probei,where q is the index of NGS Further, the set of NGS Set(xia,xib)Ng for all Nk′ persons in modal k is then obtained using (11).


## 2.5. Triplet Formation

Getting the set of CVISet(xia,xib)C.V.I. and NGS Set(xia,xib)Ng for all Nk′ persons in modal k we will now generate triplet samples to learn metric Mk. Since the positive samples for each person xi are too scarce compared to the number of negative samples, therefore, following the protocol of data augmentation in [49] we augment each person pair five times. Similarly, following the protocol in [39] we generate 20 triplets for each positive pair. Now, the triplet samples Tiimp and TiNg for person xi using impostor p and negative Gallery q are given as(12)Tiimp=xia,xib,p,TiNg=xia,xib,q,where p and q are taken from respective sets Set(xia,xib)C.V.I. and Set(xia,xib)Ng of person xi.


## 2.6. Impostor Resilient Multimodal Metric (IRM3)

Taking triplets fromTiimp and TiNg, metric IRM3 for modal k is learned using MK-LFDA [44]; however, to save both the computational time and memory requirements we adapted [44] and use three RBF kernels and one χ2 kernel. The weights for these kernels are learned globally for once for each dataset in our work using the similar method in [44]. The reason to learn weights globally is to save both time and computational burden. Further, there is considerably minor effect on kernel weights; even the weights are learned globally. This is due to the fact that the global space is comprised of all the existing modals, and thus, all the modals contribute in learning the global weights. For learning weights of kernels all the extracted features are used individually, and the dimensions of these features are also individually reduced to 450 by CCA before learning weights. In all our experiments the obtained weights for VIPeR are 0.3, 0.22, and 0.22 for RBF kernels, while weight for χ2 kernel is 0.26. For CUHK01 and CUHK03 the obtained weights for RBF kernels are 0.28, 0.24, and 0.24, while weight for χ2 kernel is 0.24. The σ values in all the datasets for the three RBF kernels are set to the mean value of modal k, as well as (mean value + mean/2) and (mean value-mean/2). These values for σ are chosen to model all the different variations in the modal k, while the σ value for χ2 kernel is also set to mean value of modal k. The mean value in our work is the similarity value between Probe and Gallery samples of center mk. Finally, the metric Mk is learned as(13)maxMk⁡trMkTSBMkMkTSWMk,where matrices SB and SW are obtained with similar method in [44]. Now, (13) is then solved using generalized eigenvalue problem [50] in (14) to obtain first r′= 300 eigenvectors corresponding to eigenvalues with largest magnitude as(14)SBφ=λSWφ.


## 2.7. Reidentification

From Figure2, reidentification between test pair (xia,xjb) is performed by first determining the transform modal the test pair belongs to usingK-NN classifier. InK-NN classifier, the parameterK is set to the number of modals in the image space; that is, in VIPeR the value ofK is set to the number of modals k=7. Then, the features of (xia,xjb) are projected into the weighted multikernel space of the respective modal, followed by the respective modal metric Mk to perform matching as(15)dxia,xjb=xia-xjbTMkxia-xjbT.


## 3. Experiments

Our IRM3 metric is evaluated on three benchmark datasets: VIPeR, CUHK01, and CUHK03. We follow the evaluation protocol of [33] for test/train split for VIPeR, CUHK01, and CUHK03 datasets. However, in our work we have tested CUHK01 for P=486 only, while CUHK03 is tested for bothLabelled andDetected settings. All the experiments are conducted insingle-shot mode, and all the reported Cumulative Matching Curves (CMC) are obtained by averaging the results over 20 trials.


### 3.1. Experiment Protocols

To thoroughly analyze the performance of IRM3 we have devised three evaluation strategies. These strategies evaluate IRM3 performance with different number of discovered modalsK in X, withGallery view impostors (GVI) (GVI are the impostors from Gallery view only and are obtained in similar way as in previous conventional metrics [14, 20–22]), as well asCross views impostors (CVI).(i)
IRM3 only: it is basic multimodal metric, learned with onlyNegative Gallery Samples (NGS).(ii)
IRM3 + GVI (p′): IRM3 is learned with impostors fromGallery view (GVI), as well as with NGS Here p′ refers to the number of impostors taken fromGallery view to form triplet samples and have values p′=5, 10, and 15, while the remaining triplets are formed using NGS(iii)
IRM3 + CVI (p′): IRM3 is learned with CVI, as well as with NGS Here p′ refers to number of CVI samples used to form triplets and have values p′=5, 10, and 15, while the remaining triplets are formed using NGSAll the samples from NGS, GVI, and CVI contain most difficult instances for a person and are randomly sampled off-line, before training metric. In all the three strategies above, we have partitioned image space intok=3, 5, and 7 for VIPeR, while, for CUHK01 we have used k=6, 7, and 10 partitions, and for CUHK03 k=13, 14, and 16 partitions are used, respectively.


### 3.2. Results on VIPeR

Comparison with State-of-the-Art Features.Results of IRM3 metric are compared with three state-of-the-art features LOMO [11], GoG [25], and momfLE [24] in Table 1. All the results in Table 1 are obtained for K=7 modals, and our IRM3 + CVI (p′ = 15) has attained rank@152.81% and has outperformed all the three features of reidentification, providing evidence that if the metric can address multimodal transform variations well as well as have strong resistance against impostors then the matching accuracy can be improved. Our learned IRM3 + CVI (p′ = 15) considers optimizing all the rank orders simultaneously and, thus, has large improvement at rank@5 and rank@10.Table 1
Top matching comparison on VIPeR.
Single-shot, P = 316 Method r = 1 r = 5 r = 10 F LOMO [11] 40.0 - 80.51 m o M f L E [24] 48.0 76.8 85.4 GoG [25] 49.7 79.7 88.7  DF SIR-CIR [26] 35.76 68.38 82.9  DMN GS-CNN [27] 37.8 66.9 77.4 DGD [28] 38.6 - - LSTM [29] 42.4 68.7 79.4 MuDeep [30] 43.03 74.36 85.76 E2E-CAN [31] 47.2 79.2 89.2 DLPA [32] 48.7 74.7 85.1 Quadruplet-net [33] 49.05 73.10 81.96 JLML [34] 50.2 74.2 84.3  M ITL [35] 15.2 34.2 45.9 LAFT [23] 29.6 - 69.3 WARCA [36] 37.47 70.78 - LISTEN [21] 39.62 69.97 82.9 L-1 graph [37] 41.5 - - SS-SVM [16] 42.66 70.1 84.27 SCSP [38] 53.54 82.59 91.49 IRM3 only 45.92 82.90 91.63 IRM3 + GVI (p′ = 15) 50.39 85.79 95.73 IRM3 + CVI (p′ = 15) 52.81 87.95 97.29Comparison with Metric Learning. We also compared metric IRM3 with 7 metrics. From Table 1 IRM3 + CVI (p′ = 15) has outperformed both multimodal metric LAFT [23] and impostor resistance metric LISTEN [21]. The prime difference between IRM3 and [21, 23] is its capability of addressing both the person modal transform, as well as capability of further maximizing the matching against joint constraint of cross views impostors. All these are the causes of great challenge in matching pedestrians. In Table 1 only SS-SVM [16] is a metric that tries to model the transform modal for each individual person; however, it never paid attention to acquire resistance against impostors and thus has19.21% lower rank@1 accuracy than IRM3 + CVI (p′= 15). Though IRM3 has successful results, still it has1.36% lower rank@1 than SCSP [38]. Obviously, VIPeR has large pose, misalignment, and body parts displacement issues which are specifically not addressed in our work and, thus, is necessarily needed to improve the matching and results largely.Comparison with Deep Methods. Though, deep features (DF) and deep matching networks (DMN) have no match with conventional metric learning methods, however, from the results in Table 1 it is clearly evident if two major issues of reidentification (i.e., multimodal transforms, and strong rejection capability against impostors) can be well handled simultaneously, then comparable or even higher performance than deep methods can be attained. Our IRM3 + CVI (p′ = 15) has7.1% and4.94% higher rank@1 than Quadruplet-Net [33] and JLML [34], respectively. These obtained results demonstrate the fact that for smaller dataset like VIPeR deep matching networks have insufficient training samples to learn a discriminative network.At last, Figure3 shows the comparison of retrieval results of two queries from VIPeR dataset for XQDA [11] and our IRM3 + CVI (p′=15) when K=7 modals are used. Retrieval results ofQuery 1 for XQDA find the correct match atrank =4 enclosed in green rectangle (b), while IMR3 finds the match atrank =2 enclosed in green rectangle (e). Similarly, forQuery 2 our IMR3 finds the match atrank =1 enclosed in green rectangle (j); in contrast, XQDA finds the correct match atrank =3 enclosed in green rectangle (h). Thus, our IRM3 approach improves matching, and consequently rank gets higher.Figure 3
Two queries are shown,Query 1 andQuery 2, and their retrieval results using XQDA [11] and our IRM3. Correct match is shown in green rectangle, while blue rectangle shows impostors.


### 3.3. Results on CUHK01

Comparison with State-of-the-Art Features. Table 2 summarizes results of IRM3 for K = 10 modals and compares the obtained results with LOMO [11], GoG [25], and momfLE [24]. Though the three features are discriminative, however, our IRM3 approach is better than the three features in solving the two big challenges of Re-ID, that is, multimodal pedestrians matching and impostors resistance. Since CUHK01 has larger training set than VIPeR, thus, modal transforms can be well learned, and therefore, IRM3 + CVI (p′ = 15) attains larger discrimination than momfLE [24]. Our IRM3 + CVI (p′ = 15) has15.15% higher rank@1 accuracy than momfLE due to inherent virtue of handling different modals, person specific variations, and rejecting large number of impostors, all simultaneously.Table 2
Top matching comparison on CUHK01.
Single-shot, P = 486 Method r = 1 r = 5 r = 10 F LOMO [11] 49.2 75.7 84.2 m o M f L E [24] 64.6 84.9 90.6 GoG [25] 57.8 79.1 86.2  DMN MCP-CNN [39] 53.7 84.3 91.0 DGD [28] 66.6 - - E2E-CAN [31] 67.2 87.3 92.5 DLPA [32] 75.0 93.5 95.7 Quadruplet-net [33] 62.55 83.44 89.71 JLML [34] 69.8 88.4 93.3  M CVAML [40] 57.3 81.2 86.5 WARCA [36] 58.34 79.76 - L-1 graph [37] 50.1 - - IRM3 only 68.24 88.71 94.31 IRM3 + GVI (p′ = 15) 72.91 94.61 97.6 IRM3 + CVI (p′ = 15) 76.14 96.90 98.35Comparison with Metric Learning. In Table 2 three most recently proposed metrics CVAML [40], WARCA [36], and L-1 Graph [37] are compared with our IRM3 approach. All the three metrics have assumption of unimodal intercamera transform, rather than multimodal image space. Though WARCA [36] employed hard negative samples as learning constraint, however, ignoring other negative samples fromGallery viewand not taking into consideration a person modal during learning have made it suffer greatly to attain higher accuracy. On the other hand, IRM3 + CVI (p′=15) has capability to deal all these challenges and, thus, has attained76.14% rank@1 accuracy.Comparison with Deep Methods. In Table 2, we can see several deep matching networks (DMN) have performed much well than conventional metrics on CUHK01. Only K-LFDA when trained with momfLE [24] feature attains comparable performance than DMN. However, motivated to resolve the challenges for reidentification in real world (i.e., multimodal image space, and diverse impostors) IRM3 + CVI (p′=15) has much better results than MCP-CNN [39], E2E-CAN [31], Quadruplet-Net [33], and JLML [34], while our IRM3 + CVI (p′ = 15) has1.49% higher rank@1 than DLPA [32]. DLPA extracts deep features by semantically aligning body parts, as well as rectifying pose variations. We believe if sematic body parts alignment and rectification of poses variations are included in our IRM3 then the results can be further improved.


### 3.4. Results on CUHK03

Comparison with State-of-the-Art Features. Table 3 compares LOMO [11] and GoG [25] features with our IRM3 metric in bothLabelled andDetected settings. All the results in Table 3 are obtained for K = 16 modals. In Table 3, obtained results are much higher than the two features. The primary reason of gain in performance for IRM3 against the features [11, 25] is mainly due to the difference in their approaches. In [11, 25] a universal feature representation is proposed for all the different persons, which may not be optimal for all the persons at the same time residing on different modals; in contrast, our motivation is based on discovering distinct modals in the image space and then addressing each modal specifically with empowerment of large number of impostors rejection. Therefore, our IRM3 + CVI (p′ = 15) (in Labelled setting) has rank@1 accuracy of about86.17%.Table 3
Top matching comparison on CUHK03.
Labelled, P = 100 Method r = 1 r = 5 r = 10 F LOMO [11] 52.2 82.23 92.14 GoG [25] 67.3 91.0 96.0  DF DCAF [41] 74.21 94.33 97.54  DFM DGD [28] 75.3 - - Quadruplet-net [33] 75.53 95.15 99.16 MuDeep [30] 76.87 96.12 98.41 E2E-CAN [31] 77.6 95.2 99.3 JLML [34] 83.2 98.0 99.4 DLPA [32] 85.4 97.6 99.4  M SS-SVM [16] 57.0 85.7 94.3 Null Sp. [42] 62.55 90.05 94.80 SSM [43] 76.6 94.6 98.0 WARCA [36] 78.38 94.55 - IRM3 only 78.83 95.97 98.37 IRM3 + GVI (p′ = 15) 83.32 98.70 99.54 IRM3 + CVI (p′ = 15) 86.17 99.02 99.68  Detected, P = 100 Method r = 1 r = 5 r = 10  F LOMO [11] 46.25 78.9 88.55 GoG [25] 65.5 88.4 93.7  DF SIR-CIR [26] 52.17 83.7 90.4 DCAF [41] 67.99 91.04 95.36  DFM LSTM [29] 57.3 80.10 88.3 GS-CNN [27] 68.1 88.1 94.6 E2E-CAN [31] 69.2 88.5 94.1 MuDeep [30] 75.64 94.36 97.46 JLML [34] 80.6 96.9 98.7 DLPA [32] 81.6 97.3 98.4  M L-1 graph [37] 39.0 - - SS-SVM [16] 51.2 80.8 89.6 Null Sp. [42] 54.70 84.75 94.80 SSM [43] 72.7 92.4 96.1 IRM3 only 72.98 91.7 93.02 IRM3 + GVI (p′ = 15) 78.68 95.60 98.09 IRM3 + CVI (p′ = 15) 80.77 96.94 98.67Comparison with Metric Learning. In Table 3, recently proposed WARCA [36] and SSM [43] are compared with our IRM3 approach. WARCA [36] differs with our IRM3 approach in a way that it only addresses hard negative samples, while SSM [43] differs in a way that it has no measure to account for different modal transforms, as well as having no resistance against impostors. Our IRM3 + CVI (p′ = 15) (in Labelled setting) has surpassed [36] and [43] and has attained9.04% and11.1% rise at rank@1 accuracy, respectively.Comparison with Deep Methods. Interestingly, in Table 3 all the deep methods inLabelled andDetected settings have very high performance on CUHK03. These high results demonstrate the fact that CUHK03 is the largest dataset among all and, thus, can help in learning a more discriminative DMN. Even though both JLML [34] and DLPA [32] learn deep body features with global and local body parts alignment, as well as, pose alignment, however, our IMR3 approach benefitted with transform specific metrics empowered with impostors rejection still maintained to attain better results. Our IRM3 considers optimizing all the rank orders simultaneously and, thus, have large gain at rank@5 and rank@10 inLabelled setting.


### 3.5. Analysis

In Table4, we analyzed the effect of number of modals K in testing for VIPeR. Initially, we have partitioned image space into K = 5 and then tested it without using any impostor sample (K = 5, p′ = 0) to obtain rank@1 results of about 45.27%. As the more modals are discovered in the image space, such as K=7, then the results get further improved even without using any impostor sample (K=7, p′ = 0), and rank@1 becomes 45.92%. The main reason behind this increment is the fact that now we could match more test samples correctly by using their actual modal transforms which were lost when the modals are less discovered in K=5.Table 4
Effect of multimodal transforms + impostor resistance (VIPeR,P = 316).
Method r = 1 r = 5 r = 10 k = 5, p′=0 45.27 82.16 91.1 k = 7, p′=0 45.92 82.90 91.63 k = 5, GVI (p′=15) 48.88 85.33 95.37 k = 7, GVI (p′=15) 50.39 85.79 95.73 k = 5, CVI (p′=15) 52.10 87.14 96.87 k = 7, CVI (p′=15) 52.81 87.95 97.29In addition, we could also see a positive increment in results when impostors fromGallery view are also added in learning metric. Both (K = 5, GVI (p′ = 15)) and (K = 7, GVI (p′ = 15)) have attained more higher differentiating capability than [14, 20–22], as now, they can restrict impostors by taking into care transform modals a positive pair and impostors undergo.Interestingly, in our work this impostor resistance can be further enhanced. This is done by usingCross views impostors (CVI). From Table 4, it is clear that even for same number of modals say K=7, when (CVI) are used then the differentiation capability of (K = 7, CVI (p′ = 15)) gets further enhanced than (K = 7, GVI (p′ = 15)) and rank@1 becomes 52.81%. This increment in rank@1 provides a strong evidence that CVI have ability to maximize the similarity of positive pair more than GVI by taking into care both the transform modal, as well as various different changes a given query and Gallery samples undergo in different views.At last, in Figure4 we have provided a performance comparison at rank@1 when the modal centers are chosen randomly, as well as when the centers are obtained using our method in Section 2.2. Obtained rank@1 accuracy for random centers is poor, because these random centers are obtained just by simply choosing the top-K persons without taking into care their reliability, stability, and IDs.Figure 4
Performance at rank@1 when centersmk are selected randomly, and when centers are selected with our approach provided in Section 2.2.


### 3.6. Efficiency

We computed the run time of our IRM3 approach using MK-LFDA [44], XQDA [11], and K-LFDA [45] (with χ2 kernel) on CUHK03. There are 1260 training persons and 100 testing identities. All the algorithms are implemented in MATLAB and run on server machine having 6 CPUs (Xeon(r)e5-2620) with each CPU having 6 cores and total memory size of 256 GB. In Table 5, training time of MK-LFDA [44] is faster than XQDA [11] but lower than K-LFDA [45]. However, in testing when the weights of kernels are not learned MK-LFDA [44] is faster than both XQDA and K-LFDA. These timing results support the fact that our proposed method is well applicable in real time applications and in public spaces.Table 5
Run time comparison on CUHK03 (in seconds).
Method MK-LFDA [44] XQDA [11] K-LFDA [45] Training 171.26 174.7 160.03 Testing 28.8 30.23 32.45


## 3.1. Experiment Protocols

To thoroughly analyze the performance of IRM3 we have devised three evaluation strategies. These strategies evaluate IRM3 performance with different number of discovered modalsK in X, withGallery view impostors (GVI) (GVI are the impostors from Gallery view only and are obtained in similar way as in previous conventional metrics [14, 20–22]), as well asCross views impostors (CVI).(i)
IRM3 only: it is basic multimodal metric, learned with onlyNegative Gallery Samples (NGS).(ii)
IRM3 + GVI (p′): IRM3 is learned with impostors fromGallery view (GVI), as well as with NGS Here p′ refers to the number of impostors taken fromGallery view to form triplet samples and have values p′=5, 10, and 15, while the remaining triplets are formed using NGS(iii)
IRM3 + CVI (p′): IRM3 is learned with CVI, as well as with NGS Here p′ refers to number of CVI samples used to form triplets and have values p′=5, 10, and 15, while the remaining triplets are formed using NGSAll the samples from NGS, GVI, and CVI contain most difficult instances for a person and are randomly sampled off-line, before training metric. In all the three strategies above, we have partitioned image space intok=3, 5, and 7 for VIPeR, while, for CUHK01 we have used k=6, 7, and 10 partitions, and for CUHK03 k=13, 14, and 16 partitions are used, respectively.


## 3.2. Results on VIPeR

Comparison with State-of-the-Art Features.Results of IRM3 metric are compared with three state-of-the-art features LOMO [11], GoG [25], and momfLE [24] in Table 1. All the results in Table 1 are obtained for K=7 modals, and our IRM3 + CVI (p′ = 15) has attained rank@152.81% and has outperformed all the three features of reidentification, providing evidence that if the metric can address multimodal transform variations well as well as have strong resistance against impostors then the matching accuracy can be improved. Our learned IRM3 + CVI (p′ = 15) considers optimizing all the rank orders simultaneously and, thus, has large improvement at rank@5 and rank@10.Table 1
Top matching comparison on VIPeR.
Single-shot, P = 316 Method r = 1 r = 5 r = 10 F LOMO [11] 40.0 - 80.51 m o M f L E [24] 48.0 76.8 85.4 GoG [25] 49.7 79.7 88.7  DF SIR-CIR [26] 35.76 68.38 82.9  DMN GS-CNN [27] 37.8 66.9 77.4 DGD [28] 38.6 - - LSTM [29] 42.4 68.7 79.4 MuDeep [30] 43.03 74.36 85.76 E2E-CAN [31] 47.2 79.2 89.2 DLPA [32] 48.7 74.7 85.1 Quadruplet-net [33] 49.05 73.10 81.96 JLML [34] 50.2 74.2 84.3  M ITL [35] 15.2 34.2 45.9 LAFT [23] 29.6 - 69.3 WARCA [36] 37.47 70.78 - LISTEN [21] 39.62 69.97 82.9 L-1 graph [37] 41.5 - - SS-SVM [16] 42.66 70.1 84.27 SCSP [38] 53.54 82.59 91.49 IRM3 only 45.92 82.90 91.63 IRM3 + GVI (p′ = 15) 50.39 85.79 95.73 IRM3 + CVI (p′ = 15) 52.81 87.95 97.29Comparison with Metric Learning. We also compared metric IRM3 with 7 metrics. From Table 1 IRM3 + CVI (p′ = 15) has outperformed both multimodal metric LAFT [23] and impostor resistance metric LISTEN [21]. The prime difference between IRM3 and [21, 23] is its capability of addressing both the person modal transform, as well as capability of further maximizing the matching against joint constraint of cross views impostors. All these are the causes of great challenge in matching pedestrians. In Table 1 only SS-SVM [16] is a metric that tries to model the transform modal for each individual person; however, it never paid attention to acquire resistance against impostors and thus has19.21% lower rank@1 accuracy than IRM3 + CVI (p′= 15). Though IRM3 has successful results, still it has1.36% lower rank@1 than SCSP [38]. Obviously, VIPeR has large pose, misalignment, and body parts displacement issues which are specifically not addressed in our work and, thus, is necessarily needed to improve the matching and results largely.Comparison with Deep Methods. Though, deep features (DF) and deep matching networks (DMN) have no match with conventional metric learning methods, however, from the results in Table 1 it is clearly evident if two major issues of reidentification (i.e., multimodal transforms, and strong rejection capability against impostors) can be well handled simultaneously, then comparable or even higher performance than deep methods can be attained. Our IRM3 + CVI (p′ = 15) has7.1% and4.94% higher rank@1 than Quadruplet-Net [33] and JLML [34], respectively. These obtained results demonstrate the fact that for smaller dataset like VIPeR deep matching networks have insufficient training samples to learn a discriminative network.At last, Figure3 shows the comparison of retrieval results of two queries from VIPeR dataset for XQDA [11] and our IRM3 + CVI (p′=15) when K=7 modals are used. Retrieval results ofQuery 1 for XQDA find the correct match atrank =4 enclosed in green rectangle (b), while IMR3 finds the match atrank =2 enclosed in green rectangle (e). Similarly, forQuery 2 our IMR3 finds the match atrank =1 enclosed in green rectangle (j); in contrast, XQDA finds the correct match atrank =3 enclosed in green rectangle (h). Thus, our IRM3 approach improves matching, and consequently rank gets higher.Figure 3
Two queries are shown,Query 1 andQuery 2, and their retrieval results using XQDA [11] and our IRM3. Correct match is shown in green rectangle, while blue rectangle shows impostors.


## 3.3. Results on CUHK01

Comparison with State-of-the-Art Features. Table 2 summarizes results of IRM3 for K = 10 modals and compares the obtained results with LOMO [11], GoG [25], and momfLE [24]. Though the three features are discriminative, however, our IRM3 approach is better than the three features in solving the two big challenges of Re-ID, that is, multimodal pedestrians matching and impostors resistance. Since CUHK01 has larger training set than VIPeR, thus, modal transforms can be well learned, and therefore, IRM3 + CVI (p′ = 15) attains larger discrimination than momfLE [24]. Our IRM3 + CVI (p′ = 15) has15.15% higher rank@1 accuracy than momfLE due to inherent virtue of handling different modals, person specific variations, and rejecting large number of impostors, all simultaneously.Table 2
Top matching comparison on CUHK01.
Single-shot, P = 486 Method r = 1 r = 5 r = 10 F LOMO [11] 49.2 75.7 84.2 m o M f L E [24] 64.6 84.9 90.6 GoG [25] 57.8 79.1 86.2  DMN MCP-CNN [39] 53.7 84.3 91.0 DGD [28] 66.6 - - E2E-CAN [31] 67.2 87.3 92.5 DLPA [32] 75.0 93.5 95.7 Quadruplet-net [33] 62.55 83.44 89.71 JLML [34] 69.8 88.4 93.3  M CVAML [40] 57.3 81.2 86.5 WARCA [36] 58.34 79.76 - L-1 graph [37] 50.1 - - IRM3 only 68.24 88.71 94.31 IRM3 + GVI (p′ = 15) 72.91 94.61 97.6 IRM3 + CVI (p′ = 15) 76.14 96.90 98.35Comparison with Metric Learning. In Table 2 three most recently proposed metrics CVAML [40], WARCA [36], and L-1 Graph [37] are compared with our IRM3 approach. All the three metrics have assumption of unimodal intercamera transform, rather than multimodal image space. Though WARCA [36] employed hard negative samples as learning constraint, however, ignoring other negative samples fromGallery viewand not taking into consideration a person modal during learning have made it suffer greatly to attain higher accuracy. On the other hand, IRM3 + CVI (p′=15) has capability to deal all these challenges and, thus, has attained76.14% rank@1 accuracy.Comparison with Deep Methods. In Table 2, we can see several deep matching networks (DMN) have performed much well than conventional metrics on CUHK01. Only K-LFDA when trained with momfLE [24] feature attains comparable performance than DMN. However, motivated to resolve the challenges for reidentification in real world (i.e., multimodal image space, and diverse impostors) IRM3 + CVI (p′=15) has much better results than MCP-CNN [39], E2E-CAN [31], Quadruplet-Net [33], and JLML [34], while our IRM3 + CVI (p′ = 15) has1.49% higher rank@1 than DLPA [32]. DLPA extracts deep features by semantically aligning body parts, as well as rectifying pose variations. We believe if sematic body parts alignment and rectification of poses variations are included in our IRM3 then the results can be further improved.


## 3.4. Results on CUHK03

Comparison with State-of-the-Art Features. Table 3 compares LOMO [11] and GoG [25] features with our IRM3 metric in bothLabelled andDetected settings. All the results in Table 3 are obtained for K = 16 modals. In Table 3, obtained results are much higher than the two features. The primary reason of gain in performance for IRM3 against the features [11, 25] is mainly due to the difference in their approaches. In [11, 25] a universal feature representation is proposed for all the different persons, which may not be optimal for all the persons at the same time residing on different modals; in contrast, our motivation is based on discovering distinct modals in the image space and then addressing each modal specifically with empowerment of large number of impostors rejection. Therefore, our IRM3 + CVI (p′ = 15) (in Labelled setting) has rank@1 accuracy of about86.17%.Table 3
Top matching comparison on CUHK03.
Labelled, P = 100 Method r = 1 r = 5 r = 10 F LOMO [11] 52.2 82.23 92.14 GoG [25] 67.3 91.0 96.0  DF DCAF [41] 74.21 94.33 97.54  DFM DGD [28] 75.3 - - Quadruplet-net [33] 75.53 95.15 99.16 MuDeep [30] 76.87 96.12 98.41 E2E-CAN [31] 77.6 95.2 99.3 JLML [34] 83.2 98.0 99.4 DLPA [32] 85.4 97.6 99.4  M SS-SVM [16] 57.0 85.7 94.3 Null Sp. [42] 62.55 90.05 94.80 SSM [43] 76.6 94.6 98.0 WARCA [36] 78.38 94.55 - IRM3 only 78.83 95.97 98.37 IRM3 + GVI (p′ = 15) 83.32 98.70 99.54 IRM3 + CVI (p′ = 15) 86.17 99.02 99.68  Detected, P = 100 Method r = 1 r = 5 r = 10  F LOMO [11] 46.25 78.9 88.55 GoG [25] 65.5 88.4 93.7  DF SIR-CIR [26] 52.17 83.7 90.4 DCAF [41] 67.99 91.04 95.36  DFM LSTM [29] 57.3 80.10 88.3 GS-CNN [27] 68.1 88.1 94.6 E2E-CAN [31] 69.2 88.5 94.1 MuDeep [30] 75.64 94.36 97.46 JLML [34] 80.6 96.9 98.7 DLPA [32] 81.6 97.3 98.4  M L-1 graph [37] 39.0 - - SS-SVM [16] 51.2 80.8 89.6 Null Sp. [42] 54.70 84.75 94.80 SSM [43] 72.7 92.4 96.1 IRM3 only 72.98 91.7 93.02 IRM3 + GVI (p′ = 15) 78.68 95.60 98.09 IRM3 + CVI (p′ = 15) 80.77 96.94 98.67Comparison with Metric Learning. In Table 3, recently proposed WARCA [36] and SSM [43] are compared with our IRM3 approach. WARCA [36] differs with our IRM3 approach in a way that it only addresses hard negative samples, while SSM [43] differs in a way that it has no measure to account for different modal transforms, as well as having no resistance against impostors. Our IRM3 + CVI (p′ = 15) (in Labelled setting) has surpassed [36] and [43] and has attained9.04% and11.1% rise at rank@1 accuracy, respectively.Comparison with Deep Methods. Interestingly, in Table 3 all the deep methods inLabelled andDetected settings have very high performance on CUHK03. These high results demonstrate the fact that CUHK03 is the largest dataset among all and, thus, can help in learning a more discriminative DMN. Even though both JLML [34] and DLPA [32] learn deep body features with global and local body parts alignment, as well as, pose alignment, however, our IMR3 approach benefitted with transform specific metrics empowered with impostors rejection still maintained to attain better results. Our IRM3 considers optimizing all the rank orders simultaneously and, thus, have large gain at rank@5 and rank@10 inLabelled setting.


## 3.5. Analysis

In Table4, we analyzed the effect of number of modals K in testing for VIPeR. Initially, we have partitioned image space into K = 5 and then tested it without using any impostor sample (K = 5, p′ = 0) to obtain rank@1 results of about 45.27%. As the more modals are discovered in the image space, such as K=7, then the results get further improved even without using any impostor sample (K=7, p′ = 0), and rank@1 becomes 45.92%. The main reason behind this increment is the fact that now we could match more test samples correctly by using their actual modal transforms which were lost when the modals are less discovered in K=5.Table 4
Effect of multimodal transforms + impostor resistance (VIPeR,P = 316).
Method r = 1 r = 5 r = 10 k = 5, p′=0 45.27 82.16 91.1 k = 7, p′=0 45.92 82.90 91.63 k = 5, GVI (p′=15) 48.88 85.33 95.37 k = 7, GVI (p′=15) 50.39 85.79 95.73 k = 5, CVI (p′=15) 52.10 87.14 96.87 k = 7, CVI (p′=15) 52.81 87.95 97.29In addition, we could also see a positive increment in results when impostors fromGallery view are also added in learning metric. Both (K = 5, GVI (p′ = 15)) and (K = 7, GVI (p′ = 15)) have attained more higher differentiating capability than [14, 20–22], as now, they can restrict impostors by taking into care transform modals a positive pair and impostors undergo.Interestingly, in our work this impostor resistance can be further enhanced. This is done by usingCross views impostors (CVI). From Table 4, it is clear that even for same number of modals say K=7, when (CVI) are used then the differentiation capability of (K = 7, CVI (p′ = 15)) gets further enhanced than (K = 7, GVI (p′ = 15)) and rank@1 becomes 52.81%. This increment in rank@1 provides a strong evidence that CVI have ability to maximize the similarity of positive pair more than GVI by taking into care both the transform modal, as well as various different changes a given query and Gallery samples undergo in different views.At last, in Figure4 we have provided a performance comparison at rank@1 when the modal centers are chosen randomly, as well as when the centers are obtained using our method in Section 2.2. Obtained rank@1 accuracy for random centers is poor, because these random centers are obtained just by simply choosing the top-K persons without taking into care their reliability, stability, and IDs.Figure 4
Performance at rank@1 when centersmk are selected randomly, and when centers are selected with our approach provided in Section 2.2.


## 3.6. Efficiency

We computed the run time of our IRM3 approach using MK-LFDA [44], XQDA [11], and K-LFDA [45] (with χ2 kernel) on CUHK03. There are 1260 training persons and 100 testing identities. All the algorithms are implemented in MATLAB and run on server machine having 6 CPUs (Xeon(r)e5-2620) with each CPU having 6 cores and total memory size of 256 GB. In Table 5, training time of MK-LFDA [44] is faster than XQDA [11] but lower than K-LFDA [45]. However, in testing when the weights of kernels are not learned MK-LFDA [44] is faster than both XQDA and K-LFDA. These timing results support the fact that our proposed method is well applicable in real time applications and in public spaces.Table 5
Run time comparison on CUHK03 (in seconds).
Method MK-LFDA [44] XQDA [11] K-LFDA [45] Training 171.26 174.7 160.03 Testing 28.8 30.23 32.45


## 4. Conclusion

This paper presents a metric learning approach that exploits both multimodal transforms and Cross views impostors to improve the capability of metric to differentiate among different persons, as well as enhance rejection capability to decline large number of real world diverse impostors. In real world mostly pedestrian images are multimodal, and in public spaces several persons share similar clothing; therefore, our IRM3 is learned to tackle such issues of reidentification and person tracking in public spaces. Extensive experiments on three challenging datasets (VIPeR, CUHK01, and CUHK03) demonstrate the effectiveness of our IRM3 metric which has outperformed many previous state-of-the-art metrics. In addition, we further intend to extend our approach for testing in real world scenario and intend to solve various other issues for real time implementation.



---

*Source: 3202495-2018-05-03.xml*