Appearance model update based on online learning and soft‐biometrics traits for people re‐identification in multi‐camera environments

IET Image ProcessingVolume 13, Issue 12 p. 2162-2168 Research ArticleFree Access Appearance model update based on online learning and soft-biometrics traits for people re-identification in multi-camera environments Daniela Moctezuma, Corresponding Author dmoctezuma@centrogeo.edu.mx orcid.org/0000-0003-3038-4642 Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de GeoInteligencia - Centro de Investigación en Ciencias de Información Geoespacial, Circuito Tecnopolo Norte No. 117, Col. Tecnopolo Pocitos II, C.P., Aguascalientes, Ags, 20313 MexicoSearch for more papers by this authorEric S. Tellez, orcid.org/0000-0001-5804-9868 Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de Analitica Computacional - Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación, Circuito Tecnopolo Norte 117, Col. Tecnopolo Pocitos II, C.P., 20313 Aguascalientes, Ags, MexicoSearch for more papers by this authorSabino Miranda-Jiménez, Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de Analitica Computacional - Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación, Circuito Tecnopolo Norte 117, Col. Tecnopolo Pocitos II, C.P., 20313 Aguascalientes, Ags, MexicoSearch for more papers by this authorMario Graff, Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de Analitica Computacional - Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación, Circuito Tecnopolo Norte 117, Col. Tecnopolo Pocitos II, C.P., 20313 Aguascalientes, Ags, MexicoSearch for more papers by this author Daniela Moctezuma, Corresponding Author dmoctezuma@centrogeo.edu.mx orcid.org/0000-0003-3038-4642 Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de GeoInteligencia - Centro de Investigación en Ciencias de Información Geoespacial, Circuito Tecnopolo Norte No. 117, Col. Tecnopolo Pocitos II, C.P., Aguascalientes, Ags, 20313 MexicoSearch for more papers by this authorEric S. Tellez, orcid.org/0000-0001-5804-9868 Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de Analitica Computacional - Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación, Circuito Tecnopolo Norte 117, Col. Tecnopolo Pocitos II, C.P., 20313 Aguascalientes, Ags, MexicoSearch for more papers by this authorSabino Miranda-Jiménez, Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de Analitica Computacional - Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación, Circuito Tecnopolo Norte 117, Col. Tecnopolo Pocitos II, C.P., 20313 Aguascalientes, Ags, MexicoSearch for more papers by this authorMario Graff, Cátedras - CONACyT Consejo Nacional de Ciencia y Tecnología, Av. Insurgentes Sur 1582, Col. Crédito Constructor, Del. Benito Juárez, C.P., 03940 Ciudad de México, Mexico Laboratorio de Analitica Computacional - Centro de Investigación e Innovación en Tecnologías de la Información y Comunicación, Circuito Tecnopolo Norte 117, Col. Tecnopolo Pocitos II, C.P., 20313 Aguascalientes, Ags, MexicoSearch for more papers by this author First published: 17 September 2019 https://doi.org/10.1049/iet-ipr.2019.0083Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Intelligent surveillance systems in multi-camera environments pose a hard-open problem for computer vision. The way the people look changes inside and also among cameras, so people re-identification task can be largely improved collecting data about people already identified and take advantage of it as time advances in surveillance video. Furthermore, a camera change or a slight change in the objective traits may require the complete re-formulation of the appearance models. In this paper, we propose several heuristics for updating the appearance model in a multi-camera surveillance environment. Through these heuristics, the subject's appearance model is updated across different time and environmental conditions. The update process is carried out primarily in three different aspects: 1) based on time lapses, 2) based on the change of camera, and 3) based on the automatic selection of the most representative samples selected through decision functions of the classifier. The proposed system focuses on video surveillance environments, that is, the objective is to identify an individual across the set of cameras in the surveillance area, the comparison considers only those people that share time and space. We used four public benchmarks to test our claims; the results confirm the importance of continuous appearance model's updating. 1 Introduction An intelligent video surveillance system attempts to understand and describe behaviours in some environment. The aim behind this topic is to provide computer vision algorithms that can assist traditional video surveillance operators, especially in environments with a large number of cameras. Human attention is surpassed as the number of cameras, and the number of objectives, increases. Whenever the number of cameras is large, the amount of images produced is difficult to handle. Moreover, the computational resources needed for some tasks such as people re-identification and detection, event recognition, tracking, or trajectory analysis, pose additional tremendous challenges to these surveillance tasks. People re-identification tackles the problem of recognising a target (a person) across several cameras in a surveillance environment. For this task, the techniques focus on the representation of people using both biometric and soft-biometric traits [1]. The representativeness of samples is critical in the people re-identification task; i.e. it is essential to model people's appearance adequately, according to the environment's conditions and time. Under these circumstances, online and incremental learning is helpful. The related literature on incremental learning is vast, see for instance [1-3], where it is shown that the model's representativeness of any data is critical in any pattern-recognition problem. In the case of people re-identification, the human appearance model obtained in one camera is usually different from those from other cameras, because of variations in view angle, illumination, body poses, clothing, background clutter, and occlusion. As a consequence, the performance of the identification system is degraded when original appearance features are inadequate or non-representative for the substantial intra-class variations of the input samples [4]. The kinds of features to model each people appearance are also essential. Many people re-identification systems are based on biometrics traits; nevertheless, in surveillance environments, these features are useless due to the low quality and the distance acquisition conditions [5]; in contrast, the soft-biometrics features are better suited to these circumstances. However, they are designed to provide clues about the individuals with lack of distinctiveness and permanence [6]; among these characteristics, we found traits related to clothing, aspects of the human body, or gender. The use of soft-biometric traits can improve the performance of the system. In this paper, we propose several methods to update people's appearance model. Our models use a set of soft-biometrics features; this model can change with time using several strategies to provide online learning capabilities. Our learning machinery is based on popular machine-learning techniques such as linear support vector machine (SVM) classifier with stochastic gradient descent (GD) learning. To evaluate the performance of the proposed approaches, four public data sets were considered: PETS 2006, PETS 2009, CAVIAR4REID, and SAIVT-SoftBio. The experimental analysis shows how competitive our online learning strategies are under different conditions and scenarios. Furthermore, we present and discuss a comprehensive comparison of our work with nine-related state-of-the-art alternatives. The rest of this paper is organised as follows. Section 2 reviews the related work and Section 3 describes our benchmarks used in our experimental section. Section 4 is dedicated to describe our approaches for model updating based on online learning. The experimental validation of our approach is made in Section 5; finally, Section 6 is devoted to summarise and conclude our findings. 2 Related work In the past few years, several studies on person re-identification based on appearance models have been published; some of them use online learning approaches to perform the recognition step. For instance, Lu et al. [1] propose an online human recognition system with an incremental classifier based on SVM, which updates only the required aspects. The authors extract features related to the colour and the texture of three different human body parts (i.e. head, torso, and legs) to represent each frame. The CASIA Gait data set containing 20 different people are used in its experimental validation. In [7], a temporal model adaptation scheme is proposed through a similarity learning method. The similarity is learnt through a stochastic derivation of alternative directions methods for multipliers [8]. The method is validated experimentally with VIPeR, PRID450S, and Market1501 benchmarks. An appearance-based person re-identification approach based on a human detection algorithm with a discriminatively trained human-part model is introduced in [9]. The idea is to identify, separately, the human front, side, and back; also, it computes the proportion of the head and body as a trait. So, different features are extracted from human body parts, and these features are compared with a predefined template to find out the similarity value among the body parts and the template. Another work is presented in [10], where an image pool for each person's identity is created. This image pool is created by a set of images produced by a tracking method; the method considers the diversity and representativeness of each image according to the angle and lighting variations. Furthermore, several rules to update this image pool are proposed; these rules are based on the diversity factor to generate a ranking of them. Deep learning is also presented in people re-identification task such as in the case of [11]. Here, a deep-ranking model with feature learning is proposed. In particular, the authors create a convolutional neural network model to generate a set of local and global features; the model also performs the fusion of these features. The experiments were done using several data sets and, in some cases, the results show a slight improvement in comparison with some of the related works. Fang et al. [12] propose to combine three different types of features to tackle the problem. The kinds of features include the combination of two space colour models, red, green, and blue (RGB) and hue, saturation, and value (HSV), and second, the first- and second-order magnitude gradients are calculated using the first ones. The authors also use histograms, mean, and co-occurrence matrix. The concatenation of all these characteristics defines the multi-statistics on hash feature map descriptor. A method, which employs several descriptors such as local maximal occurrence (LOMO), Weighted Histogram of Overlapping Stripes (WHOS), and Gaussian Of Gaussian (GOG), are presented in [13]. These characteristics are used to fit a learning model. The authors use VIPeR and CAVIAR4REID for benchmarking purposes. In [14], an efficient algorithm (called early active learning) is proposed for selecting a subset of samples considered as more informative for training. For sample representation, the LOMO feature was used. Four public data sets (VIPeR, PRID, i-LID, and CAVIAR4REID) were considered for experimental validation. Another online update scheme for the people re-identification task is proposed in [15]. Here, the evaluation protocol is handled more realistically; that is, people are compared only with those sharing camera and time. The algorithm keeps it simple and straightforward. Metric Keep it Simple and Straightforward Metric (KISSME) is used as the metric learning method. The UCR, NLPR, RAiD, and SAIVT data sets were used for the validation. To solve the variance issue in images from multiple cameras, an energy-based loss function is presented in [16] to tackle the people re-identification. This function takes into account two aspects: it favours short distances, which indicate a high level of similarity between instances of people's appearances, and penalise large distances which reflect low similarity, as well as the number of overlaps between the local neighbours of two compared people and local neighbours of other people. The evaluation was done with three public databases: ETHZ, the CAVIAR4REID, and Person Re-ID 2011. In [17], a method to analyse a set of different kinds of features is introduced. The proposed features introduce semantic information, and these features and a novel discriminative model are proposed to learn the attributes correlation. The experimental comparison includes four standard benchmarks, i.e. PKU-REID, SAIVT-SoftBio, iLIDS-VID, and PRID. With the purpose of transfer, the appearance variations in [18] proposed a model using cumulative weighted brightness transfer function employing a set of images instead of image-per-image transfer. For the experiments, the VIPeR, CAVIAR4REID, PRID2011, and SAIVT-SoftBio data sets were considered. A method, which exploits the body's visual clues, is presented in [19]. This descriptor supposes that the person is in an upright pose to capture the chromatic content, the spatial arrangement of colour in the region and the presence of recurrent local motifs with high entropy in the regions. García et al. [20] consider the pose and the orientation of the camera to obtain people's representation. The proposed method extracts multiple frames of the same person with different orientations. Furthermore, it learns a pairwise feature dissimilarity function in several sub-spaces; from each sub-space, a classifier is trained according to all the variations of pose and orientations corresponding to each camera view. Several researchers tackle people's re-identification problem with metrics learning methods. The metric learning is conducted once per individual, but it is not updated according to time or space, see, for instance, [11, 21, 22]. The interested reader on the traditional people's re-identification problem is referred to [23-26], which survey the problem. The work presented in this paper combines the model appearance with an online learning approach to outperform the identification rate in the people re-identification problem. The following sections detail our contribution. 3 Databases The research community of the intelligent video surveillance area considers the experimental evaluation as a critical step. Therefore, we found a wide variety of benchmarking data sets. In this paper, we select four standard data sets in the area: CAVIAR4REID, PETS 2006, PETS 2009, and SAIVT-SoftBio. The CAVIAR4REID [27] data set contemplates a real video surveillance environment with complex images with low resolution, light changes, occlusions, and variations in pose and angle. CAVIAR4REID contains images from 72 different people and two non-overlapping cameras. In Fig. 1, several images from CAVIAR4REID can be seen. Fig. 1Open in figure viewerPowerPoint Small sample of the CAVIAR4REID data set The PETS 2006 data set [28, 29] has been designed for activity recognition; nevertheless, it has been employed in both people detection and recognition tasks. In particular, we use the sequences named S2-T3-C and S4-T5-A-C (for camera 1, camera 3, and camera 4, which are overlapping cameras). From PETS 2006, we select manually nine different people who stand in the three considered views. The PETS 2009 data set [30, 31] comprises image sequences containing crowd scenarios in an outdoor environment. We used named sequences S2-L1-Time, that is, camera 1, camera 5, camera 7, and camera 8, and all overlapping cameras. Similarly, to PETS 2006, from PETS 2009, we selected nine different people who stand in the four cameras used. Fig. 2 shows several images from these two data sets. Fig. 2Open in figure viewerPowerPoint Images of (a) PETS 2009, (b) PETS 2006 The most recent data set considered in this work is SAIVT-SoftBio [32]. This data set consists of several image sequences at an indoor environment. The images proceed from eight overlapping cameras and include 25 different people. Fig. 3 shows images from its eight different indoor cameras. Considering these four data sets, we test our approaches with 115 people, and with a variety of multi-cameras conditions such as the number and overlapping of the condition of the cameras. Fig. 3Open in figure viewerPowerPoint Images from SAIVT-SoftBio data set 4 Appearance model updating Generation and updating of the people's appearance model are essential parts of the people's re-identification. In this work, people detection is made following the approach presented in [33]. A set of appearance-based soft-biometric characteristics were extracted for each detected person. Since these traits are highly related to people's clothing, once a person is detected, we compute characteristics based grey-scale statistics and histograms, co-occurrence matrix and local binary patterns (LBPs), and also features related to RGB and HSV colour spaces, and geometry. As this work, it is focused on the updating process of an appearance model; all the features were computed following the work presented in [34]. These features were selected to work under sub-optimal resolution and lighting conditions, as it is the rule in a real security camera environment. In summary, we use 23 features, normalised between 0 and 1; these features are summarised in Table 1. Here, SD means standard deviation and Matrix of Co-Occurrence (MCO) means co-occurrence matrix. From these features, a vector is generated to identify each people in the scene or for a more detailed analysis, see [34]. Next, with this set of soft-biometric features, a model is constructed; that is, the people appearance model is generated and updated over several conditions. Table 1. General description of the extracted features Category Description RGB colour mean of red colour RGB colour mean of green colour RGB colour mean of blue colour RGB colour mean of three RGB RGB colour SD of three RGB RGB colour brightness grey scale mean grey scale SD geometry eccentricity grey-scale histogram entropy grey-scale histogram dispersion grey-scale histogram mean grey-scale histogram SD grey-scale histogram energy grey-scale histogram kurtosis HSV colour mean HSV colour SD statistics of the co-occurrence matrix energy statistics of the co-occurrence matrix maximum probability statistics of Matrix of Co-Occurrence (MCO) entropy statistics of MCO inertia statistics of MCO homogeneity LBP simple LBP In this paper, we introduce three different strategies for updating people's model. The first one is based on time lapses, the second one, on camera's changes, and the last one is based on an automatic selection of the one considered as the most representative of samples. We use a linear SVM with stochastic GD (SGD) training [35] as the classifier. In particular, we employ Python's Scikit-learn implementation [36]; this library implements mini-batch learning that helps us to carry out a sort of online learning; hence, we specify through these heuristics when this partial fit is done. The SVM has a well-studied potential to deal with online and incremental learning [2, 3]. The SGD is a stochastic approximation of the GD iterative optimisation method [37, 38]. A significant difference between GD and SGD is that SGD needs more iterations than GD, but each iteration needs lesser net computational resources. The SGD provides a solution for the underlying SVM optimisation problem, which minimises a cost function, that is, given the training set in the case of binary classification problems. The first step is the initialisation of ; then for each iteration, this weight vector must be updated. With the SGD method, in each iteration, a random sample for the training set I is chosen. Then, this subset is used as the full data set to calculate and update the new values of weight vector through the SGD optimisation approach [37, 38]. Leaving aside the learning procedure, we concentrate on the heuristics to determine the time when the re-trained process must be done. In the following sections, we focus on the details of the model's updating strategies. 4.1 Update based on time lapses In multi-camera scenarios, we found that people's aspect changes dramatically from one camera to another due to changes in lighting, perspective, distance from the camera, the zooming capacity, or quality, in general. These variabilities make the identification of people across cameras difficult. With the purpose of minimising this problem, it is necessary to update the model calculated for re-identification. The first proposed approach is based on the construction of an initial appearance model, and then, after a time lapse, the model is updated regardless of any change detected. Hereafter, this method will be called 'update based on time lapses' (UBTL). In this method, when a specified period has elapsed, the partial fit is done, that is, the SVM linear classifier with SGD learning is re-trained integrating new samples to the model. The specified time t is varied from 5, 10 to 15, that is, every t seconds the re-train is performed. Algorithm 1 (see Fig. 4) describes the process of this updated approach. The approach receives as input the images sequence Images, the parameter t which specifies when the update must be done, and the parameter iModel, which is the value for the generation of the initial model. The first images are used to generate an initial appearance model; the parameter that controls the number of images for this starting model is iModel. For instance, we set the value of iModel to 10; please note that setting iModel to a low value allows the use of the rest of the images for updating and testing. After this initial period, the updating process is done through the partial fit method of our classifier. That is, all the individuals in the image, at the specified time, are used as new samples. All identities are included in the classifier. The simplicity and determinism are the main advantages of this method. Fig. 4Open in figure viewerPowerPoint Algorithm 1: Update based on Time Lapses. 4.2 Update based on change of camera As mentioned before, people's appearance changes across different cameras. Therefore, our second proposal is the model's updating based on camera's change. This strategy starts with an initial appearance model from one camera, and when the re-identified subject crosses from one camera to another, the appearance model is updated. This method will be named update based on change of camera (UBCC). Algorithm 2 (see Fig. 5) details on how to proceed with camera changes; as before, the first iModel images are used to generate an initial model. The classifier is updated whenever the suspect is identified using a different camera. The core idea is to feed the classifier with images from the current cameras. For each camera, the first iModel images are used to re-train the classifier and the remaining to testing. Note that in this method, the more cameras are crossed by the suspect, the more samples are considered in the appearance model. Fig. 5Open in figure viewerPowerPoint Algorithm 2: update based on change of camera 4.3 Update based on the most representative samples This method is designed to take advantage of the most representative samples. Here, an initial model is generated with the iModel first images. In the next stages, most representative samples are selected with a quality threshold for the positive samples and another quality threshold for the negative samples. To calculate these scores, we apply the classification process to a subset of the data set. This procedure gives us a set of decision functions values from all images considered. With these values, we plotted a histogram to visualise the distribution of both positive and negative decision functions values of the whole subset of images. Later, we choose threshold values by quartiles using the histogram's values; the final thresholds were established with the idea of selecting those values located in the limits of the histogram. Fig. 6 illustrates the above; here, we can see the distribution of both negative and positive classes, and the location of the extreme values. Note that positive-score values are defined by those where the target person is identified correctly, and the negative-score values are defined by values obtained from people not being the target. Fig. 6Open in figure viewerPowerPoint Histograms of positive and negative scores Using this methodology in this subset, we found that positive scores have values between 0 and 6, and negative scores have values between −10 and 0. We consider the extreme values, so we fix the positive threshold to 1.5 and the negative threshold to −2.5. Therefore, the updating process is started whenever any of these thresholds are crossed. Algorithm 3 (see Fig. 7) describes the behaviour of this approach. As well as prior methods, an initial appearance model is generated with the first images (specified by iModel). The parameters posTh and negTh are the positive and negative thresholds, respectively. Later, for each image, a score is obtained for each detected person using the prior appearance model, when the score of a sample is higher or lower than the considered quality thresholds, the appearance model is updated through the re-training of the classifier. Owing to the high number of calculations, this method is the most time-consuming among our approaches. Fig. 7Open in figure viewerPowerPoint Algorithm 3: update based on most representative samples In all the cases, the parameters of the algorithms are the set of images, t is the time associated with each image, and the iModel value, which controls the bootstrapping of the system. In particular, the last value is provided by the operator of the surveillance system. On the other hand, all the evaluation metrics [see (1)–(4)], are calculated in the test phase. 5 Experiments and analysis Our experimental evaluation consists of the use of four popular public benchmarks, PETS 2006, PETS 2009, CAVIAR4REID, and SAIVT-SoftBio databases. These data sets were created on both overlapping and non-overlapping surveillance environments. From these databases, a total of 115 different individuals were extracted, with about 274 images per person; the experiments were carried out with more than 31,000 images. Furthermore, the experiments were conducted regarding spatial and temporal restrictions, i.e. two individuals need to be compared if they occur in the same space (or camera) at the same time. The evaluation is oriented to simulate an operational environment; that is, a suspect must be compared with other people present in the same scene at the same time. These benchmarks help us to provide experimental evidence to support the importance of updating models in multi-camera surveillance methods and, in particular, to measure the performance of our strategies. Note that for some time lapse, we use all previously known images of an objective person as the training data and the remaining images in time as testing data. This scheme avoids the use of the same data for training and testing, but each approach has a different size of data and time; we present the average values of ten independent runs to reduce the variation among the stochastic behaviour of underlying methods. For the evaluation, we use and accuracy scores. Equation (1) shows how the accuracy is calculated. Here, TP means for true positives, that is, it counts all correct predictions of the positive class. TN means for true negatives, i.e. it counts all correctly predicted negatives. FP counts the mismatch regarding positive predictions, while FN (false negatives) counts failures regarding predictions of the negative class. The recall and precision metrics are calculated as (2)–(4) indicate (1) (2) (3) (4)It is possible to know the interconnection of the cameras in