# Reliable Recognition of Partially Occluded Objects with Correlation Filters

Authors: Alexey Ruchay; Vitaly Kober; Jose A. Gonzalez-Fraga
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

Design of conventional correlation filters requires explicit knowledge of the appearance and shape of a target object, so the performance of correlation filters is significantly affected by changes in the appearance of the object in the input scene. In particular, the performance of correlation filters worsens when objects to be recognized are partially occluded by other objects, and the input scene contains a cluttered background and noise. In this paper, we propose a new algorithm for the design of a system consisting of a set of adaptive correlation filters for recognition of partially occluded objects in noisy scenes. Since the input scene may contain different fragments of the target, false objects, and background to be rejected, the system is designed in such a manner to guarantee equally high correlation peaks corresponding to parts of the target in the scenes. The key points of the system are as follows: (i) it consists of a bank of composite optimum filters, which yield the best performance for different parts of the target; (ii) it includes a fragmentation of the target into a given number of parts in the training stage to provide equal intensity responses of the system for each part of the target. With the help of computer simulation, the performance of the proposed algorithm for recognition partially occluded objects is compared with that of common algorithms in terms of objective metrics.


---

## Body

## 1. Introduction

Recognition and tracking of objects in observed scenes degraded by additive noise, in the presence of cluttering backgrounds, geometric modifications such as pose changing and scaling, nonuniform illumination, and eventual object occlusions are challenges that a modern recognition algorithm must solve. In this paper, we deal with partial occlusion of objects to be recognized, in other words, when only some parts of the target are visible. Recent works have paid much attention to this problem [1–5].Nowadays, object recognition based on correlation filters receives much research interest due to its high impact in real-life activities, such as video surveillance, human-computer interaction, robotics, biometrics, and target tracking [6–12]. Correlation filtering is a powerful technique for object recognition because of its ability to perform two essential tasks simultaneously: detection of a target within an observed scene and computation of the exact position of the detected object [13, 14]. Another advantage of correlation filters is their ability to detect multiple objects in a single scene simultaneously [15–17].The performance of correlation pattern recognition may be improved either by discarding noise components from the output of a linear system [18] or by using an adaptive approach to the filter design [19]. The former approach is suitable for classification problems [20], whereas the latter is preferable for detection and tracking applications. For the case of nonstationary noise such as a cluttered background, statistical parameters of the noise are space-variant. The frequency response of a correlation filter is locally adapted to the parameters estimated in small spatially homogeneous fragments of the input scene. The locally adaptive filter improves pattern recognition in terms of location errors for a noisy environment that is important for accurate target detection.Conventional correlation filters without training may yield a poor performance to recognize a target partially occluded by other objects [21], for example, to recognize a pedestrian partially covered with a tree or a man wearing sunglasses. There are several proposals to treat partial occlusions with correlation filters [22–29]. All of them use independent parts of the target to synthesize a composite correlation filter. However, no study was carried out on an augmented division of the object into parts.Campos et al. [22] carried out a study on the performance of some correlation filters to discriminate occluded objects. They compare the phase-only filter, the inverse filter, and the trade-off filter between the minimum variance and minimum average correlation energy. All used filters enhance the edges of the object in order to have a good discrimination. The target is divided uniformly into seven parts without any justification. Moreover, the performance of the filters in the presence of noise and geometric distortions was not analyzed. Adaptive correlation filters for recognition of fragmented objects imbedded into real-life scenes and in the presence of additive noise were presented [23, 24]. The target is divided into independent fragments for the design of an adaptive filter. It was supposed that at least one of the fragments responses to the visible fragment of the target is embedded into the scene. Additionally, the algorithm uses available contour and texture information to improve recognition of partially occluded objects. Recent work [28] improves recognition of partially occluded objects embedded into a known cluttered background with an adaptive composite filter. The proposed filters are able to discriminate noisy similar objects, even, when available information of a target is about 19%. Khoury et al. [30] developed several optimal correlation algorithms for detection of obscured targets embedded into a disjoint background. It was noted that the boundary between obscuring and obscured objects makes a significant contribution to the correlation peak. So, blurring of the boundaries was utilized for detection of obscured targets.Recently, masked correlation filters (MCFs) were designed [31] to handle partial occlusions in face images. MCFs utilize prior knowledge of the location of partial occlusions in test images as well as the zero-aliasing correlation filtering (ZACF) [27]. Since in real-life applications the location of partial occlusions is usually unknown, the filters cannot be widely used.Finally, note that, in the design of common correlation-based methods, the target is arbitrarily divided into a number of parts, which are used for the design of composite filters. One of the motivations of this research is to determine a reasonable way for the target division to guarantee a high level of the overall system performance. In order to obtain a good recognition of each target part in noisy input scenes, the optimum correlation filters are also utilized [32].The paper is organized as follows. Section2 recalls the design of composite correlation filters. Section 3 describes the proposed algorithm for target fragmentation and robust recognition of partially occluded objects with multiple composite filters. Section 4 with the help of computer simulation presents the performance of the proposed algorithms in terms of detection efficiency. The results are discussed and compared with those obtained with common correlation filters. Finally, Section 5 presents our conclusions.


## 2. Composite Correlation Filters

We are interested in the design of a correlation filter that is able to recognize a fragment of the target embedded into a disjoint background in the scene corrupted with additive noise. The designed filter should be also able to recognize geometrically distorted versions of the target. LetT={ti(x,y);i=1,…,N} be an image set containing geometrically distorted versions of the target. The input scene is assumed to be composed by the target t(x,y) embedded into a disjoint background b(x,y) at unknown coordinates (τx,τy), and the scene is corrupted with additive noise n(x,y), as follows:(1)fx,y=tx-τx,y-τy+bx,yw¯x-τx,y-τy+nx,y,where w¯(x,y) is a binary function defined as zero inside the target area and unity elsewhere. The optimum filter for detecting the target, in terms of the maximum of the signal-to-noise ratio (SNR) and the minimum variance of location errors (LE), is the generalized matched filter (GMF) [13], whose frequency response is given by(2)GMF∗u,v=Tu,v+μbW¯u,vPbu,v⊗W¯u,v2+Pnu,v,where T(u,v) and W¯(u,v) are the Fourier transforms of t(x,y) and w¯(x,y), respectively; μb is the mean value of the background b(x,y); Pb(u,v) and Pn(u,v) denote the power spectral densities of b0(x,y)=b(x,y)-μb and n(x,y), respectively. Symbol ⊗ denotes convolution.Lethi(x,y) be the impulse response of a GMF constructed for the ith available view of the target ti(x,y). Let H={hi(x,y);i=1,…,N} be the set of all GMF impulse responses constructed for all training images ti(x,y). Additionally, let S={si(x,y);i=1,…,M} be an image set containing M unwanted patterns to be rejected. In order to recognize all target views in T and reject the false patterns in S, by combining the optimal filter templates contained in H, we synthesize a composite correlation filter. Filter p(x,y) can be constructed as follows [33]:(3)px,y=∑i=1Nαihix,y+∑i=N+1N+Mαisix,y,where the coefficients {αi;i=1,…,N+M} are chosen to satisfy prespecified output values for each pattern in U=T∪S. Using vector-matrix notation, we denote by R a matrix with N+M columns and d rows equal to the size of the images, where each column is the vector version of each element of H∪S. Let a=[αi;i=1,…,N+M]T be a vector of coefficients. Thus, (3) can be rewritten as(4)p=Ra.Let us denote by(5)u=1,…,1︸Nones,0,…,0︸MzerosTthe desired responses to the training patterns and denote by Q the matrix whose columns are the elements of U. The response constraints can be expressed as(6)u=Q+p,where superscript + denotes conjugate transpose. Substituting (4) into (6), we obtain (7)u=Q+Ra,(8)a=Q+R-1u.Finally, substituting (10) into (4), the solution for the composite filter is given by(9)p=RQ+R-1u.Note that the value of the correlation peak obtained with (9) is expected to be close to unity for true-class objects and close to zero for false-class objects.The MACE [34] filter minimizes the average correlation energy of the correlation outputs for the training images while simultaneously satisfying the correlation peak constraints at the origin. The effect of minimizing the average correlation energy is that the resulting correlation planes would yield values close to zero everywhere except at the location of a trained object, where it would produce an intense peak. In the Fourier domain, the MACE filter can be expressed in vector form as follows:(10)pMACE=D-1RR+D-1R-1u,where matrix D contains along its diagonal the average power spectrum of the training images (i.e., average of the magnitude squares of the columns of R).The Optimal Trade-off Synthetic Discriminant Function (OTSDF) [14] filter is a correlation filter that is similar to the MACE filter. In the OTSDF formulation, matrix D is replaced with V=D+δI, where I is an identity matrix and δ>0. The inclusion of the identity matrix improves noise tolerance.The discrimination capability (DC) is a measure of the ability of the filter to distinguish a target from unwanted objects; it is defined by the following [33]:(11)DC=1-cb2ct2,where cb is the value of the maximum correlation sidelobe in background area and ct is the value of the correlation peak generated by the target. A DC value close to unity indicates that the filter has a good capability to distinguish between the target and any false object. Negatives values of the DC indicate that the filter is unable to detect the target. Also, if the obtained DC is greater than a prespecified threshold (DC>DCth), then the target is considered as detected and, otherwise, the target is rejected.


## 3. Recognition of Partially Occluded Objects

In this section we describe the proposed algorithm for recognition of partially occluded objects using a new target fragmentation procedure and a bank of composite correlation filters. To improve the detection performance of correlation filters an adaptive approach to the filter design is utilized [35].The proposed algorithm for automatic fragmentation of the target inton parts is shown in Figure 1.Figure 1
Block diagram of optimal fragmentation of the target inton fragments.First, suppose that a visible fragment of the object obtained due to occlusion of the object always contains a part of the object contour. So, we define fragmentsF1,…,Fn as sectors of the circle inscribing the object. For each of the fragments F1,…,Fn and the object O(x,y), the output of the linear system can be defined as correlation peaks between the impulse response of the correlation filter and the corresponding fragment or the entire object, that is, PF1,…,PFn, and PO, respectively. In general, the impulse response of the optimum filter (see (2)) depends on the input scene information. If such information is unavailable, the phase-only filter may be used for approximate solution of the problem. We want to divide the target into n sectors in such a manner to obtain equal responses of the linear system to each fragment. In other words, we look for the solution to the following functional:(12)Fi∗i=1n=argminFii=1n⁡∑i=1nPOn-PFi,where {Fi}i=1n are all possible divisions of the object O(x,y). Actually, minimization of the functional (12) may produce numerous solutions. In this case, additional constraints such as equal area of the fragments can be used for selection a unique one.Figure2 illustrates the optimal fragmentation of the object from 2 to 9 fragments with the help of the proposed algorithm.Figure 2
Results of the optimal fragmentation of the object into2,…,9 fragments.The filter design requires knowledge of a typical background image and a target. We construct a bank of composite optimum filters. The proposed algorithm for the design of composite correlation filters is given as follows.Step 1.
The algorithm starts with an optimal fragmentation of the objectO(x,y) into n fragments {Fi∗(x,y)}i=1n (see Figure 1).Step 2.
Synthesize GMF correlation filters{hi(x,y)}i=1n in (2) for reliable detection and location estimation of the fragments.Step 3.
Fori=1,…,n, synthesize a composite adaptive correlation filter pi(x,y) as follows [35]: (i) perform correlation between hi(x,y) and the background image. If the obtained DC is greater than a prespecified threshold (DC>DCrec), then the fragment could be successfully detected in the input scene, and Fi(x,y) is added to set T; otherwise, the detected object si(x,y) around false peak is added to the set S; (ii) synthesize a composite filter pi(x,y) with the help of (9); (iii) iteratively perform steps (i) and (ii) with pi(x,y) until the condition DC>DCrec is satisfied.Step 4.
The bank of composite adaptive filters{pi(x,y)}i=1n is used for reliable recognition of partially occluded objects. Detection can be carried out by correlating the input scene with each filter of the bank. Next, the DC in each of the correlation planes is calculated, and the plane with the highest DC value is chosen as the system output. If the obtained DC is greater than a prespecified threshold (DC>DCth), then the target is considered as detected; otherwise, the object is rejected. Finally, the location of the correlation peak in the chosen output plane is taken as an estimate of the location of the object in the scene. The recognition procedure with the bank of filters is summarized in Figure 3.Figure 3
Block diagram of the proposed algorithm for partially occluded object recognition.Note that in the proposed algorithm the numbern of fragments can be properly chosen to ensure a desired accuracy using a minimum number of correlation operations. Detection performance and location accuracy are monotonically increasing with the number of filters in the bank; therefore, there exists a trade-off between a desired quality of detection and computational complexity in terms of required correlations.


## 4. Computer Simulation

In this section, the performance of the proposed algorithm for recognition of partially occluded objects is presented in terms of detection efficiency. The results are compared with those obtained with successful composite correlation filters, that is, MACE [34], OTSDF [22], SDF [14, 35], and SDF with MACE (SMACE) filters [23, 24].In this paper, type I and type II recognition errors are used for comparing the recognition accuracy of tested algorithms. Type I error occurs when the algorithm asserts something that is absent, a false hit. Type I error is called false positive (FP). Type II error occurs when the algorithm fails to assert what is present, a miss. Type II error is called false negative (FN).With the help of extensive computer simulation we show how detection reliability and localization accuracy for recognition of partially occluded objects with common and proposed correlation filters depend on the number of fragments, level of input noise, and level of target overlapping. In order to guarantee correctly statistical results, we use 55 different scenes and 16 different objects (see Figure4). The algorithms are tested in input scenes containing the entire target and randomly occluded targets with the level of available information of 20%, 40%, 60%, 80%, and 100% of the object area. Also, input scenes are corrupted by additive white noise with signal-to-noise ratio (SNR) of 5 dB, 10 dB, 15 dB, 20 dB, and 25 dB.Figure 4
Examples of scenes with different objects.Tables1 and 2 show the FP and FN errors for the proposed algorithm as a function of number n=1,…,9 of fragments, SNR of 5 dB, 10 dB, 15 dB, and 25 dB, and the level of available information of 20%, 40%, 60%, 80%, and 100%. Note that the threshold DCth = 0.6 provides the minimum of the FP and FN errors.Table 1
FN errors (%) for the proposed algorithm depending on numbern=1,…,9 of fragments (F1,…,F9); SNR of 5 dB, 10 dB, 15 dB, and 25 dB; level of available information (OV) of 20%, 40%, 60%, 80%, and 100%.
SNR, OV% F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 5 dB 20% 100 99.4 99.1 98.6 98.5 97.4 98.3 97.6 95 5 dB 40% 96.2 86.8 80.8 70.9 74.8 71.2 66.7 69 62.6 5 dB 60% 47.3 36.2 31.1 25.1 28.2 22.6 21.2 22.1 19.8 5 dB 80% 12.7 11.6 10.5 8.1 9.2 6.4 7.5 6.5 5.8 5 dB 100% 6 5.7 5.4 5.3 4.5 3.7 3.8 4 3.5 10 dB 20% 96.6 90.6 86.9 85.7 85.8 84.9 85 83.1 83.5 10 dB 40% 49.8 24.8 17.8 12.6 13.1 10.6 7.4 8.4 7.2 10 dB 60% 2.1 0.8 0.6 0.1 0.5 0 0.1 0.5 0.2 10 dB 80% 0.2 0 0.2 0 0 0.1 0.1 0 0 10 dB 100% 0 0 0 0 0 0 0 0 0 15 dB 20% 82.9 75.2 73 73.3 72.1 72.1 72.2 71.5 71.1 15 dB 40% 3.9 1.3 0.5 0.8 0.5 0.1 0 0.2 0 15 dB 60% 0 0 0 0 0 0 0 0 0 15 dB 80% 0 0 0 0 0 0 0 0 0 15 dB 100% 0 0 0 0 0 0 0 0 0 25 dB 20% 71.8 70 69.1 67.1 66.4 65.3 64.3 62.3 62.6 25 dB 40% 0.2 0.1 0 0 0 0 0 0 0 25 dB 60% 0 0 0 0 0 0 0 0 0 25 dB 80% 0 0 0 0 0 0 0 0 0 25 dB 100% 0 0 0 0 0 0 0 0 0Table 2
FP errors (%) for the proposed algorithm depending on numbern=1,…,9 of fragments (F1,…,F9); SNR 5 dB, 10 dB, 15 dB, and 25 dB.
SNR, OV% F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 5 dB 0% 0 0 0 0 0 0.1 0.2 0.2 0.1 10 dB 0% 0 0 0 0 0 0 0.1 0.2 0.2 15 dB 0% 0 0 0.1 0.2 0.1 0.1 0.3 0.4 0.2 25 dB 0% 0 0.1 0.9 1.1 1.4 1.7 2 2.1 2.3One can observe that the proposed algorithm with a strong occlusion (20% of available information) yields high FN errors of 62%–100% depending on the number of fragments and noise level. However, the recognition performance of the algorithm improves rapidly when either the number of the fragments or SNR increases. On the other hand, the performance of the proposed algorithm in terms of FP errors is excellent.Tables3 and 4 show FP and FN errors for the proposed algorithm with division of the object into 9 fragments (F9), MACE, OTSDF, SDF, and SDF with MACE.Table 3
Comparison of the performance in terms of FN errors (%) of the proposed algorithm (F9), MACE, OTSDF, SDF, and SDF with MACE (SMACE) filters; SNR of 5 dB, 10 dB, 15 dB, and 25 dB; level of available information (OV) of 20%, 40%, 60%, 80%, and 100%.
SNR, OV% F 9 MACE OTSDF SDF SMACE 5 dB, 20% 95 96.2 96 96.6 96.9 5 dB, 40% 62.6 89.6 90.1 91.2 75.5 5 dB, 60% 19.8 73.5 76.5 72.4 69.8 5 dB, 80% 5.8 61.4 64.5 62.4 59.4 5 dB, 100% 3.5 30.9 32.3 31.7 30.8 10 dB, 20% 83.5 93.3 90.2 91 89.7 10 dB, 40% 7.2 52.2 58.6 53.3 30.4 10 dB, 60% 0.2 24.9 30.7 23.5 20.3 10 dB, 80% 0 17.1 18.6 16.4 16 10 dB, 100% 0 8.6 9.9 8.4 8.4 15 dB, 20% 71.1 83.4 78.4 82.4 78.3 15 dB, 40% 0 22.5 22.2 18.4 8.7 15 dB, 60% 0 5.8 7 5.7 3.2 15 dB, 80% 0 2.7 2.9 2.4 3.1 15 dB, 100% 0 1.1 1.3 1.4 1.2 25 dB, 20% 62.6 66.6 66.9 69 64.4 25 dB, 40% 0 14.3 13 11.7 6.5 25 dB, 60% 0 2.5 3.1 2.4 2.5 25 dB, 80% 0 1.1 1.2 1.5 1.7 25 dB, 100% 0 0.4 0.2 0.2 0.4Table 4
Comparison of the performance in terms of FP errors (%) of the proposed algorithm (F9), MACE, OTSDF, SDF, and SDF with MACE (SMACE) filters; SNR of 5 dB, 10 dB, 15 dB, and 25 dB.
SNR, OV% F 9 MACE OTSDF SDF SMACE 5 dB 0% 0.1 0.4 0.6 0.5 0.5 10 dB 0% 0.2 0.7 0.9 0.8 0.7 15 dB 0% 0.2 0.8 1.1 0.9 1 25 dB 0% 2.3 2.1 2.5 2.1 2.5The tested algorithms use the same number of fragments. However, the division of the object into nonoverlapping fragments is performed uniformly according to the authors’ recommendation. One can observe that the proposed algorithm yields much better performance for different circumstances with respect to the objective criteria.


## 5. Conclusion

In this paper, we proposed a new algorithm for the design of a linear system consisting of a set of adaptive correlation filters for recognition of partially occluded objects in noisy scenes. The system consists of a bank of composite optimum filters, which yield the best performance for different parts of the target. In the training stage, the system divides the target into a given number of parts to provide equal intensity responses of the system for each part of the target. With the help of computer simulation, we showed that the performance of the proposed algorithm for recognition partially occluded objects is much better that of common algorithms in terms of objective metrics.



---

*Source: 8284123-2018-05-14.xml*