# The Removal of False Detections from Foreground Regions Extracted Using Adaptive Background Modelling for a Visual Surveillance System

Dariusz Frejlichowski, Katarzyna Gościewska, Pawel Forczmański, Adam Nowosielski, Radosław Hofman

► To cite this version:

Dariusz Frejlichowski, Katarzyna Gościewska, Pawel Forczmański, Adam Nowosielski, Radosław Hofman. The Removal of False Detections from Foreground Regions Extracted Using Adaptive Background Modelling for a Visual Surveillance System. 12th International Conference on Information Systems and Industrial Management (CISIM), Sep 2013, Krakow, Poland. pp.253-264, 10.1007/978-3-642-40925-7_24. hal-01496072

HAL Id: hal-01496072

https://inria.hal.science/hal-01496072

Submitted on 27 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The Removal of False Detections from
## Foreground Regions Extracted using Adaptive
## Background Modelling for a Visual Surveillance
## System

Dariusz Frejlichowski¹, Katarzyna Gościewska¹,², Paweł Forczmański¹,
Adam Nowosielski¹, and Radosław Hofman²

¹ West Pomeranian University of Technology, Szczecin
Faculty of Computer Science and Information Technology
Żołnierska 52, 71-210, Szczecin, Poland
{dfrejlichowski,pforczmanski,anowosielski}@wi.zut.edu.pl

² Smart Monitor, sp. z o.o.
Niemierzyńska 17a, 71-441, Szczecin, Poland
{katarzyna.gosciewska,radekh}@smartmonitor.pl

**Abstract.** For recent surveillance systems, the false detection removal process is an important step which succeeds the extraction of foreground regions and precedes the classification of object silhouettes. This paper describes the false object removal process when applied to the 'Smart-Monitor' system — i.e. an innovative monitoring system based on video content analysis that is currently being developed to ensure the safety of people and assets within small areas. This paper firstly briefly describes the basic characteristics and advantages of the system. A description of the methods used for background modelling and foreground extraction is also given. The paper then goes on to explain the artefacts removal process using various background models. Finally the paper presents some experimental results alongside a concise explanation of them.

**Keywords:** 'SmartMonitor', visual surveillance system, video content analysis

## 1 Introduction

The 'SmartMonitor' system is being developed to combine the advantages of small closed-circuit television systems (CCTV) and visual content analysis algorithms (VCA). It aims to increase the safety of individual clients and their assets — i.e. houses, apartments, shops, enterprises, etc. — and help to ensure security in their surroundings. Through customisation of system parameters, all users will be able to set individual safety rules and adjust the system sensitivity level to suit their actual needs. Such customizability is one of the most important advantages. The system will be an affordable solution and will utilize only commonly available hardware, i.e. a personal computer and digital camera(s). Human
participation will be reduced to the minimum and only the calibration process
will require user interaction. 'SmartMonitor', as an innovative surveillance sys-
tem, will utilize specific algorithms to perform video analysis and to react to
particular events in pre-specified ways. These algorithms will be integrated into
six main system modules responsible for background modelling, object tracking,
artefacts removal, object classification, event detection and system response.

The system will operate under four independent scenarios, which concern
home/surroundings protection against unauthorized intrusions (Scenario A), su-
pervision of a person who is ill (Scenario B), crime detection (Scenario C) and
smoke/fire detection (Scenario D). Each scenario can be characterized by several
possible actions and conditions, such as movement detection, object tracking,
region limitations, object classification, object size limitations, object feature
changes, weather conditions, and work time. The conditions which the system
will finally work under are very important. For example, changes in lighting or
the appearance of either shadows or reflections in a scene can significantly influ-
ence the background model and result in the foreground image being affected by
false objects. Therefore, it is crucial to appropriately eliminate false detections
in order to extract only the actual object of interest (OOI). Each OOI should
be a coherent region of a pre-specified minimum size. More detailed system de-
scription is provided in [1]. 'SmartMonitor' has been described in [2] and [3] as
well.

In this paper the process of false object removal for the 'SmartMonitor' sys-
tem is presented. Firstly, it briefly describes the method used for background
modelling and foreground extraction, which directly precedes the elimination of
falsely detected regions. Additionally, the types of false detections used are sum-
marised. Secondly, the artefacts removal process as applied to two background
models using additional operations is described. The paper then presents some
experimental results and a concise discussion of them. The database utilized in
the process of artefacts removal is very specific and was prepared only for the ex-
periments concerning SmartMonitor system. The final section contains summary
and conclusions.

## 2 The Process of Background Subtraction and Types of False Detections

The background modelling, foreground localization and artefacts removal pro-
cesses are elements of the background subtraction process, which is presented as
a flowchart in Fig. 1. The first stage, image pre-processing, utilizes algorithms
that are mainly for colour conversion or image enhancement and helps facil-
itate image processing during the subsequent stages. The background models
are built adaptively using a Gaussian Mixture Model (GMM), based on various
colour components. The foreground image is the result of the subtraction of the
background image from a pre-processed video frame. During the data validation
stage, the system verifies whether the foreground regions correspond to real or
false objects. This is done during the false detection removal process and the
final foreground binary mask should only be created for the regions that the
system is interested in.

Building an adequate background model is very important since the model
influences both the obtained foreground regions and the number of false detec-
tions. The background model must be both sensitive enough to localise any real
moving objects and sufficiently robust to particular environmental changes in
the scene. In this case it cannot be static [4] or averaged in time [5] because real
scenes are very changeable in time. Therefore, the GMM method (e.g. [6-8])
has been selected. It allows the adaptive modelling of each pixel as a mixture
of Gaussians. This method has been evaluated to be a reliable real-time tracker
that can quickly adapt any changes appearing within the scene and can deal
with slow variations in illumination and repeated disturbances from unexpected
motion in the background clutter [9].

<figure><img src="image_2.png" /><figcaption>Fig. 1. Flowchart of the background subtraction algorithm (based on [10]).</figcaption></figure>

Despite the advantages, the method has some drawbacks that are mainly
associated with the utilized colour information and the type of the environment
which is present in the scene. False detections can take various forms, such as
large coherent regions, isolated pixels or small groups consisting of several pixels.
The reasons for artefacts occurrence are [11]:

- sudden changes in illumination, that can completely change the background colours and increase the difference between the model and the current frame;

- shadows of moving objects, that cause illumination changes and might be classified as foreground regions;

- background movements — i.e. the relocation of a part of the background, which causes a change within two regions, namely the newly acquired and previous positions. Both become considered as part of the foreground whereas only the previous position should;

- background initialization in the presence of moving objects — moving objects that belong to the foreground are mistakenly incorporated into the background and partly occlude it.

Fig. 2 shows examples of various types of false detections obtained for a back-
ground model with 256 grayscale values. Each row contains a sample video frame,
a background model and a foreground image respectively. It is evident that the resulting foreground regions are larger than they should be, with the area of each moving object being surrounded by redundant pixels. Fig. 2 illustrates the results of:

(a) a sudden illumination change caused by sunrays passing through a window. Other possible causes are: turning the light on and off, a camera flash or changing weather conditions such as a partially cloudy sky with sun shining through the clouds;

(b) the appearance of a moving object shadow area;

(c) the movement of small background elements, mainly tree leaves and grass. As a result the foreground region is very noisy;

(d) model initialization with a frame containing a moving object which occludes part of the background. Because there is no information about the occluded background region, it is considered as part of the foreground.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Examples of false detections for the intensity model.</figcaption></figure>
In summary, the foreground image can include pixels that correspond to both moving objects and false detections. Artefacts can take the form of larger coherent regions, small groups of pixels, or single isolated pixels. False detections are often connected with the actual object of interest, which makes them more difficult to eliminate. Areas of noise can also result from image compression or low data transmission quality.

# 3 The Process of Artefacts Removal

For the 'SmartMonitor' system artefacts removal process, three key issues were taken into consideration and needed to be eliminated, namely the influence of a moving object that was visible in the scene during background model initialization, shadow areas and noisy regions. First of all, two separate background models based on the Y component (intensity) of the YIQ colour scheme and the H component (hue) of the HSV colour scheme were built. Various background models (see Fig. 3(a) and Fig. 3(b) for examples) allowed different information about possible ways to simplify the artefacts removal process to be obtained. Moreover, additional operations and modifications were introduced in order to eliminate the aforementioned false detections.

The first problem, i.e. the presence of a moving object in the background model, occurs mainly when a model is initialized using the first frame obtained from the video stream. However, it is often impossible to obtain an image without moving objects in it, especially from busy environments. Therefore, the initial background image pixel values should be random. On the one hand, this will cause the model to require more time to adapt to the current situation of the scene, and on the other, the resultant background image will be more accurate and the influence of moving objects will be reduced.

The next issue is associated with shadow detection and removal. Here, the utilization of two different background models is discussed. It was stated in [9] that by building the background model using intensity values it is impossible to distinguish between moving objects and moving shadows. The authors of [12] proposed the utilization of a colour component to detect moving shadows and to reduce the computation time. According to [13], shadows affect only intensity values and not the hue of shaded and open regions to a significant extent. Therefore, the comparison of two different foreground images, obtained using intensity and hue background models, allows for shadow exclusion. Two exemplary foregrounds are illustrated in Fig. 3(c) and Fig. 3(d).

The last problem to be solved is that of the elimination of noisy areas. This is done in two stages and each foreground image is processed separately before shadow exclusion is performed. Firstly, image thresholding is performed in which non-zero pixel values are changed to 1. This results in an image with white foreground areas and a black background. Secondly, two morphological operations are applied — i.e. erosion and dilation. Erosion allows for the elimination of isolated groups of several pixels, hence resulting in reduction of the foreground region. Dilation then fills-in the holes and completes the pixels removed from
the object region. Afterwards, the process of shadow elimination takes place. Here, two foreground binary images without noisy areas are multiplied using an entrywise product — i.e. when two pixels of the same coordinates are considered to be part of the foreground, their respective pixel in the final binary mask is white, otherwise, the zero value is assigned. A pictorial representation of the particular stages of the artefacts removal process is shown in Fig. 4.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Examples of: (a) an intensity background model, (b) a hue background model, (c) an intensity foreground image and (d) a hue foreground image.</figcaption></figure>

As can be seen from Fig. 4, both the shadow area and the majority of noisy regions were eliminated. The obtained foreground mask contains a moving object shape and several smaller regions that are not taken into consideration since the system only analyses objects of a pre-specified minimum size.

# 4 Experimental Conditions and Results

The experiments were performed using a test database containing a set of video sequences which were suitable for pre-planned system scenarios with specified parameters. The main goal was to investigate the effectiveness of the false object removal process when applied to the 'SmartMonitor' system. Each experiment was carried out in the same way. The background model was first initialized with random pixel values and then iteratively adapted to each subsequently processed frame. The subsequent stages of the background subtraction process for an individual video frame are shown in Fig. 5.
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** The illustration of the consecutive steps of the artefact removal process: input image thresholding, erosion, dilation and foreground mask extraction.</figcaption></figure>

Fig. 6–9 present some experimental results for various system working conditions and different environments — i.e. scenes recorded both inside and outside a building, with changing weather or lighting conditions, and background movements present. Each figure contains: (a) a sample frame, (b) a foreground image of the intensity model, (c) a foreground image of the hue model and (d) a
foreground binary mask. A sample frame from the video sequence fulfilling the conditions of Scenario A, which shows a person walking in a garden, is depicted in Fig. 6(a). A shadow area was detected in the intensity model (Fig. 6(b)) and the foreground (Fig. 6(c)) was affected by noisy areas resulting from background movements and video compression. However, the moving object shape was extracted as expected (Fig. 6(d)). Fig. 7 shows the results for Scenario B. Here, the sample frame shows a person who fell over and is lying still. Noise and shadow areas were smaller than in the previous example and were completely removed. A similar situation is visible in Fig. 8, which concerns Scenario C, with sample frame presenting a person with hands raised. The moving shadow visible in Fig. 9(a) was also eliminated and so the final binary mask (Fig. 9(d)) does not contain any foreground areas.

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** The diagram of background subtraction process with artefacts removal stages.</figcaption></figure>

Experimental results were evaluated visually by the users and showed that the majority of falsely detected objects have been successfully removed. This proves the effectiveness of the false object removal process in application to the 'SmartMonitor' system. There is no available data that could be compared with the results as a ground-truth — the utilized database is very specific and adapted to system scenarios.
<figure><img src="image_7.png" /><figcaption>Fig. 6. Exemplary results of the artefacts removal process for scenario A.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7. Exemplary results of the artefacts removal process for scenario B.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Fig. 8. Exemplary results of the artefacts removal process for scenario C.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9. Exemplary results of the artefacts removal process for shadow removal.</figcaption></figure>
# 5 Summary and Conclusions

In the paper, the application of the artefacts removal process to the 'SmartMonitor' system was presented. Firstly, it introduced the main characteristics of the system and some basic information about the background subtraction process. The method for eliminating false detections within two background models was then described. Despite the fact that the system is currently under development, some experimental results were also provided and discussed.

Different background models allow various foreground images to be obtained. Therefore, we focused on the intensity component of the YIQ colour scheme and the hue component of the HSV colour scheme in order to exclude shadow arteas. Moreover, we introduced additional operations and modifications in order to eliminate false detections. These were morphological operations for noise reduction, and the use of an initial background model with random pixel values to decrease the influence of moving objects present during initialization. The experiments showed that the proposed method gave satisfactory results. However, several additional issues, such as camera placement or system sensitivity level, could influence both the number of artefacts and number of false alarms.

**Acknowledgements.** The project 'Innovative security system based on image analysis — "SmartMonitor" prototype construction' (original title: Budowa prototypu innowacyjnego systemu bezpieczeństwa opartego o analize obrazu — "SmartMonitor") is a project co-founded by the European Union (project number PL: UDA-POIG.01.04.00-32-008/10-01, Value: 9.996.604 PLN, EU contribution: 5.848.800 PLN, realization period: 07.2011-04.2013). *European Funds — for the development of an innovative economy* (*Fundusze Europejskie — dla rozwoju innowacyjnej gospodarki*).

# References

1. Frejlichowski, D., Forczmański, P., Nowosielski, A., Gościewska, K., Hofman, R.: SmartMonitor: An Approach to Simple, Intelligent and Affordable Visual Surveillance System. In: Bolic, L. et al. (eds.) ICCVG 2012. LNCS, vol. 7594, pp. 726–734. Springer, Heidelberg (2012)

2. Forczmański, P., Frejlichowski, D., Nowosielski, A., Hofman, R.: Current trends in developing of intelligent visual monitoring systems (in Polish). Methods of Applied Computer Science 4/2011(29), 19–32 (2011)

3. Frejlichowski, D., Gościewska, K., Forczmański, P., Nowosielski, A., Hofman, R.: SmartMonitor: recent progress in the development of an innovative visual surveillance system. Journal of Theoretical and Applied Computer Science 6(3), 28–35 (2012)

4. Horprasert, T., Harwood, D., Davis L. S.: A robust background subtraction and shadow detection. In: Proceedings of the Asian Conference on Computer Vision (2000)

5. Frejlichowski, D.: Automatic Localisation of Moving Vehicles in Image Sequences Using Morphological Operations. In: Proceedings of the 1st IEEE International Conference on Information Technology, Gdańsk 2008, pp. 439–442 (2008)
6. Wang, W., Chen, D., Gao, W., Yang, J.: Modeling Background from Compressed Video. IEEE Transactions on Circuits and Systems for Video Technology 5, 670–681 (2008)

7. Piccardi, M.: Background Subtraction Techniques: A Review. IEEE International Conference on Systems, Man and Cybernetics 4, 3099–3104 (2005)

8. Zivkovic, Z.: Improved Adaptive Gaussian Mixture Model for Background Subtraction. In: Proceedings of the 17th International Conference on Pattern Recognition, vol. 2, pp. 28–31 (2004)

9. Stauffer, C., Grimson, W. E. L.: Adaptive background mixture models for real-time tracking. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2–252 (1999)

10. Sen-Ching, S. C. S., Kamath, C.: Robust Techniques for Background Subtraction in Urban Traffic Video. In: Bhaskaran, V., Panchanathan, S. (eds.) Visual Communications and Image Processing, vol. 5308, pp. 881–892 (2004)

11. Javed, O., Shafique, K., Shah, M.: A Hierarchical Approach to Robust Background Subtraction Using Color and Gradient Information. Workshop on Motion and Video Computing, 22–27 (2002)

12. Kaewtrakulpong, P., Bowden, R.: An Improved Adaptive Background Mixture Model for Real-Time Tracking with Shadow Detection. In: Proceedings of the 2nd European Workshop on Advanced Video Based Surveillance Systems, Computer Vision and Distributed Processing (2001)

13. Forczmański, P., Seweryn, M.: Surveillance Video Stream Analysis Using Adaptive Background Model and Object Recognition. In: Bolc, L. et al. (eds.) ICCVG 2010, Part I. LNCS, vol. 6374, pp. 114–121. Springer, Heidelberg (2010)