# Precision Security: Integrating Video Surveillance with Surrounding Environment Changes

Authors: Wenfeng Wang; Xi Chen; Guiwei Zhang; Jing Qian; Peng Wei; Boqian Wu; Hongwei Zheng
Journal: Complexity
 (2018)

---

## Abstract

Video surveillance plays a vital role in maintaining the social security although, until now, large uncertainty still exists in danger understanding and recognition, which can be partly attributed to intractable environment changes in the backgrounds. This article presents a brain-inspired computing of attention value of surrounding environment changes (EC) with a processes-based cognition model by introducing a ratio valueλ of EC-implications within considered periods. Theoretical models for computation of warning level of EC-implications to the universal video recognition efficiency (quantified as time cost of implication-ratio variations from λk to λk+1, k=1,2,…) are further established. Imbedding proposed models into the online algorithms is suggested as a future research priority towards precision security for critical applications and, furthermore, schemes for a practical implementation of such integration are also preliminarily discussed.


---

## Body

## 1. Introduction

Surveillance plays a vital role in maintaining social security and protecting infrastructure facilities of a country [1, 2]. But until now, there are still considerable uncertainties associated with danger understanding and recognition, especially for engineering-critical applications [3–5], which can be partly attributed to implications of environment conditions to video recognition efficiency of the surveillance system. It has been demonstrated that suitable model parameters in online algorithms and difficulty level of object detection tasks in different environments can be much different [6].Surrounding environment changes as particular changes in backgrounds are also responsible for some significant but still unresolved issues in objects recognition and tracking [7]. Because the backgrounds cannot be well-characterized in uncontrolled environments changes, the surveillance video recognition becomes more intractable [8]. Recognition of objects, accidents, and behaviors in dynamic environments is still a great challenge in video surveillance [9], which should be carried out through objects detection, motion tracking and analyses, and understanding and recognition of other details with robust and efficient algorithms. Environments changes are so rich and varied that an online algorithm with universal significance is demanded towards the effective dangers detection and warning in dynamic environment changes [10–17].Numerous algorithms have been developed to tackle video recognition challenges in various environments; however, a full understanding of environmental implications to video recognition efficiency demands learning models with universal significance (ignoring uncontrolled differences in real scenarios) [18–27]. That is the essential reason why the current online algorithms, even for latest algorithms, for example, the latest models for tackling crowd segmentation for the high-dimensional, large-scale anomaly detection, still encounter considerable uncertainties [23, 24]. How to evaluate and compute the regulated attention in implications of the surrounding environment changes and, furthermore, how to define the warning level of EC-implications to video recognition efficiency should be research priorities towards precision security in intelligent surveillance [21–27].It has been widely recognized that video surveillance should consider the implications of surrounding environments changes to video surveillance, but until now, there are still no models for a universal evaluation of EC-implications to video recognition efficiency [4, 12–27]. To solve the unresolved issues associated with uncontrolled EC-implications, various novel optimization models were proposed and further applied in current learning systems [13–15]. Robustness and efficiency of some online algorithms in tackling special EC-implications in special scenario were validated in a series of previous studies although, until now, universal models for computation of the attention value and warning level of EC-implications to video recognition efficiency remain unaddressed and, hence, an emergent issue is improving the current surveillance systems [16, 17].Objectives in this study are(1) to present a brain-inspired computing of warning level of the implications of surrounding environment changes to video recognition efficiency, (2) to model brain cognition processes and establish theoretical models for precision computation of attention value of EC, and (3) to highlight necessity of introducing proposed models in critical applications.


## 2. Preliminary Formulation

A conceptual framework of precision security to integrate video surveillance with EC is shown in Figure1. Danger detection in EC-implications is of great complexity because of features diversity. Precision security aims to present a better understanding of EC-implications to danger detection efficiency in sensitive areas and allows us to consider not only “who are dangerous” but also “who are in danger” and to reduce uncertainties in uncontrolled and complicated real scenarios [28–31].Figure 1
A conceptual framework of precision security with four real scenarios as examples, (a) smog (captured by an android camera), (b) sandstorm (captured by a mobile phone), (c) blizzard (with videos collected from a drone), and (d) truck exhaust (with videos collected by a driving recorder).Brain cognition of EC-implications can be approached in four processes, data acquirement, classification, computation, and inference. Throughout the paper, the original, classified, computed, and inferred data, respectively, are denoted by EC1, EC2, EC3, and EC4. Obviously, ECi generates ECi+1  (i=1,2,3). To reduce uncertainty, assume that only EC3 and EC4 contribute to dispelling the EC-implications and generate regulated attention-effective data (denoted by S), which is generated from determination by EC4 (with a contribution θ3) and a part of EC3 (with a contribution θ4).Denote byI(k) amounts of newly generated effective data in the k-th brain learning period, k = 1, 2, 3, …. Denote Ai(t) as the amounts of ECi at the t-th frame and let Ai0(k), Ai1(k) be ECi at the beginning and end of the k-th period, respectively, i = 1, 2, 3, 4, k = 1, 2, 3, …. Assume that the average efficiency of data exploitation is r and employ a function δ(x)=1/(1+x) to estimate EC1 loss. Let si be degree of ECi importance and βi be the ECi contributions to S, i=1,2,3,4; it is clear that s1=s2=0,β1=β2=0,β3<β4=1. During the k-th learning period (with length Tk), define the theoretical quantification of attention value of EC-implications as the amounts of S and define(1)fkt=0,λk≤t≤T,1,0<t<λk,where λk can be interpreted as EC attention-time ratio in the k-th learning period, k=1,2,….Based on the performance of a rapid DL (deep learning) method, YOLO, which is one of the most efficient algorithms for objects detection, classification, and tracking [32–36], such implications of EC to video surveillance and the attention value and warning level are displayed in Figure 2.Figure 2
Unneglectable surrounding environment changes (EC) with various EC-implications in video recognition, objects vague (a1 and b1), occlusion (a2 and b2), or dummy (c1 and c2).Obviously, attention-time ratio of EC is reduced in regulated attention and EC-warning level (denoted byα) is measured by corresponding time cost. Throughout the paper, computation of α is formulated as evaluation of time cost in implication-ratio changes from λk to λk+1, k = 1, 2, ….


## 3. Theoretical Analyses

Nonlinear functional analyses were confirmed suitable for solving the real scenario analyses and, exactly, multistage approach has been widely employed in simulating disaster responses [37–42]. But dangers of understanding and recognition in precision security are worthy of reconsideration to dispel EC-implications, utilizing determined EC-attention value and warning level for such implications. Recall that brain cognition of EC-implications can be theoretically approached in four processes and hence, correspondingly, the formulated problem should be resolved in a four-stage approach [40–46].


### 3.1. Attention Value of EC

Brain-inspired approach to attention value and warning level of EC are shown in Figure3, where the EC-implications are manifested as an evolution of attention value and warning level. Such approach is independent of EC-types and hence it has universal significance. Regulated attention in brain-inspired data mining approach for behavior, accidents, and emotion understanding can be carried out through the whole video sampling, training, and recognition processes [47, 48].Figure 3
Brain-inspired evolution of attention value and warning level of environment changes, the attention values of EC-implications and behaviors/accidents are represented by the circles size and text color, respectively, while the warning level of EC-implications is represented by the arrows color.First, we have(2)Akit=Ai0ke-rt,i=1,2,Akjt=Aj0ke-r+fktβjt,j=3,4,which imply(3)A11k=A10ke-r,A21k=A20ke-r,A31k=A30ke-r-λkβ3,A41k=A40ke-r-λkβ4.Suppose that EC3 can fully convert to EC4; we obtain(4)A20k=A11k-1,A30k=A21k-1,A40k=A31k-1+A41k-1,A10k=δIk-1Ik-1.Letλk=λ, k=1,2,…. From (2)–(4) and preliminary formulation, we have (5)Ik-1=∫0λA3t+A4t1-λdt=1-e-λ3r+β31-λ1-r+β3A30k+1-e-λ4r+β41-λ1-r+β4A40k.Therefore, theoretical quantification of S (i.e., the attention value of EC-implications) is (6)S=∫0λs3β3A3t+s4β4A4tdt=s3β31-r+β31-e-λ3r+β3A30k+s4β41-r+β41-e-λ4r+β4A40k=s3θ3+s4θ4.


### 3.2. Determined Warning Level

It remains to determine warning level of EC-implications. To reduce time complexity of learning periods for the EC-universal significance, analyses can be divided into two cases:(1) time cost in different learning periods is independent or (2) considering periods are mutually dependent.Within a single learning period, if EC evolution rate is fixed (denoted byxr), then we have (7)dλtdt=λt•xr.Letλ(0)=λ0; we have(8)λt=λ0•exrt.Taking into account the variation ofxr within this period, for example, let xr=k(K-λ(t)); we have (9)dλtdt=λt•k•K-λt,and hence(10)λt=λ0•Kλ0+K-λ0•e-kKt.For a video withn learning periods, let xri=ki(Ki-λi(t)); we have(11)dλitdt=λit•ki•Ki-λit,λi0=λi0,i=1,2,…,n.The solution of (11) is(12)λit=λi0•Kiλi0+Ki-λi0•e-kiKit,i=1,2,…,n.Equivalently, we have(13)Ki-λi0λite-kiKit=Ki-λitλi0,i=1,2,…,n.To simplify the representation of (13), define the following:The Time-Parameters Matrices
(14)
k
=
d
i
a
g
k
1
,
k
2
,
…
,
k
n
,
K
=
d
i
a
g
K
1
,
K
2
,
…
,
K
n
.The Original Status Matrix
(15)
λ
0
=
d
i
a
g
λ
10
,
λ
20
,
…
,
λ
n
0
.The Dynamic Functions Matrix
(16)
λ
t
=
d
i
a
g
λ
1
t
,
λ
2
t
,
…
,
λ
n
t
.We obtain the matrix form of (13):(17)K-λ0λte-kKt=K-λtλ0.Further considering relationship between surveillance videos, let(ki,Ki)=(ui(k1),ui(K1)); then(18)λit=λi0•uiK1λi0+uiK1-λi0•e-uik1uiK1t,u1=1,i=1,2,…,n.The symmetric form of (17) is(19)uiK1-λi0λite-uik1uiK1t=uiK1-λitλi0,i=1,2,…,n.Definingu=diag[u1,u2,…,un], λ0=diag[λ10,λ20,…,λn0], λ(t)=diag[λ1(t),λ2(t),…,λn(t)], we obtain(20)uK1-λ0λte-uk1uK1t=uK1-λtλ0,where ui is the correlative function of the i-th video in the consider security system, i=1,2,…,n.Finally, EC-warning level can be computed as time cost fromλj to λj+1, j=1,2,…. Regulated attention can be theoretically implemented in multidata fusion, learning, and modelling. Region of interest (ROI) or pedestrians of interest (POI) corresponds to GIS-data, including time, place, and EC through Internet of things applicable for real scenarios, as seen in Figure 4. It is worth noting that the 3D stereo generated from a 2D video sequence is advantageous to highlight EC evolution and therefore is also advantageous to determine length of learning periods.Figure 4
Implementation of the brain mechanisms of regulated attention and the corresponding determination of attention value and warning level, where a simple algorithm [28] is employed to generate a 3D stereo from a 2D video sequence and highlight evolution of environment changes.


## 3.1. Attention Value of EC

Brain-inspired approach to attention value and warning level of EC are shown in Figure3, where the EC-implications are manifested as an evolution of attention value and warning level. Such approach is independent of EC-types and hence it has universal significance. Regulated attention in brain-inspired data mining approach for behavior, accidents, and emotion understanding can be carried out through the whole video sampling, training, and recognition processes [47, 48].Figure 3
Brain-inspired evolution of attention value and warning level of environment changes, the attention values of EC-implications and behaviors/accidents are represented by the circles size and text color, respectively, while the warning level of EC-implications is represented by the arrows color.First, we have(2)Akit=Ai0ke-rt,i=1,2,Akjt=Aj0ke-r+fktβjt,j=3,4,which imply(3)A11k=A10ke-r,A21k=A20ke-r,A31k=A30ke-r-λkβ3,A41k=A40ke-r-λkβ4.Suppose that EC3 can fully convert to EC4; we obtain(4)A20k=A11k-1,A30k=A21k-1,A40k=A31k-1+A41k-1,A10k=δIk-1Ik-1.Letλk=λ, k=1,2,…. From (2)–(4) and preliminary formulation, we have (5)Ik-1=∫0λA3t+A4t1-λdt=1-e-λ3r+β31-λ1-r+β3A30k+1-e-λ4r+β41-λ1-r+β4A40k.Therefore, theoretical quantification of S (i.e., the attention value of EC-implications) is (6)S=∫0λs3β3A3t+s4β4A4tdt=s3β31-r+β31-e-λ3r+β3A30k+s4β41-r+β41-e-λ4r+β4A40k=s3θ3+s4θ4.


## 3.2. Determined Warning Level

It remains to determine warning level of EC-implications. To reduce time complexity of learning periods for the EC-universal significance, analyses can be divided into two cases:(1) time cost in different learning periods is independent or (2) considering periods are mutually dependent.Within a single learning period, if EC evolution rate is fixed (denoted byxr), then we have (7)dλtdt=λt•xr.Letλ(0)=λ0; we have(8)λt=λ0•exrt.Taking into account the variation ofxr within this period, for example, let xr=k(K-λ(t)); we have (9)dλtdt=λt•k•K-λt,and hence(10)λt=λ0•Kλ0+K-λ0•e-kKt.For a video withn learning periods, let xri=ki(Ki-λi(t)); we have(11)dλitdt=λit•ki•Ki-λit,λi0=λi0,i=1,2,…,n.The solution of (11) is(12)λit=λi0•Kiλi0+Ki-λi0•e-kiKit,i=1,2,…,n.Equivalently, we have(13)Ki-λi0λite-kiKit=Ki-λitλi0,i=1,2,…,n.To simplify the representation of (13), define the following:The Time-Parameters Matrices
(14)
k
=
d
i
a
g
k
1
,
k
2
,
…
,
k
n
,
K
=
d
i
a
g
K
1
,
K
2
,
…
,
K
n
.The Original Status Matrix
(15)
λ
0
=
d
i
a
g
λ
10
,
λ
20
,
…
,
λ
n
0
.The Dynamic Functions Matrix
(16)
λ
t
=
d
i
a
g
λ
1
t
,
λ
2
t
,
…
,
λ
n
t
.We obtain the matrix form of (13):(17)K-λ0λte-kKt=K-λtλ0.Further considering relationship between surveillance videos, let(ki,Ki)=(ui(k1),ui(K1)); then(18)λit=λi0•uiK1λi0+uiK1-λi0•e-uik1uiK1t,u1=1,i=1,2,…,n.The symmetric form of (17) is(19)uiK1-λi0λite-uik1uiK1t=uiK1-λitλi0,i=1,2,…,n.Definingu=diag[u1,u2,…,un], λ0=diag[λ10,λ20,…,λn0], λ(t)=diag[λ1(t),λ2(t),…,λn(t)], we obtain(20)uK1-λ0λte-uk1uK1t=uK1-λtλ0,where ui is the correlative function of the i-th video in the consider security system, i=1,2,…,n.Finally, EC-warning level can be computed as time cost fromλj to λj+1, j=1,2,…. Regulated attention can be theoretically implemented in multidata fusion, learning, and modelling. Region of interest (ROI) or pedestrians of interest (POI) corresponds to GIS-data, including time, place, and EC through Internet of things applicable for real scenarios, as seen in Figure 4. It is worth noting that the 3D stereo generated from a 2D video sequence is advantageous to highlight EC evolution and therefore is also advantageous to determine length of learning periods.Figure 4
Implementation of the brain mechanisms of regulated attention and the corresponding determination of attention value and warning level, where a simple algorithm [28] is employed to generate a 3D stereo from a 2D video sequence and highlight evolution of environment changes.


## 4. Simulation and Discussion

Our proposed models in the present study are learning models with universal significance (ignoring uncontrolled differences in real scenarios), which aim to establish theoretical framework of the environmental implications to video recognition efficiency. It will serve for a universal evaluation of EC-implications to video recognition efficiency. Numerous algorithms have been developed to tackle video recognition challenges in various environments, but it is still difficult to describe the time complexity of learning periods. This can be largely attributed to the complexity of video recognition issues. Even for a given issue, it is not easy to determine learning periods for different EC-scenarios. Generally, attention-time ratio of EC is reduced in regulated attention and EC-warning level can be measured by corresponding time cost in reducing the attention-time ratio of EC. So we formulate the parameterα as the time cost in implication-ratio changes from λk to λk+1, k=1,2,…. For detailed analysis on the time complexity, some examples of learning periods for video detection and tracking in different surveillance scenarios are presented in Figure 5. One possible solution to treat the time complexity is to imbed proposed models into online algorithms in critical applications, utilizing these newly added examples and evidences.Figure 5
Examples of learning periods for video recognition in different surveillance scenarios for detailed analysis on the time complexity, under implications of smog (in the blue-rectangles, captured by an android camera), sandstorm (in the green rectangles, captured by a webcam on mobile), blizzard (in those yellow rectangles with videos collected from a drone), and truck exhaust (in the red rectangles with videos collected by a driving recorder).Because of time complexity of learning periods, we give EC-attention values for simulation, ten videos with given EC-attention values in Table1. Equations (17)–(20) are employed to simulate brain-inspired computing of corresponding EC-warning level.Table 1
Attention values of ten videos with unneglectable EC.
Period 1
Period 2
Period 3
Period 4
Period 5
30.21
2.68
1.01
0.81
0.33
29.88
1.62
1.43
0.76
0.31
26.66
1.23
1.29
0.71
0.28
25.12
1.28
1.26
0.65
0.27
23.33
1.24
1.11
0.54
0.26
13.33
1.18
1.12
0.48
0.21
0.32
1.02
1.01
0.39
0.22
0.18
0.98
0.96
0.35
0.23
0.17
0.86
0.91
0.31
0.23
0.16
0.79
0.68
0.27
0.25Ignoring the association among ten surveillance videos, from (17) and (18), the EC-warning levels from λi to λi+1 are α1 = 0.8868, α2 = 0.1363, α3 = 1.5691, and α4 = 0.9220, respectively, i=1,2,3,4. Taking into account the association among ten surveillance videos, utilizing (19) and (20) and letting k1=0.4503, K2=-1.6369 and finding a suitable association function (here u(x)=ax+blnx), EC-warning levels from λi to λi+1 are α1 = 0.4096, α2 = 0.0984, α3 = 0.6314, and α4 = 0.9220, respectively, i=1,2,3,4.Characterizing EC-warning level and the implied dangers is helpful for learning how well can potential dangers be detected by video surveillance in changing environments, especially in unmanned driving, where one major bottleneck is finding effective and efficient algorithms for the danger detection and caution, majorly due to lack of adaptive attention in utilized learning systems [49–51]. Numerous issues remain unresolved, a part of which are resulted from poorly understood EC-implications [52–58]. Brain-inspired modelling approach to such implications in the present study majorly depends on amounts of attention data and length of attention time, ignoring the differences in real scenarios. Therefore, the proposed models have universal significance for its critical applications. It is therefore necessary to consider integration of proposed models with the online surveillance algorithms towards precision security [59–61]. Such precision security can be a great challenge because that performance degradation of video recognition efficiency in critical environments has been demonstrated in some previous studies [6, 17, 21, 35].For special scenarios when EC-implications are not significant, integration of our models with online algorithms is not necessary. Computation can be largely simplified in special applications. Taking the lane detection as an example, the biological principles are to detect and recognize a line, which can work well even if the lanes are partly missing [62–64], as seen in Figure 6.Figure 6
An example does not normally require integration with proposed models; truck exhaust as a regional environmental change has no significant implications to efficiency of the lanes detection and warning of conflict danger. Lanes detection is always working well although a part of detected lane is temporally missing (highlighted by the red caution-texts “Right Departure”) within the period of right departures and a robust and efficient warning of conflict danger (highlighted by the red caution-texts “Conflict Alert”) works well simultaneously during the period (Frame #156 to #164).For complex applications, however, imbedding proposed models in current security systems becomes necessary, such as compressive sensing for sparse tracking [18] (it can be improved as locally compressive sensing within ROI), VIBE algorithm for real-time object detection from a moving camera [19], Adaboost algorithm for noise-detection in ROI [20], optical flow for robots’ recognition of environments [21], SVM clustering for accidents classification [22], deep learning algorithms for anomaly detection, crow analysis, and hierarchical tracking within ROI [23–27]. Objects understanding and detection in dynamic environment changes are usually based on the adaptive background subtraction and other objects recognition methods [17, 21, 35, 65–68]. A preliminary scheme for the practical integration of proposed models with these algorithms is presented in Figure 7, where smog as a global environmental change has significant implications to video behaviors recognition and loitering detection within a hovering period of two persons; only half of hovering behaviors is detected; only one person is red-highlighted and the other person is always in a green rectangle, indicating the degradation of video surveillance efficiency within the considered periods under any real challenging scenarios. It is worth noting that the proposed models have analytic solutions and the time cost in each iteration is much shorter than the time cost of any video recognition algorithms. Therefore, imbedding the proposed models in current security systems for critical applications is not only necessary but also feasible; proposed models can work well with any online algorithms without a great loss in surveillance efficiency.Figure 7
An example demands integration of proposed models with other online algorithms; smog as a global environmental change has significant implications to loitering detection within a hovering period of two persons, where only half warning of hovering behaviors is detected (highlighted by the red rectangles). A preliminary scheme for practical integration is subsequently presented.


## 5. Conclusion

Despite previous studies on algorithms for video surveillance in various environments, there are still considerable uncertainties in objects detection, classification, and tracking. Understanding and recognition of implications of surrounding environment changes to surveillance efficiency are still very limited. Brain-inspired modelling approach to such implications in the present study majorly depends on the amounts of attention data and attention time, ignoring difference in real scenarios. Therefore, proposed models represent biological principles of computational intelligence and have universal significance for its practical integration with online algorithms. Nevertheless, a full understanding of complexity of learning periods for different EC-scenarios is still necessary. This is also a next research priority towards a universal evaluation of implications of the surrounding environments changes to video recognition efficiency.



---

*Source: 2959030-2018-02-08.xml*