# The  Complex Action Recognition via the Correlated Topic Model

Authors: Hong-bin Tu; Li-min Xia; Zheng-wu Wang
Journal: The Scientific World Journal
 (2014)

---

## Abstract

Human complex action recognition is an important research area of the action recognition. Among various obstacles to human complex action recognition, one of the most challenging is to deal with self-occlusion, where one body part occludes another one. This paper presents a new method of human complex action recognition, which is based on optical flow and correlated topic model (CTM). Firstly, the Markov random field was used to represent the occlusion relationship between human body parts in terms of an occlusion state variable. Secondly, the structure from motion (SFM) is used for reconstructing the missing data of point trajectories. Then, we can extract the key frame based on motion feature from optical flow and the ratios of the width and height are extracted by the human silhouette. Finally, we use the topic model of correlated topic model (CTM) to classify action. Experiments were performed on the KTH, Weizmann, and UIUC action dataset to test and evaluate the proposed method. The compared experiment results showed that the proposed method was more effective than compared methods.


---

## Body

## 1. Introduction

Automatic recognition of human actions from video is a challenging problem that has attracted the attention of researchers in recent decades. It has applications in many areas such as entertainment, virtual reality, motion capture, sport training [1], medical biomechanical analysis, ergonomic analysis, human-computer interaction, surveillance and security, environmental control and monitoring, and patient monitoring systems. Human complex action recognition is an important research field of the action recognition. Among various obstacles to human complex action recognition, one of the most challenging is to deal with “self-occlusion”, where one body part occludes another. Adaptive self-occlusion behavior recognition has been traditionally tackled by applying statistical prediction and inference methods [2]. Unfortunately, basic numerical methods have proved to be insufficient when dealing with complex occlusion scenarios that present interactions between objects (e.g., occlusions, unions, or separations), modifications of the objects (e.g., deformations), and changes in the scene (e.g., illumination). These events are hard to manage and frequently result in tracking errors, such as track discontinuity and inconsistent track labeling.Our models are motivated by the recent success of “bag-of-words” representations for object recognition problems in computer vision. The common paradigm of these approaches consists of extracting local features from a collection of images, constructing a codebook of visual words by vector quantization, and building a probabilistic model to represent the collection of visual words. While these models of an object as a collection of local patches are certainly not “correct” ones, (e.g., they only model a few parts of objects and often ignore many structures), they have been demonstrated to be quite effective in object recognition tasks [3–5]. For the recognition approaches, some structures have been lost by moving to this representation. However, this representation is much simpler than the one that explicitly models temporal structures. there has been previous work (e.g., Yamato et al. [6], Bobick and Wilson [7], and Xiang and Gong [8]) that tries to model the full dynamics of videos using sophisticated probabilistic models (e.g., hidden Markov models and dynamic Bayesian networks). Li and Perona [9] use a variant of LDA for natural scene categorization. Sivic et al. [10] perform unsupervised learning of object categories using variants of the pLSA model. In this models, the “words” correspond to local patches extracted by interest point operators, and the “topics” correspond to the different object categories. Fergus et al. [11] extend pLSA to incorporate spatial information in a translation and scale-invariant manner and apply them to learn object categories from Google’s image search. Wang et al. [12] designs a simultaneous classification and annotation framework which extends from LDA and allows image feature and text word share the same dimensional topic space. Putthividhya et al. [13] propose a more general and flexible annotation model which allows different topic spaces for image feature and text word. Bissacco et al. [14] use LDA for human detection and pose classification. The “visual words” in their model are vector quantizations of histogram of oriented gradients in the training images. Niebles et al. [15] recently demonstrate some impressive results on unsupervised learning of human action categories using pLSA and LDA models for human action recognition. Wong et al. [16] adopt pLSA models to capture both semantic (content of parts) and structural (connection between parts) information for recognizing actions and inferring the locations of certain actions.Optical flow-based action detection methods are well known [17–19]. Efros et al. [20] recognize human actions at a distance in low resolution by introducing a motion descriptor based on optical flow measurements. Ahmad and Lee [21] propose a view independent recognition method by using the C-artesian component of optical flow velocity and human body shape feature vector information. Usually optical flow is used with other features, because it is noisy and inconsistent between frames [22]. Optical flow histograms have also been used to analyze the motion of individual behavior videos. The time series of histogram of optical flow has been modeled as a nonlinear dynamical system using Binet-Cauchykernels in [23]. However, this approach cannot deal with large motion, for example, a rapid move across frames.In order to overcome the shortcomings mentioned above, we propose an adaptive self-occlusion action recognition method that not only estimates the occlusion states of body parts but also recognizes the occlusion behavior. Firstly, the Markov random field was used to represent the occlusion state of human body parts. Secondly, the structure from motion (SFM) is used for reconstructing the missing data of point trajectories. Then, we can extract the key frame based on motion feature from optical flow and the ratios of the width and height are extracted by the human silhouette. Finally, we use the topic model of correlated topic model (CTM) for action recognition. Experiments were performed on the KTH, Weizmann, and UIUC action dataset to test and evaluate the proposed method. The experiment results have shown that the proposed method is effective in action recognition.The reminder of this paper is organized as follows. Section2 presents the adaptive occlusion state estimation by Markov random field (MRF). In Section 3, we reconstruct the missing data of point trajectories by SFM (structure from motion). Section 4 explains feature representation. Section 5 explains algorithm of action models and the design of the classifier. Section 6 explains the results and analysis of the proposed approach. Finally, we conclude the paper in Section 7.


## 2. The Adaptive Occlusion State Estimation

The human body is divided into 15 key points, namely, 15 joint points representing the human body structure (torso, pelvis, left upper leg, left lower leg, left foot, rightupper leg, right lower leg, right foot, left upper arm, left lower arm, left hand, rightupper arm, right lower arm, right hand, and head) [24], which represent the human body behavior. In order to calculate the observation, spatial relations, and the motion relationship, we use Markov random field (MRF), which can determine the occlusion positions of the body joints.In this paper, we use a state variable in the Markov random field (MRF) for representing the self-occlusion relationship between body parts. The MRF is a graphG
=
(
V
,
E
), where V is the set of nodes and E was the set of edges. The graph nodes V represent the state of a human body part and graph edges E model the relationships between the parts [25]. The probability distribution over this graph was specified by the set of potentials defined over the set of edges. The MRF structural parameters [24, 25] are defined as follows:X
i
=
(
x
i
,
y
i
,
z
i
): the ith joint point coordinates; X
=
{
X
1
,
X
2
,
…
,
X
15
}: extract the key points of the body 15, γ
(
X
i
) (i
≤
15): the ith joints visible parts, and this parameter is used to determine occlusion relation between nodes. When occlusion occurred, trajectories intersected between
(1)
X
i
(
X
i
(
x
i
,
y
i
,
z
i
)
)
,
X
j
(
X
j
(
x
j
,
y
j
,
z
j
)
)
;
Λ
=
{
Λ
i
j
}
(
i
≤
15
,
j
≤
15
): the occlusion relation among the 15 body joints. When Λ
i
,
j
=
0, the ith and jth joints are not occluded. When Λ
i
,
j
=
1, the ith occluded jth. When Λ
i
,
j
=
-
1, the jth occluded ith; λ
i
=
{
λ
1
,
…
,
λ
15
}: the ith occluded joints node.We apply the MRF model presented in [24, 25] to optimally estimate potential of kinematic relationship and similar to [26] for calculating three potential functions in video activity analysis. The potential of kinematic relationship and three potential functions were defined as follows.(1)
Kinematic relationship is calculated(2)
ψ
i
j
K
(
X
i
,
X
j
)
=
N
(
d
(
x
i
,
x
j
)
;
μ
k
,
δ
K
)
f
(
θ
i
,
θ
j
)
.
This function indicates the position of two adjacent joints and the angles among joints.d
(
x
i
,
x
j
) is the Euclidean distance between two adjacent joints. N
(
·
) is the normal distribution with μ
k
=
0 and standard deviation δ
K.
(2)
The Potential Functions
The observation potential function is(3)
ϕ
i
(
I
,
X
i
;
∧
i
)
=
ϕ
i
C
(
I
,
X
i
;
∧
i
)
+
ϕ
i
E
(
I
,
X
i
;
∧
i
)
,
whereϕ
i
(
I
,
X
i
;
∧
i
) is the potential of observation, ϕ
i
C
(
I
,
X
i
;
∧
i
j
) is the potential of the color, and ϕ
i
E
(
I
,
X
i
;
∧
i
) is the potential of the edge, I: input image.The potential of the color is(4)
ϕ
i
C
(
I
,
X
i
;
∧
i
j
)
=
ϕ
i
C
visible
(
I
,
X
i
;
∧
i
j
)
+
ϕ
i
C
occluded
(
I
,
X
i
;
∧
i
j
)
,
where the first term is X
i of probability of occurrence of color in the visible area and the second term is for the occluded area as follows:ϕ
i
C
visible
(
I
,
X
i
;
∧
i
j
): the motion state of X
i (the ith body joint) in the viewing area;
ϕ
i
C
occluded
(
I
,
X
i
;
∧
i
j
): the motion state of X
i (the ith body joint) in the occluded area.The visible term is formulated as(5)
ϕ
i
C
visible
(
I
,
X
i
;
λ
i
)
=
∏
u
∈
(
γ
(
X
i
)
-
(
γ
(
X
i
)
∩
γ
(
X
j
)
)
)
P
C
(
I
u
)
=
∏
u
∈
(
γ
(
X
i
)
-
(
γ
(
X
i
)
∩
γ
(
X
j
)
)
)
P
(
I
u
∣
foreground
)
P
(
I
u
∣
background
)
P
(
I
u
∣
foreground
) and P
(
I
u
∣
background
) are the distributions of the color of pixel u given the foreground and background:
(6)
ϕ
i
C
occluded
(
I
,
X
i
;
λ
i
)
=
∏
u
∈
(
γ
(
X
i
)
∩
γ
(
X
j
)
)
[
z
i
(
I
u
)
+
(
1
-
z
i
(
I
u
)
)
P
C
(
I
u
)
]
and z
i
(
I
) is calculated as follows:
(7)
z
i
(
I
)
=
1
N
∑
(
ϕ
j
C
(
I
u
,
x
j
(
t
)
;
λ
j
)
)
;
u
∈
(
γ
(
X
i
)
∩
γ
(
X
j
)
): the occlusion area is determined by the calculated overlapping region of X
i and X
j and N is the sum of all occlusion nodes. f
(
θ
i
,
θ
j
)
=
1, T
lower
≤
θ
i
-
θ
j
≤
T
upper, where T
lower and T
upper are the lower and upper bound of motion area between X
i and X
j defined by kinesiology.Finally, the potential of temporal relationship is calculated as follows:(8)
ψ
i
T
(
X
i
t
,
X
i
t
-
1
)
=
p
(
X
i
t
-
X
i
t
-
1
;
μ
i
,
Σ
i
)
,
where μ
i is the dynamics of X
i at the previous time step and Σ
i is a diagonal matrix with a diagonal element which is identical to |
μ
i
|, similar to a Gaussian distribution with the time.In this paper, the posterior distribution of modelX conditioned on all input images up to the current joint s structure, the current time step τ and occlusion state variable ∧
1
:
τ are
(9)
p
(
X
τ
∣
I
1
:
τ
;
∧
1
:
τ
)
=
1
Z
exp
⁡
{
-
∑
i
∈
X
1
:
τ
ϕ
i
C
(
I
,
X
i
;
λ
i
)
-
∑
i
j
∈
E
K
1
:
τ
ψ
i
j
K
(
X
i
,
X
j
)
cccccici
-
∑
i
∈
E
T
1
:
τ
,
t
∈
1
:
τ
ψ
i
T
(
X
i
t
,
X
i
t
-
1
)
}
,
where Z is a normalization constant.In other words, we putϕ
i
,
ψ
i
j
K
,
ψ
i
T into (4) and get body occluded joints positions:
(10)
X
^
t
=
arg
max
⁡
X
t
p
(
X
t
∣
I
1
:
t
;
∧
^
t
-
1
)
,
where X
t is X joint location at t time.The occluded relation among joints can be obtained by formula (2) as follows:
(11)
λ
^
i
,
j
t
=
arg
max
⁡
∧
i
j
≠
0
ϕ
i
(
I
t
,
X
^
i
t
;
∧
i
j
)
,
where X
^
i
t is X
i position at t time.Therefore, the occluded positions can be calculated by MRF at the entire time of motion.


## 3. Reconstruct the Missing Data of Point Trajectories

We use the SFM (structure from motion) model to reconstruct the missing trajectories of the occluded joints [27]. Consider a set of P
n point trajectories extracted from the n parts of human body that rigidly moves in f frames. By stacking each image trajectory in a single matrix w
n of 2
F
×
P
n, it is possible to express the global motion, which represents the complete trajectory. We define it as
(12)
w
n
=
M
n
S
n
=
[
R
1
n
t
1
n
′
⋮
R
F
n
t
F
n
′
]
[
X
1
⋯
X
n
Y
1
⋯
Y
n
Z
1
⋯
Z
n
1
⋯
1
]
,
where M
n is the 2
F
×
4 human body motion matrix and S
n is the human body contour matrix in homogenous coordinates. Each frame-wise element R
f
n, for f
=
1
,
…
,
F, is a 2
×
3 orthographic camera matrix that has to satisfy the metric constraints of the model (i.e., R
f
n
R
f
n
T
=
I
2
×
2). The 2-vector t
F
n
′ represents the 2D translation of the rigid object (in this paper, we consider the human body as a rigid object). We introduce the registered w
n
′ measurement matrix such that w
n
′
=
w
n
-
t
′
1
P
n
′
T, where 1
P
n
′
T is a vector of P
n ones and t
′
=
[
t
1
′
T
,
…
,
t
F
′
T
].In the case of missing data due to occlusions, we define the binary mask matrixG
n of size 2
F
×
P
n such that 1 represents a known entry and 0 denotes a missing one. In order to solve the components and thus the SFM problem, the equivalent optimization problem [28] can be defined as
(13)
Minimize
∥
G
n
⊗
(
w
n
-
M
n
S
n
)
∥
2
,
Subject
to
R
f
n
R
f
n
T
=
I
2
×
2
,
f
=
1
,
…
,
F
.Therefore, we can get to make up for the missing point of the trajectory. Figure1 shows the reconstructed trajectory of the missing point.Examples of reconstructed 3-D trajectories. Hip rotations for walking, jogging, and running actions in the KTH dataset [30]. (a) It shows original images. (b) It shows reconstructing the missing data of point trajectories.
(a)
(b)


## 4. Feature Representation

### 4.1. Motion Feature Extraction

The human action can be recognized in terms of hierarchical area model and relative velocity. In this paper, we use optical flow to detect the relative direction and magnitude of environmental motion observed in reference to an observer and also describe the movement of object from current image with respect to the last image. The optical flow [22, 29] equation can be assumed to hold for all pixels within a window centered at p, the local image flow (velocity) vector (
V
x
,
V
y
) must be satisfied, and we define some equations as follows.                               (14)
I
x
(
s
1
)
V
x
+
I
y
(
s
1
)
V
y
=
-
I
t
(
s
1
)
,
I
x
(
s
2
)
V
x
+
I
y
(
s
2
)
V
y
=
-
I
t
(
s
2
)
,
⋮
I
x
(
s
d
)
V
x
+
I
y
(
s
d
)
V
y
=
-
I
t
(
s
d
)
,
where s
1
,
s
2
,
…
,
s
d are the pixels inside the window and I
x
(
s
i
), I
y
(
s
i
), and I
t
(
s
i
) are the partial derivatives of the image I with respect to position x, y and time t, evaluated at the point S
i and the current time. These equations can be written in matrix form A
v
=
b, where
(15)
A
=
[
I
x
(
s
1
)
I
y
(
s
1
)
I
x
(
s
2
)
I
y
(
s
2
)
⋮
I
x
(
s
n
)
I
y
(
s
n
)
]
,
v
=
[
V
x
V
y
]
,
b
=
[
-
I
t
(
s
1
)
-
I
t
(
s
2
)
⋮
-
I
t
(
s
d
)
]
.The optical flow vector fieldV
x is further half-wave rectified into four nonnegative channels:F
x
+
,
F
x
-
,
F
y
+
,
F
y
-, so that
(16)
F
x
=
F
x
+
-
F
x
-
,
F
y
=
F
y
+
-
F
y
-
.The motion descriptors of two different frames are compared using a version of the normalized correlation.Suppose the four channels for framei of sequence A are a
1, a
2, a
3, a
4 and the four channels for frame j of sequence B are b
1, b
2, b
3, b
4; then the similarity between frame A
i and B
i is
(17)
S
(
A
i
,
B
i
)
=
∑
t
∈
T
∑
d
=
1
4
∑
x
,
y
∈
I
a
c
i
+
t
(
x
,
y
)
b
b
j
+
t
(
x
,
y
)
,
where T and I are the temporal and spatial extent of the motion descriptors. In this paper, we choose T
=
20. Therefore, we can obtain the key frames by the frames cluster. Figure 2 depicts results of key-frame extracted by the optical flow.Figure 2
Results of key-frame extracted method.Then, we extract the feature from the key frame. We assume that the overall key frame number isi, the lengths of silhouette in horizontal and vertical are h
(
i
) and v
(
i
), respectively, and their ratio is L, so we define it as
(18)
L
=
v
(
i
)
h
(
i
)
,
v
(
i
)
=
m
θ
+
(
π
/
2
)
(
0,1
)
Δ
θ
+
(
π
/
2
)
.We assume that this area has a maximum value in theθ direction.The length of the projection area in theθ
+
(
π
/
2
) direction is defined as the area width:
(19)
h
(
i
)
=
max
⁡
θ
{
m
θ
(
0,1
)
Δ
θ
}
min
⁡
θ
{
m
θ
(
0,1
)
Δ
θ
}
,
where Δ is the distance of pixels [31], B is the binary image of the human body area, m
θ
(
0,1
) is the region B in the perpendicular direction to the parallel lines intersecting the number of lines in group, and Δ
θ is the distance between the scan lines. In this paper, we use the eight search directions as shown in Figure 3. Therefore, distances among search lines, respectively, are
(20)
Δ
0
=
Δ
π
/
2
=
Δ
,
Δ
π
/
4
=
Δ
3
π
/
4
=
Δ
2
Δ
arctg
(
1
/
2
)
=
Δ
arctg
2
=
Δ
(
π
/
2
)
+
arctg
(
1
/
2
)
=
Δ
(
π
/
2
)
+
arctg
2
=
Δ
5
.Figure 3
Eight searching directions in image.In order to implement the silhouette of human body area, a series of steps must be followed:(1)
using Gaussian filter to eliminate any noise in video sequence;
(2)
using filter to find the edge strength, which estimates the gradient in thex-direction and the other estimating the gradient in the y-direction;
(3)
the direction of the edge is computed using the gradient in thex and y directions;
(4)
after the edge directions are known nonmaximum suppression now has to be applied. Nonmaximum suppression is used to trace along the edge in the edge direction and suppress any pixel value (sets it equal to 0) that is not considered to be an edge. This will give a thin line in the output;
(5)
thresholdT1 is applied to the frames, and an edge has an average strength equal to T1. Any pixel in the frame that has a value greater than T1 is presumed to be an edge and is marked as such immediately and then any pixels that are connected to this edge pixel and that have a value. Figure 4 shows the silhouette of human body area.Figure 4
The silhouette of human body area.Therefore, we can obtain the body contours by this step and use the optical flow descriptor andL descriptor (the ratio of the width and height) to represent the video frames. To construct the codebook, we randomly select a subset from all the frames and compute the affinity matrix on this subset of frames, where each entry in the affinity matrix is the similarity between frame i and frame j calculated using the normalized correlation described above. Then, we run k-medoid clustering on this affinity matrix to obtain V clusters. Code-words are then defined as the centers of the obtained clusters.


### 4.2. Construct the Codebook

In the end, each video sequence is converted to the “bag of words” representation by replacing each frame by its corresponding codeword and removing the temporal information.We follow the instruction from statistical text document analysis, and each image is represented as a bag of code-words. Given a training set of images with annotation words, we use the following notation. Each image is a collection ofM visual feature code-words, denoted as v
=
{
v
1
:
M
}, where v
m is a unit-basis vector of size V
s with exactly one nonzero entry representing the index of current visual feature in the visual feature dictionary of size V
s. Similarly, for one image annotated with N words W
=
{
w
1
:
N
}, we denote each word w
n as a unit-basis vector of size V
t again with only one taking values 1 and 0 otherwise; here V
t is the word dictionary size. Therefore, a collection of D training image word pairs can be denoted as {
v
1
:
D
,
W
1
:
D
}.


## 4.1. Motion Feature Extraction

The human action can be recognized in terms of hierarchical area model and relative velocity. In this paper, we use optical flow to detect the relative direction and magnitude of environmental motion observed in reference to an observer and also describe the movement of object from current image with respect to the last image. The optical flow [22, 29] equation can be assumed to hold for all pixels within a window centered at p, the local image flow (velocity) vector (
V
x
,
V
y
) must be satisfied, and we define some equations as follows.                               (14)
I
x
(
s
1
)
V
x
+
I
y
(
s
1
)
V
y
=
-
I
t
(
s
1
)
,
I
x
(
s
2
)
V
x
+
I
y
(
s
2
)
V
y
=
-
I
t
(
s
2
)
,
⋮
I
x
(
s
d
)
V
x
+
I
y
(
s
d
)
V
y
=
-
I
t
(
s
d
)
,
where s
1
,
s
2
,
…
,
s
d are the pixels inside the window and I
x
(
s
i
), I
y
(
s
i
), and I
t
(
s
i
) are the partial derivatives of the image I with respect to position x, y and time t, evaluated at the point S
i and the current time. These equations can be written in matrix form A
v
=
b, where
(15)
A
=
[
I
x
(
s
1
)
I
y
(
s
1
)
I
x
(
s
2
)
I
y
(
s
2
)
⋮
I
x
(
s
n
)
I
y
(
s
n
)
]
,
v
=
[
V
x
V
y
]
,
b
=
[
-
I
t
(
s
1
)
-
I
t
(
s
2
)
⋮
-
I
t
(
s
d
)
]
.The optical flow vector fieldV
x is further half-wave rectified into four nonnegative channels:F
x
+
,
F
x
-
,
F
y
+
,
F
y
-, so that
(16)
F
x
=
F
x
+
-
F
x
-
,
F
y
=
F
y
+
-
F
y
-
.The motion descriptors of two different frames are compared using a version of the normalized correlation.Suppose the four channels for framei of sequence A are a
1, a
2, a
3, a
4 and the four channels for frame j of sequence B are b
1, b
2, b
3, b
4; then the similarity between frame A
i and B
i is
(17)
S
(
A
i
,
B
i
)
=
∑
t
∈
T
∑
d
=
1
4
∑
x
,
y
∈
I
a
c
i
+
t
(
x
,
y
)
b
b
j
+
t
(
x
,
y
)
,
where T and I are the temporal and spatial extent of the motion descriptors. In this paper, we choose T
=
20. Therefore, we can obtain the key frames by the frames cluster. Figure 2 depicts results of key-frame extracted by the optical flow.Figure 2
Results of key-frame extracted method.Then, we extract the feature from the key frame. We assume that the overall key frame number isi, the lengths of silhouette in horizontal and vertical are h
(
i
) and v
(
i
), respectively, and their ratio is L, so we define it as
(18)
L
=
v
(
i
)
h
(
i
)
,
v
(
i
)
=
m
θ
+
(
π
/
2
)
(
0,1
)
Δ
θ
+
(
π
/
2
)
.We assume that this area has a maximum value in theθ direction.The length of the projection area in theθ
+
(
π
/
2
) direction is defined as the area width:
(19)
h
(
i
)
=
max
⁡
θ
{
m
θ
(
0,1
)
Δ
θ
}
min
⁡
θ
{
m
θ
(
0,1
)
Δ
θ
}
,
where Δ is the distance of pixels [31], B is the binary image of the human body area, m
θ
(
0,1
) is the region B in the perpendicular direction to the parallel lines intersecting the number of lines in group, and Δ
θ is the distance between the scan lines. In this paper, we use the eight search directions as shown in Figure 3. Therefore, distances among search lines, respectively, are
(20)
Δ
0
=
Δ
π
/
2
=
Δ
,
Δ
π
/
4
=
Δ
3
π
/
4
=
Δ
2
Δ
arctg
(
1
/
2
)
=
Δ
arctg
2
=
Δ
(
π
/
2
)
+
arctg
(
1
/
2
)
=
Δ
(
π
/
2
)
+
arctg
2
=
Δ
5
.Figure 3
Eight searching directions in image.In order to implement the silhouette of human body area, a series of steps must be followed:(1)
using Gaussian filter to eliminate any noise in video sequence;
(2)
using filter to find the edge strength, which estimates the gradient in thex-direction and the other estimating the gradient in the y-direction;
(3)
the direction of the edge is computed using the gradient in thex and y directions;
(4)
after the edge directions are known nonmaximum suppression now has to be applied. Nonmaximum suppression is used to trace along the edge in the edge direction and suppress any pixel value (sets it equal to 0) that is not considered to be an edge. This will give a thin line in the output;
(5)
thresholdT1 is applied to the frames, and an edge has an average strength equal to T1. Any pixel in the frame that has a value greater than T1 is presumed to be an edge and is marked as such immediately and then any pixels that are connected to this edge pixel and that have a value. Figure 4 shows the silhouette of human body area.Figure 4
The silhouette of human body area.Therefore, we can obtain the body contours by this step and use the optical flow descriptor andL descriptor (the ratio of the width and height) to represent the video frames. To construct the codebook, we randomly select a subset from all the frames and compute the affinity matrix on this subset of frames, where each entry in the affinity matrix is the similarity between frame i and frame j calculated using the normalized correlation described above. Then, we run k-medoid clustering on this affinity matrix to obtain V clusters. Code-words are then defined as the centers of the obtained clusters.


## 4.2. Construct the Codebook

In the end, each video sequence is converted to the “bag of words” representation by replacing each frame by its corresponding codeword and removing the temporal information.We follow the instruction from statistical text document analysis, and each image is represented as a bag of code-words. Given a training set of images with annotation words, we use the following notation. Each image is a collection ofM visual feature code-words, denoted as v
=
{
v
1
:
M
}, where v
m is a unit-basis vector of size V
s with exactly one nonzero entry representing the index of current visual feature in the visual feature dictionary of size V
s. Similarly, for one image annotated with N words W
=
{
w
1
:
N
}, we denote each word w
n as a unit-basis vector of size V
t again with only one taking values 1 and 0 otherwise; here V
t is the word dictionary size. Therefore, a collection of D training image word pairs can be denoted as {
v
1
:
D
,
W
1
:
D
}.


## 5. Action Classification

In order to capture the correlation of topics, we model the hyperparameter of topic prior distribution as multivariate normal distribution instead of Dirichlet, similar to [29, 32] and the structure topics of dependencies by covariance matrix, and then we use the logistic normal function:
(21)
f
(
θ
i
e
)
=
(
exp
⁡
θ
i
∑
j
=
1
K
exp
⁡
θ
j
)
to project the multivariate normal to topic proportions for each image; here K is the topic number.Let{
μ
,
Σ
} be a K
-dimensional mean and variance matrix from normal distribution, and let topics π
1
:
k be K multi-nomials over fixed vocabulary with size V
s and let β
1
:
K be K multi-nomials over a fixed text word vocabulary with size V
t. The CTM generates an image-word pair with M image code-words and N annotation words [29, 32] from the following generative process:(1)
draw topic proportionsθ
{
μ
,
Σ
}
~
N
(
μ
,
Σ
)
(2)
for each visual featurev
m, m
∈
{
1
,
…
,
m
}
(a)
draw topic assignmentz
m
~
Mult
(
f
(
θ
)
)
(b)
draw visual featurev
m
~
Mult
(
π
z
m
)
(3)
for each textual wordw
n
,
n
∈
{
1
,
,
…
,
N
}
(a)
draw feature indexy
n
~
Unif
(
1
,
,
…
,
M
)
(b)
draw textual wordw
n
~
Mult
(
β
z
y
n
).Firstly, we generateM feature v
m from correlated topic proportions θ, conditional on the topic feature multinomial π, then for each of the N text words, one of the M features is selected and correspondingly assigned to a text word w
n, conditional on the topic word multinomial β. This model is shown as a directed graphical model in Figure 5.Figure 5
Graphical representation of the variational distribution for CTM.From the generative process of CTM, it could be learned that topic correlations are modeled and generated through the covariance matrixΣ of prior multivariate normal distribution. To learn the parameters of CTM that maximizes the likelihood of training data, we iteratively estimate the model parameters of latent variables. In this paper, the first step is to add and calculate a set of variational parameters to obtain the approximate lower-bound on likelihood of each sample. The latter step is to estimate the model parameters that maximize the log likelihood of the whole training samples. In the graphical representation of CTM in Figure 5, θ is conditional dependent on the parameter π, which leads to intractability for computing the log likelihood. The mean-field variational distribution is
(22)
q
(
θ
,
z
,
y
)
=
q
(
θ
∣
γ
,
v
)
∏
m
=
1
M
q
(
Z
m
∣
ϕ
m
)
∏
n
=
1
N
q
(
y
n
∣
λ
n
)
,
where (
γ
,
v
) is variational mean and variance of normal distribution, ϕ
m is a variational multinomial over K topics, and λ
n is a variational multinomial over codewords.LetΔ
=
{
μ
,
Σ
,
π
,
β
} denote model parameters; we bound the log-likelihood of an image-annotation pair (
v
,
w
) as follows:
(23)
log
⁡
p
(
v
,
w
∣
Δ
)
=
log
⁡
∫
θ
p
(
v
,
w
,
θ
,
z
,
y
∣
Δ
)
q
(
θ
,
z
,
y
)
q
(
θ
,
z
,
y
)
d
x
d
y
≥
E
q
[
log
⁡
p
(
v
,
w
,
θ
,
z
,
y
∣
Δ
)
]
-
E
q
[
log
⁡
q
(
v
,
w
,
θ
,
z
,
y
∣
Δ
)
]
=
ι
(
γ
,
v
,
ϕ
,
λ
;
Δ
)
,
where E
q is the expectation according to the variational distribution. Taking the log likelihood function ι
(
γ
,
v
,
ϕ
,
λ
;
Δ
) as object function, we fit these parameters with coordinate ascent to maximize the object function:
(24)
ι
=
E
q
[
log
⁡
p
(
θ
∣
μ
,
Σ
)
]
+
∑
m
E
q
[
log
⁡
p
(
z
m
∣
θ
)
]
+
E
q
[
log
⁡
p
(
v
m
∣
z
m
,
π
)
]
+
∑
m
E
q
[
log
⁡
p
(
y
m
∣
M
)
]
+
E
q
[
log
⁡
p
(
w
∣
y
m
,
z
m
,
β
)
]
-
E
q
[
log
⁡
q
(
θ
∣
γ
,
v
)
]
-
∑
m
E
q
[
log
⁡
q
(
z
m
∣
θ
)
]
-
∑
n
E
q
[
log
⁡
q
(
y
m
∣
λ
)
]
,
(25)
∂
ι
(
γ
)
∂
γ
=
-
∑
-
1
(
γ
-
μ
)
+
∑
m
=
1
M
ϕ
m
,
1
:
K
-
(
M
ς
)
exp
⁡
(
γ
+
ν
2
2
)
,
(26)
∂
ι
(
ν
i
)
∂
ν
i
2
=
-
∑
i
i
-
1
2
-
(
M
2
ς
)
exp
⁡
(
γ
+
ν
i
2
2
)
+
1
(
2
ν
i
2
)
.Then, we update variational multinomialϕ. The terms in ι with respect to ϕ
m are
(27)
ι
(
ϕ
m
)
=
∑
i
=
1
K
ϕ
m
i
(
γ
i
+
log
⁡
π
i
,
v
m
-
log
⁡
ϕ
m
i
+
∑
n
=
1
N
λ
n
m
log
⁡
β
i
,
w
n
)
,
(28)
∂
ι
(
ϕ
m
i
)
∂
ϕ
m
i
=
γ
i
+
log
⁡
π
i
,
v
m
+
∑
n
=
1
N
λ
n
m
β
i
,
w
n
-
log
⁡
ϕ
m
i
+
C
,
where ∑
i
=
1
K
ϕ
m
i
=
1 is the multinomial constrain. C is the constant.The full variational inference procedure repeats the updates of (24), (25) until (23); the object function converges.Then, we will obtain the parameter estimation. Given a collection of human action image data with annotation words{
v
d
,
w
d
}
d
=
1
D, we find the maximum likelihood estimation for parameter {
μ
,
Σ
,
π
,
β
}.We defineΔ
=
{
μ
,
Σ
,
π
,
β
}; the overall log likelihood of collection is bounded by
(29)
ι
=
∑
d
=
1
D
log
⁡
p
(
v
d
w
d
∣
Δ
)
≥
∑
d
=
1
D
ι
d
(
γ
d
,
v
d
,
ϕ
d
,
λ
d
;
Δ
)
.We maximize the lower boundary ofι, by plugging (23) into (17), and then update model parameters by setting derivation equal zero with respect to each model parameter.The terms containingπ
i
j are
(30)
ι
(
π
i
j
)
=
∑
d
=
1
D
∑
m
=
1
M
d
∑
i
=
1
K
∑
j
=
1
V
s
ϕ
d
m
i
log
⁡
π
i
,
v
m
-
C
(
∑
j
=
1
V
s
π
i
j
-
1
)
.Setting∂
ι
(
π
i
j
)
/
∂
π
i
j
=
0 leads to
(31)
π
i
j
′
∝
∑
d
=
1
D
∑
m
=
1
M
d
ϕ
d
m
i
v
d
m
j
.The terms containingβ
1
:
K are
(32)
ι
(
β
i
,
j
)
=
∑
d
=
1
D
∑
n
=
1
N
d
∑
i
=
1
K
∑
j
=
1
V
t
∑
m
=
1
M
d
ϕ
d
m
i
λ
n
m
log
⁡
β
i
,
w
n
-
C
(
∑
j
=
1
V
t
β
i
j
-
1
)
.Setting∂
ι
(
β
i
j
)
/
∂
β
i
j
=
0 leads to
(33)
β
i
j
′
∝
∑
d
=
1
D
∑
n
=
1
N
d
w
d
n
j
∑
m
=
1
M
d
ϕ
d
m
i
λ
d
n
m
.The terms containingμ are
(34)
ι
(
μ
)
=
-
(
1
2
)
∑
d
=
1
D
(
γ
d
-
μ
)
T
∑
-
1
(
γ
d
-
μ
)
.Setting∂
ι
(
μ
)
/
∂
μ
=
0 leads to
(35)
μ
′
=
(
1
D
)
∑
d
=
1
D
γ
d
.The terms containingΣ are
(36)
ι
(
Σ
)
=
∑
d
=
1
D
{
(
1
2
)
log
⁡
|
σ
-
1
|
-
(
1
2
)
Tr
⁡
[
diag
⁡
(
v
d
2
)
Σ
-
1
]
c
c
c
i
i
-
(
1
2
)
(
γ
d
-
μ
)
T
Σ
-
1
(
γ
d
-
μ
)
}
.Setting∂
ι
(
Σ
)
/
∂
Σ
=
0 leads to
(37)
Σ
′
=
(
1
D
)
∑
d
=
1
D
(
I
×
v
d
2
+
(
γ
d
-
μ
′
)
(
γ
d
-
μ
′
)
T
)
.If we obtain variational parameters{
γ
d
,
v
d
,
ϕ
d
,
λ
d
,
ς
d
}
d
=
1
D for all the training samples, we can update the model parameters {
μ
,
Σ
,
π
,
β
} by plugging them into (30), (34), and (36) until the overall likelihood in (28) converges. Therefore, We approximate the conditional distribution of code-words as follows:
(38)
p
(
w
∣
v
)
=
∑
m
=
1
M
∑
z
m
q
(
z
m
∣
ϕ
m
)
p
(
w
∣
z
m
,
β
)
.
The conditional probability p
(
w
∣
v
) can be treated as the predicted confidence score of each annotation word in word vocabulary, given the whole code-words of the unknown human behavior.


## 6. Experimental Result

### 6.1. Datasets

We test our algorithm on three datasets: the Weizmann human motion dataset [33], the KTH human action dataset [30], and the UIUC action dataset [34]. All the experiments are conducted on a Pentium 4 machine with 2 GB of RAM, using the implementation on MATLAB. The dataset and the related experimental results are presented in the following sections.The KTH dataset is provided by Schuldt which contains 2391 video sequences with 25 actors showing six actions. Each action is performed in 4 different scenarios.The WEIZMANN dataset is provided by Blank which contains 93 video sequences showing nine different people, each performing ten actions, such as run, walk, skip, jump-jack, jump-forward-on-two-legs, jump-in-place-on-two-legs, gallop-sideways, wave-two-hands, wave-one-hand, and bend.The UIUC action dataset is created by the University of Illinois at Urbana-Champaign (UIUC) in 2008 for human activity recognition. The activities are walking, running, jumping, waving, jumping jacks, clapping, jumping from sit up, raising one hand, stretching out, turning, sitting to standing, crawling, pushing up, and standing to sitting.For every dataset, 12 video sequences are taken by four subjects (out of the five) used for training and the remaining three videos for testing. The experiments are repeated five times. The performance of different methods is shown using the average recognition rate. In order to evaluate the performance of action recognition, we report the overall accuracy on three datasets.


### 6.2. Comparison

KTH Dataset. It contains six types of human actions (walking, jogging, running, boxing, hand waving, and hand clapping) performed several times by 25 subjects in four different scenarios: outdoors, outdoors with scale variation, outdoors with different clothes, and indoors. Representative frames of this dataset are shown in Figure 6(a). After process of restoring missing coordinate position, we use the proposed method, and the classification results of KTH dataset obtained by this approach are shown in Figure 7 and indicate quite a small number of videos are misclassified, particularly, the actions “running” and “handclapping” which more tend to be confused.Sample frames from our datasets. The action labels in each dataset are as follows: (a) KTH dataset: walking (a1), jogging (a2), running (a3), boxing (a4), handclapping (a5); (b) Weizmann dataset: running, walking, jumping-jack, waving-two-hands, waving-one-hand, bending; (c) UIUC action dataset: raising two hands (a1), running (a2), clapping (a3), stretching out (a4), and pushing up (a5).
(a)
(b)
(c)Figure 7
Confusion matrix for KTH dataset.The Weizmann Dataset. The Weizmann human action dataset contains 83 video sequences showing nine different people and all performing nine different actions: bending (a1), jumping jack (a2), jumping forward on two legs (a3), jumping in place on two legs (a4), running (a5), galloping sideways (a6), walking (a7), waving one hand (a8), and waving two hands (a9). The figures were tracked and stabilized by using the background subtraction masks that come with this dataset. Some sample frames are shown in Figure 6(b). The classified results achieved by this approach are shown in Figure 8.Figure 8
Confusion matrix for Weizmann dataset.The UIUC Action Dataset. This dataset consists of 532 high resolution sequences of 14 activities performed by eight actors. The activities are walking, running, jumping, waving, jumping jacks, clapping, jumping from sit-up, raising one hand, stretching out, turning, sitting to standing, crawling, pushing up, and standing to sitting. Some sample frames are shown in Figure 6(c). The classified results achieved by this approach are shown in Figure 9.Figure 9
Confusion matrix for UIUC dataset.In this paper, we identify jogging, running, walking, and boxing and compare the proposed method with the four state-of-the-art methods in the literature: Zhang and Gong [35], Gong et al. [36], and Chang et al. [37] in three datasets. As shown in Tables 1, 2, and 3, the existing methods, the low recognition accuracy because these action are not only occlusion situation are complex, but also the legs have complex beat, motion and other group actions. The proposed method can overcome these problems, and the recognition accuracy and average accuracy are higher than the comparative method.Table 1
Compared with other approaches on KTH dataset.
Method
Average recognition rate (%)
The proposed method
90.60
Zhang and Gong [35]
83.40
Gong et al. [36]
89.20
Chang et al. [37]
88.03Table 2
Compared with other approaches on Weizmann dataset.
Method
Average recognition rate (%)
The proposed method
89.20
Zhang and Gong [35]
85.30
Gong et al.  [36]
86.30
Chang et al. [37]
82.10Table 3
Compared with other approaches on UIUC dataset.
Method
Average recognition rate (%)
The proposed method
93.30
Zhang and Gong [35]
89.70
Gong et al. [36]
92.50
Chang et al. [37]
91.20The experimental results show that the approach proposed in the paper can get satisfactory results and significantly performs better comparing the average accuracy with that in [35–37] because of a practical method adopted in the paper.


## 6.1. Datasets

We test our algorithm on three datasets: the Weizmann human motion dataset [33], the KTH human action dataset [30], and the UIUC action dataset [34]. All the experiments are conducted on a Pentium 4 machine with 2 GB of RAM, using the implementation on MATLAB. The dataset and the related experimental results are presented in the following sections.The KTH dataset is provided by Schuldt which contains 2391 video sequences with 25 actors showing six actions. Each action is performed in 4 different scenarios.The WEIZMANN dataset is provided by Blank which contains 93 video sequences showing nine different people, each performing ten actions, such as run, walk, skip, jump-jack, jump-forward-on-two-legs, jump-in-place-on-two-legs, gallop-sideways, wave-two-hands, wave-one-hand, and bend.The UIUC action dataset is created by the University of Illinois at Urbana-Champaign (UIUC) in 2008 for human activity recognition. The activities are walking, running, jumping, waving, jumping jacks, clapping, jumping from sit up, raising one hand, stretching out, turning, sitting to standing, crawling, pushing up, and standing to sitting.For every dataset, 12 video sequences are taken by four subjects (out of the five) used for training and the remaining three videos for testing. The experiments are repeated five times. The performance of different methods is shown using the average recognition rate. In order to evaluate the performance of action recognition, we report the overall accuracy on three datasets.


## 6.2. Comparison

KTH Dataset. It contains six types of human actions (walking, jogging, running, boxing, hand waving, and hand clapping) performed several times by 25 subjects in four different scenarios: outdoors, outdoors with scale variation, outdoors with different clothes, and indoors. Representative frames of this dataset are shown in Figure 6(a). After process of restoring missing coordinate position, we use the proposed method, and the classification results of KTH dataset obtained by this approach are shown in Figure 7 and indicate quite a small number of videos are misclassified, particularly, the actions “running” and “handclapping” which more tend to be confused.Sample frames from our datasets. The action labels in each dataset are as follows: (a) KTH dataset: walking (a1), jogging (a2), running (a3), boxing (a4), handclapping (a5); (b) Weizmann dataset: running, walking, jumping-jack, waving-two-hands, waving-one-hand, bending; (c) UIUC action dataset: raising two hands (a1), running (a2), clapping (a3), stretching out (a4), and pushing up (a5).
(a)
(b)
(c)Figure 7
Confusion matrix for KTH dataset.The Weizmann Dataset. The Weizmann human action dataset contains 83 video sequences showing nine different people and all performing nine different actions: bending (a1), jumping jack (a2), jumping forward on two legs (a3), jumping in place on two legs (a4), running (a5), galloping sideways (a6), walking (a7), waving one hand (a8), and waving two hands (a9). The figures were tracked and stabilized by using the background subtraction masks that come with this dataset. Some sample frames are shown in Figure 6(b). The classified results achieved by this approach are shown in Figure 8.Figure 8
Confusion matrix for Weizmann dataset.The UIUC Action Dataset. This dataset consists of 532 high resolution sequences of 14 activities performed by eight actors. The activities are walking, running, jumping, waving, jumping jacks, clapping, jumping from sit-up, raising one hand, stretching out, turning, sitting to standing, crawling, pushing up, and standing to sitting. Some sample frames are shown in Figure 6(c). The classified results achieved by this approach are shown in Figure 9.Figure 9
Confusion matrix for UIUC dataset.In this paper, we identify jogging, running, walking, and boxing and compare the proposed method with the four state-of-the-art methods in the literature: Zhang and Gong [35], Gong et al. [36], and Chang et al. [37] in three datasets. As shown in Tables 1, 2, and 3, the existing methods, the low recognition accuracy because these action are not only occlusion situation are complex, but also the legs have complex beat, motion and other group actions. The proposed method can overcome these problems, and the recognition accuracy and average accuracy are higher than the comparative method.Table 1
Compared with other approaches on KTH dataset.
Method
Average recognition rate (%)
The proposed method
90.60
Zhang and Gong [35]
83.40
Gong et al. [36]
89.20
Chang et al. [37]
88.03Table 2
Compared with other approaches on Weizmann dataset.
Method
Average recognition rate (%)
The proposed method
89.20
Zhang and Gong [35]
85.30
Gong et al.  [36]
86.30
Chang et al. [37]
82.10Table 3
Compared with other approaches on UIUC dataset.
Method
Average recognition rate (%)
The proposed method
93.30
Zhang and Gong [35]
89.70
Gong et al. [36]
92.50
Chang et al. [37]
91.20The experimental results show that the approach proposed in the paper can get satisfactory results and significantly performs better comparing the average accuracy with that in [35–37] because of a practical method adopted in the paper.


## 7. Conclusions and Future Work

In this paper, we proposed an adaptive occlusion action recognition method for human body movement. Our method successfully recognizes without assuming a known and fixed depth order. We have presented the MRF model and SFM model, which estimates the adaptive occlusion state and recovers the important missing parts of objects in a video clip. This paper presents a new method of human self-occlusion behavior recognition, which is based on optical flow and correlated topic model (CTM). Then, we have employed the optical flow motion feature to extract the key frame and calculated the ratio of the width and height from human silhouette. Finally, we use the topic model of correlated topic model (CTM) to classify and recognize action. Experiments were performed on the KTH, Weizmann, and UIUC action datasets to test and evaluate the proposed method. The compared experiment results showed that the proposed method was more effective than compared methods and better than other approaches [35–37].Future work will deal with adding complex event detection to the proposed system, involving more complex problems such as dealing with more variable motion, interperson occlusions, and possible appearance similarity of different people and increasing the database size.



---

*Source: 810185-2014-01-16.xml*