# Deep ChaosNet for Action Recognition in Videos

Authors: Huafeng Chen; Maosheng Zhang; Zhengming Gao; Yunhong Zhao
Journal: Complexity
 (2021)

---

## Abstract

Current methods of chaos-based action recognition in videos are limited to the artificial feature causing the low recognition accuracy. In this paper, we improve ChaosNet to the deep neural network and apply it to action recognition. First, we extend ChaosNet to deep ChaosNet for extracting action features. Then, we send the features to the low-level LSTM encoder and high-level LSTM encoder for obtaining low-level coding output and high-level coding results, respectively. The agent is a behavior recognizer for producing recognition results. The manager is a hidden layer, responsible for giving behavioral segmentation targets at the high level. Our experiments are executed on two standard action datasets: UCF101 and HMDB51. The experimental results show that the proposed algorithm outperforms the state of the art.


---

## Body

## 1. Introduction

Human action recognition in videos is an important area in computer vision, receiving sustained attention from the researchers due to its potential applications such as video supervision, entertainment, user interface, sports, video understanding, and patient monitoring. Current action recognition methods can be classified into three categories by action feature: chaos-based feature [1], manual feature [2], and deep learned feature. Inspired by the chaos-based feature and deep learned feature, we propose deep ChaosNet for action recognition to autonomously learn the nonlinear dynamical feature in video action.


## 2. Related Works

In this section, we briefly review the literature of action recognition from the chaos-based feature, manual feature, and deep learned feature.


### 2.1. Chaos-Based Feature

Ali et al. [3] introduced a human action recognition architecture by using the theory of chaotic systems to model and analyze nonlinear dynamics of human actions. Trajectories of reference joints are used as the representation of the nonlinear dynamical system that is generating the action. Xia et al. [4] proposed a human behavior recognition method based on chaotic invariant features and relevance vector machine (RVM). The trajectory generated by the motion of the human joint points is extracted to represent the nonlinear system of human action behavior, and the time delay is estimated by the C-C method. The chaotic invariants representing human behavior are extracted, and the RVM algorithm is used to identify human behavior. Venkataraman and Turaga [1] proposed to use the descriptor of the shape of the dynamical attractor as the feature representation of the nature of dynamics to solve the drawbacks of traditional approaches.


### 2.2. Manual Feature

Since human behavior is composed of body movements, general human behavior characteristics are based on the underlying visual movement characteristics. The underlying visual features are easy to extract and represent, and the underlying visual motion features of the same action have a certain degree of robustness under different cameras, so they are widely used in early human behavior recognition. There are two categories on human behavior characteristics: local feature representation and global feature representation. Existing global feature descriptions represent the formation of global spatiotemporal cues through single-frame global features and video frame sequences from aspects of human body contours, posture joint points, and saliency segmentation such as the motion history image algorithm (motion history image, MHI) proposed by Bobick and Davis [5], the adaptation of the shape context algorithm (adaptation of the shape context) proposed by Zhang et al. [6], and the kinematic feature proposed by Ali and Shah [7]. Local feature description of underlying action features is still a hotspot in human behavior recognition research in recent years. Researchers considered the changes in the motion field between frames and proposed various local spatiotemporal feature descriptions, such as STIP [8], MoSIFT [9, 10], and dense trajectories [2, 11].


### 2.3. Deep Learned Feature

It includes two aspects of deep learning: action convolution features and action timing features. The former uses convolutional neural networks (CNNs) to learn the local depth features of human behavior from different modal data such as RGB image frames and optical flow of behavior videos [12]. On the basis of behavioral convolutional features, it uses methods such as recurrent neural network (RNN), time-series segment network, or linear coding to learn time-series features in multiple stages of behavior development [13]. Due to limited memory capacity of the GPU/CPU and different lengths of behavior duration (shown as different video frames), it is difficult to send all behavior video frames into the deep learning framework for feature learning. Therefore, it is necessary to perform key frame sampling on the behavior video in the behavior recognition process. Most of the existing behavior recognition algorithms use equal sampling [13] or sequential sampling [14–16], ignoring the differences in the development process of human behavior, and the key frames obtained are less representative.


## 2.1. Chaos-Based Feature

Ali et al. [3] introduced a human action recognition architecture by using the theory of chaotic systems to model and analyze nonlinear dynamics of human actions. Trajectories of reference joints are used as the representation of the nonlinear dynamical system that is generating the action. Xia et al. [4] proposed a human behavior recognition method based on chaotic invariant features and relevance vector machine (RVM). The trajectory generated by the motion of the human joint points is extracted to represent the nonlinear system of human action behavior, and the time delay is estimated by the C-C method. The chaotic invariants representing human behavior are extracted, and the RVM algorithm is used to identify human behavior. Venkataraman and Turaga [1] proposed to use the descriptor of the shape of the dynamical attractor as the feature representation of the nature of dynamics to solve the drawbacks of traditional approaches.


## 2.2. Manual Feature

Since human behavior is composed of body movements, general human behavior characteristics are based on the underlying visual movement characteristics. The underlying visual features are easy to extract and represent, and the underlying visual motion features of the same action have a certain degree of robustness under different cameras, so they are widely used in early human behavior recognition. There are two categories on human behavior characteristics: local feature representation and global feature representation. Existing global feature descriptions represent the formation of global spatiotemporal cues through single-frame global features and video frame sequences from aspects of human body contours, posture joint points, and saliency segmentation such as the motion history image algorithm (motion history image, MHI) proposed by Bobick and Davis [5], the adaptation of the shape context algorithm (adaptation of the shape context) proposed by Zhang et al. [6], and the kinematic feature proposed by Ali and Shah [7]. Local feature description of underlying action features is still a hotspot in human behavior recognition research in recent years. Researchers considered the changes in the motion field between frames and proposed various local spatiotemporal feature descriptions, such as STIP [8], MoSIFT [9, 10], and dense trajectories [2, 11].


## 2.3. Deep Learned Feature

It includes two aspects of deep learning: action convolution features and action timing features. The former uses convolutional neural networks (CNNs) to learn the local depth features of human behavior from different modal data such as RGB image frames and optical flow of behavior videos [12]. On the basis of behavioral convolutional features, it uses methods such as recurrent neural network (RNN), time-series segment network, or linear coding to learn time-series features in multiple stages of behavior development [13]. Due to limited memory capacity of the GPU/CPU and different lengths of behavior duration (shown as different video frames), it is difficult to send all behavior video frames into the deep learning framework for feature learning. Therefore, it is necessary to perform key frame sampling on the behavior video in the behavior recognition process. Most of the existing behavior recognition algorithms use equal sampling [13] or sequential sampling [14–16], ignoring the differences in the development process of human behavior, and the key frames obtained are less representative.


## 3. Deep ChaosNet Framework

Inspired by Wang et al. and Balakrishnan et al. [15, 17], we propose deep ChaosNet framework for action recognition. The framework is illustrated in Figure 1. Deep ChaosNet features are extracted from video frames. And then, the features are sent to the low-level LSTM encoder and high-level LSTM encoder for obtaining low-level coding output and high-level coding results, respectively. The agent is a behavior recognizer for producing recognition results. The agent, based on hierarchical reinforcement learning, is mainly composed of manager and worker. Manager is a hidden layer, responsible for giving behavioral segmentation targets at the high level. Worker determines the spatiotemporal area of the video subsegment that best characterizes the segmentation target according to the segmentation target and outputs the segmentation recognition result.Figure 1
Illustration of the pipeline of the proposed deep ChaosNet feature extraction.


### 3.1. Structure of the Network

The network system structure is shown in Figure2. The manager LSTM unit obtains environmental status information htM according to the input ct−1M,ht−1W and derives meaningful behavioral stage goals, which are used as the worker LSTM input to guide the worker to select the spatiotemporal region of the next behavioral video subsegment; the formula is as follows:(1)htM=SMht−1M,ctM,ht−1W,gt=uMhtM.Figure 2
Illustration of the multilayer feature fusing.SM is the manager LSTM nonlinear function, and uM is responsible for mapping the environmental state information htM to the behavioral stage target gt. The worker LSTM unit obtains context information htW according to the input ct−1W,gt. Based on htW, we predict the next key frame position dt, sampling area lt, and behavior category pt.For manager and worker, this project uses a visual attention mechanism to explore areas of salient behavior. The manager attention model mainly explores the significant segment information of the behavior, and the worker attention model assists in searching the behavior key frames and significant areas within the frame. The parametersCtM and CtW are calculated as follows:(2)CtM=∑αt,iMhiEm,αt,iM=expet,iM∑k=1nexpet,kM,et,iM=mTtanhWaMhiEm+UaMht−1M+baM,CtW=∑αt,iWhiEw,αt,iW=expet,iW∑k=1nexpet,kW,et,iW=wTtanhWaWhiEm+UaWht−1W+baW.


### 3.2. Deep Learning Process

The worker strategy learning process is a standard reinforcement learning process. At each stept of the worker, the worker will give a classification prediction result Pt, and then the environment will give a reward Rt, so the goal of worker strategy learning is to minimize the negative value of the reward function. The loss function is(3)Lθw=−Ept∼πθwRpt.Manager does not directly interact with the environment, and its strategy learning process cannot copy the worker. Compared with manager’s timet, the worker strategy πθwpt;gt is relatively stable, and this strategy directly affects the worker’s behavior classification output results pt,c at time c. At this point, although the manager is a hidden layer, its strategic goal should be to minimize the negative value of the current reward. The loss function is(4)LθM=−EgtRgtπpt,c;gt.


## 3.1. Structure of the Network

The network system structure is shown in Figure2. The manager LSTM unit obtains environmental status information htM according to the input ct−1M,ht−1W and derives meaningful behavioral stage goals, which are used as the worker LSTM input to guide the worker to select the spatiotemporal region of the next behavioral video subsegment; the formula is as follows:(1)htM=SMht−1M,ctM,ht−1W,gt=uMhtM.Figure 2
Illustration of the multilayer feature fusing.SM is the manager LSTM nonlinear function, and uM is responsible for mapping the environmental state information htM to the behavioral stage target gt. The worker LSTM unit obtains context information htW according to the input ct−1W,gt. Based on htW, we predict the next key frame position dt, sampling area lt, and behavior category pt.For manager and worker, this project uses a visual attention mechanism to explore areas of salient behavior. The manager attention model mainly explores the significant segment information of the behavior, and the worker attention model assists in searching the behavior key frames and significant areas within the frame. The parametersCtM and CtW are calculated as follows:(2)CtM=∑αt,iMhiEm,αt,iM=expet,iM∑k=1nexpet,kM,et,iM=mTtanhWaMhiEm+UaMht−1M+baM,CtW=∑αt,iWhiEw,αt,iW=expet,iW∑k=1nexpet,kW,et,iW=wTtanhWaWhiEm+UaWht−1W+baW.


## 3.2. Deep Learning Process

The worker strategy learning process is a standard reinforcement learning process. At each stept of the worker, the worker will give a classification prediction result Pt, and then the environment will give a reward Rt, so the goal of worker strategy learning is to minimize the negative value of the reward function. The loss function is(3)Lθw=−Ept∼πθwRpt.Manager does not directly interact with the environment, and its strategy learning process cannot copy the worker. Compared with manager’s timet, the worker strategy πθwpt;gt is relatively stable, and this strategy directly affects the worker’s behavior classification output results pt,c at time c. At this point, although the manager is a hidden layer, its strategic goal should be to minimize the negative value of the current reward. The loss function is(4)LθM=−EgtRgtπpt,c;gt.


## 4. Experiments and Results

We verify the proposed deep ChaosNet on two standard action datasets: UCF101 [18] and HMDB51 [19]. UCF101 is an action recognition dataset of realistic action videos with 101 action categories collected from YouTube. Videos of the 101 action categories are divided into 25 groups, and each group can contain 4∼7 action videos. Videos from the same group may share some common features, such as similar backgrounds and similar viewpoints. HMDB51 contains 51 types of actions, a total of 6849 videos which are collected from YouTube, Google Video, etc. Each action contains at least 51 videos with a resolution of 320 ∗ 240.In the experiments, we construct 7-layer deep ChaosNet for both action datasets. The outputs of the deep ChaosNet are 2048-dim frame features, which are then projected to 512-dim. We use Bi-LSTM with hidden size 512 as the low-level encoder and LSTM with hidden size 256 as the high-level encoder [20]. The worker network consisted of worker LSTM with hidden size 1024. The manager network was composed of manager LSTM with hidden size 256, an attention module, and a linear layer that projected the output of the LSTM into the latent goal space. The environment internal critic was also an RNN, which contained a GRU, a built-in word embedding, a linear layer, and a sigmoid function.We compare deep ChaosNet with the state-of-the-art deep learning methods [2, 12–16]. The comparison results are listed in Table 1. As shown in the table, the proposed deep ChaosNet exceeds the manual features [2] by 7.3% on UCF101 and by 5.8% on HMDB51. Our method is beyond the action convolution features [12] by 0.2% on UCF101 and by 2.6% on HMDB51. The proposed method also outperforms the action timing features [13–16] by 1.1% on UCF101 and 0.7% on HMDB51 at least. Overall, our deep ChaosNet method surpasses all the state-of-the-art methods and becomes the new state of the art.Table 1
Comparison of the proposed method with the state-of-the-art approaches.
MethodUCF101 (%)HMDB51 (%)Wang et al. [2]86.060.1Wu et al. [13]88.059.4Wang et al. [15]89.165.2Donahue et al. [14]90.363.2Chen et al. [16]92.462.0Sun et al. [12]93.163.3Deep ChaosNet93.365.9


## 5. Conclusions

We extend ChaosNet to the deep neural network and apply it to action recognition. We deepen the hidden layers of ChaosNet, and then we separately input still frames and motions among frames into the deep network to extract spatial and temporal action features. The features act as the input for the attention-based action recognition framework. We verify our method on two standard action datasets: UCF101 and HMDB51, and the experimental results indicate that the proposed algorithm is competitive compared with the state of the art.



---

*Source: 6634156-2021-02-13.xml*