Query: cnn feature extraction  action recognition

title: Automatic Badminton Action Recognition Using CNN with Adaptive Feature Extraction on Sensor Data
abstract: With the fast development of sensor technology, a sensor chip can be easily inserted into the badminton racket so that the data of the activity of the badminton player can be recorded continuously. It has become a very challenging problem whether we can use these sensor based data to effectively recognize the badminton actions. Although there have been some researches on the general human action recognition such as sitting, walking and running, there is few result or public dataset concerning badminton action recognition. In order to investigate the problem of badminton action recognition, we implement a specialized sensor chip being inserted into the badminton racket to collect the data of ten major badminton actions. On such a dataset, a deep convolutional neural network (CNN) can get a much better result of badminton action recognition than the conventional methods, but its recognition accuracy is not good enough for practical applications. The key difficulty is that the attributes recorded as time series in the sensor data have different measures and magnitudes so that the conventional feature normalization cannot work effectively in this complicated situation. In order to overcome this difficulty, we propose a specific block of adaptive feature extraction to enhance the performance of CNN for badminton action recognition. It is demonstrated by the experimental results on the sensor dataset that our proposed CNN with the adaptive feature extraction block can get 98.65% action recognition accuracy and outperforms the competitive methods remarkably.

title: Human Action Recognition based on Spatio temporal Feature Extraction using Pre trained CNN Model and LSTM approach

title: A Discriminative Deep Model With Feature Fusion and Temporal Attention for Human Action Recognition
abstract: Activity recognition which aims to accurately distinguish human actions in complex environments plays a key role in human robot/computer interaction. However, long lasting and similar actions will cause poor feature sequence extraction and thus lead to a reduction of the recognition accuracy. We propose a novel discriminative deep model (D3D LSTM) based on 3D CNN and LSTM for both single target and interaction action recognition to improve the spatiotemporal processing performance. Our models have several notable properties: 1) A real time feature fusion method is used to obtain a more representative feature sequence through composition of local mixtures for enhancing the performance of discriminating similar actions; 2) We introduce an improved attention mechanism that focuses on each frame individually by assigning different weights in real time; 3) An alternating optimization strategy is proposed for our model to obtain parameters with the best performance. Because the proposed D3D LSTM model is efficient enough to be used as a detector that recognizes various activities, a Real set database is collected to evaluate action recognition in complex real world scenarios. For long term relations, we update the present memory state via the weight controlled attention module that enables the memory cell to store better long term features. The densely connected bimodal modal makes local perceptrons of 3D Conv motion aware and stores better short term features. The proposed D3D LSTM model has been evaluated through a series of experiments on the Real set and open source datasets, i.e. SBU Kinect and MSR action 3D. Experimental results show that the proposed D3D LSTM model achieves new state of the art results, including pushing the average rate of the SBU Kinect to 92.40% and the average rate of the MSR action 3D to 95.40%

title: Human Action Recognition Based on Improved Fusion Attention CNN and RNN
abstract: The attention mechanism based models for computer vision and natural language processing are widely utilized, and action recognition in videos is no exception. In this paper, we develop a novel convolutional and recurrent network for action recognition which is "doubly deep" in spatial and temporal layers. First, in the feature extraction stage, we propose an improved p non local operations as a simple and effective component to capture long distance dependencies with deep convolutional neural networks. Second, in the class prediction stage, we propose Fusion KeyLess Attention combining with the forward and backward bidirectional LSTM to learn the sequential nature of the data more efficiently and elegantly, which uses multi epoch models fusion based on confusion matrix. Experiments on two heterogeneous datasets, HMDB51 and Hollywood2 show that our model has distinct advantages over traditional models also only utilizing RGB features for action recognition based on CNN and RNN.

title: A novel action recognition system for smart monitoring of elderly people using Action Pattern Image and Series CNN with transfer learning
abstract: Falling of elderly people who are staying alone at home leads to health risks. If they are not attended immediately even it may lead to fatal danger to their life. In this paper a novel computer vision based system for smart monitoring of elderly people using Series Convolutional Neural Network (SCNN) with transfer learning is proposed. When CNN is trained by the frames of the videos directly, it learns from all pixels including the background pixels. Generally, the background in a video does not contribute anything in identifying the action and actually it will mislead the action classification. So, we propose a novel action recognition system and our contributions are 1) to generate more general action patterns which are not affected by illumination and background variations of the video sequences and eliminate the obligation of image augmentation in CNN training 2) to design SCNN architecture and enhance the feature extraction process to learn large amount of data, 3) to present the patterns learnt by the neurons in the layers and analyze how these neurons capture the action when the input pattern is passing through these neurons, and 4) to extend the capability of the trained SCNN for recognizing fall actions using transfer learning.

title: Action Recognition Based on the Modified Twostream CNN
abstract: Human action recognition is an important research direction in computer vision areas. Its main content is to simulate human brain to analyze and recognize human action in video. It usually includes individual actions, interactions between people and the external environment. Space time dual channel neural network can represent the features of video from both spatial and temporal perspectives. Compared with other neural network models, it has more advantages in human action recognition. In this paper, a action recognition method based on improved space time two channel convolutional neural network is proposed. First, the video is divided into several equal length non overlapping segments, and a frame image representing the static feature of the video and a stacked optical flow image representing the motion feature are sampled at random part from each segment. Then these two kinds of images are input into the spatial domain and the temporal domain convolutional neural network respectively for feature extraction, and then the segmented features of each video are fused in the two channels respectively to obtain the category prediction features of the spatial domain and the temporal domain. Finally, the video action recognition results are obtained by integrating the predictive features of the two channels. Through experiments, various data enhancement methods and transfer learning schemes are discussed to solve the over fitting problem caused by insufficient training samples, and the effects of different segmental number, pre training network, segmental feature fusion scheme and dual channel integration strategy on action recognition performance are analyzed. The experiment results show that the proposed model can better learn the human action features in a complex video and better recognize the action.

title: SRNet: Structured Relevance Feature Learning Network From Skeleton Data for Human Action Recognition
abstract: In recent years, human action recognition based on skeleton information has recently drawn increasing attention with published large scale skeleton datasets. The most crucial factors for this task line in two aspects: the intra frame representation for joint co occurrences and the inter frame representation for skeletons' temporal evolution. The most effective ways focus on spontaneous feature extraction by using deep learning. However, they ignore the structure information of skeleton joints and the correlation between two different skeleton joints for human action recognition. In this paper, we do not simply treat the joints position information as unordered points. Instead, we propose a novel data reorganizing strategy to represent the global and local structure information of human skeleton joints. Meanwhile, we also employ the data mirror to increase the relationship between skeleton joints. Based on this design, we proposed an end to end multi dimensional CNN network (SRNet) to fully consider the spatial and temporal information to learn the feature extraction transform function. Specifically, in this CNN network, we employ different convolution kernels on different dimensions to learn skeleton representation to make the most of human structural information to generate robust features. Finally, we compare with other state of the art on action recognition datasets like NTU RGB+D, PKU MMD, SYSU, UT Kinect, and HDM05. The experimental results also demonstrate the superiority of our method.

title: Hallucinating Bag of Words and Fisher Vector IDT terms for CNN based Action Recognition
abstract: In this paper, we revive the use of old fashioned handcrafted video representations and put new life into these techniques via a CNN based hallucination step. Specifically, we address the problem of action classification in videos via an I3D network pre trained on the large scale Kinetics 400 dataset. Despite of the use of RGB and optical flow frames, the I3D model (amongst others) thrives on combining its output with the Improved Dense Trajectory (IDT) and extracted with it low level video descriptors encoded via Bag of Words (BoW) and Fisher Vectors (FV) Such a fusion of CNNs and hand crafted representations is time consuming due to various pre processing steps, descriptor extraction, encoding and fine tuning of the model. In this paper, we propose an end to end trainable network with streams which learn the IDT based BoW/FV representations at the training stage and are simple to integrate with the I3D model. Specifically, each stream takes I3D feature maps ahead of the last 1D conv. layer and learns to `translate' these maps to BoW/FV representations. Thus, our enhanced I3D model can hallucinate and use such synthesized BoW/FV representations at the testing stage. We demonstrate simplicity/usefulness of our model on three publicly available datasets and we show state of the art results.

title: Viewpoint projection based deep feature learning for single and dyadic action recognition
abstract: Usage of depth data in various computer vision applications gains popularity with cheap depth sensors available on the market. In this study, a method that utilizes Deep Learning is proposed for single human actions and dyadic actions. Depth data of the actions are used to construct a three dimensional (3D) template. These templates are rotated in different directions and two dimensional (2D) views from different angles are stored. Acquired 2D images are used in deep feature extraction. AlexNet (Krizhevsky, Sutskever, Hinton, 2012) pre trained convolutional neural network is used for deep feature extraction. For all viewpoints, deep features are extracted and concatenated. After that random forest classifier is used for action recognition. The contributions of this paper are as follows. First, a 3D isosurface models is constructed to figure actions sequence. Second, we project 3D shapes into a 2D feature space by taking its snapshots from different views and giving them to a pre trained CNN as input for feature extraction. We explore more information about the actions by extracting features from different layers of the deep neural network. We compare the results of the deep features extracted from different layers of the pre trained CNN. Various classifiers are trained with extracted deep features. Because of the complex structure of the 3D shapes, there are a limited number of feature extraction methods. Our method obtains close to state of the art performance on various single and two person action datasets.

title: Human action recognition using convolutional LSTM and fully connected LSTM with different attentions
abstract: This paper aims to address the human action recognition issue by using convolutional long short term memory networks (Conv LSTM) and fully connected LSTM (FC LSTM) with different attentions. To this end, the spatial temporal dual attention network (STDAN) which is mainly composed of feature extraction, attention and fusion modules, is designed. Different from the features of high level fully connected layer mostly used in previous work, the features of convolution and fully connected layers of convolutional neural network (CNN) are both extracted in STDAN, which can enrich the initial level of video representation. Besides, the Conv LSTM and FC LSTM are employed to handle the long duration sequential features with different temporal context information. To reinforce the spatial temporal attention ability, a temporal attention module (TAM) and a joint spatial temporal attention module (JSTAM) are implemented. Through the principle components analysis (PCA) and features fusion, the potential of STDAN is effectively explored and weighted. Finally, the experimental results show that the proposed STDAN has better recognition performance than existing state of the art methods.