A stepped sampling method for video detection using LSTM

Artificial neural networks are considered to simulate the human neural networks , and achieves great progress on object detection , natural language processing ( NLP ) , image generation , etc . Hermann Ebbinghaus proposed the law of human memory and how to improve human learning in 1885 . Inspiring from Ebbinghaus ’ work , we propose a stepped sampler based on the “ repeated input ” , which is Ebbinghaus ’ contribution that how to strengthen the learning . We repeatedly inputted data to the LSTM model stepwise in a batch . The stepped sampler is used to strengthen the ability of fusing the temporal information in LSTM . We tested the stepped sampler on the LSTM offered by PyTorch . Compared with the traditional sampler of PyTorch , such as sequential sampler , batch sampler , the training loss of the proposed stepped sampler converges faster in the model training , and the training loss after convergence is more stable , which means that there is no large jitter after the convergence . Meanwhile , it can maintain a higher test accuracy , compared with the traditional sampler . We quantified the algorithm of the stepped sampler . We assume that , the artificial neural networks may have human-like characteristics , and human learning method could be used for machine learning . Our code will be available online soon . 1 INTRODUCTION . The emergence of convolutional neural networks ( CNN ) ( LeCun et al. , 1989 ) has improved the selflearning ability of artificial neural networks . Recurrent Neural Network ( RNN ) ( Mikolov et al. , 2010 ) is used to process the temporal information data . RNN takes the output of the previous time period as the input of the next time period , effectively using the temporal information of the input sequence . RNN sometimes may have the problem of gradient disappearance or gradient explosion . Hochreiter et al . ( Hochreiter & Schmidhuber , 1997 ) proposed LSTM . LSTM adds gates to RNN , thus it can effectively avoid the problem of gradient disappearance or explosion . These gates include the forgetting gates , the input gates , and the output gates . The forgetting gate seems to be the most important among them . LSTM may simulate the memory process of human brain . Human brain selectively forgets some information for learning better . Consider that one of the principles of neural networks may be learned from biological neural networks , for those artificial neural networks with the memory effects , such as LSTM , learning from the memory method of human , which is the repeated input and timely review , we study the effect of this method with repeated input on LSTM detection results , without considering changing the LSTM network structure . In this study , we learn the effect of the proposed input method on neural networks with memory characteristics , such as LSTM . Specifically , it is to repeatedly input training data by simulating the “ repeated input and timely review ” method of the human memory , and the “ repeated input and timely review ” method is proposed by Hermann Ebbinghaus ( Ebbinghaus , 1913 ) in 1885 , which is the “ Increasing Memory Strength and Rate of Learning ” in his literature . 1.1 OUR CONTRIBUTION . Our views in this paper mainly include the following 3 aspects : a ) A novel sampler is proposed , which implements sampling in a circular and stepwise manner . Compared with the traditional sampler , the loss curve of the LSTM model using this stepped sampler converges faster in training , and is more stable after the convergence , namely there is no large jitter after the convergence . Moreover , its test accuracy curve is more stable either , which has no jitter . When the batch size is 15 , the test accuracy of the stepped sampler LSTM is much higher than that of the traditional sampler with the same parameters . b ) The idea of this sampler comes from the laws of human memory , which was proposed by Ebbinghaus ( Ebbinghaus , 1913 ) . We courageously assume that , other human learning methods can also be applied to machine learning . One example is the proposal of the attention mechanism ( Vaswani et al. , 2017 ) . Moreover , we believe that artificial neural networks have human-like characteristics from the experimental performance . c ) We try to use mathematical language to describe the temporal information of the video frames . We try to apply the mathematical equations to our experimental results , and analyze that the test accuracy in the experiment is the temporal information between video frames . The derivation process is shown in Appendix A and Appendix B . 2 RELATED WORK . Gibbs sampling is one of the earlier data sampling algorithms , which is proposed by Geman et al . ( Geman & Geman , 1984 ) in 1984 . Gibbs sampling is to make the probability of the data sample approximately equal to the required probability distribution via iterations . Gibbs sampling randomly selects data from an initial input sequence , and iterates according to the specified conditional probabilities , which are related to the required probability distribution of the final sampling data . After iterations , Gibbs sampling generates data which is consistent with the required probability distribution . Hu et al . ( Hu et al. , 2018 ) used neural networks to generate a sampler , which transfer the initial data distribution to the target distribution . The method can generate the sampling data at the same time of training . This method works with the un-normalized probability density function . Wang et al . ( Wang et al. , 2018 ) used Generative Adversarial Nets ( GAN ) ( Goodfellow et al. , 2014 ) to generate the negative samples . The approach is the first to combine GAN with the negative sampling method , which improves the training effect of the streaming recommend system . Chu et al . ( Chu et al. , 2019 ) proposed a novel sampler that can sample both the positive and the negative data from the input data sequences , so as to let the classifier utilize the Regions of Interests and the background of the data . The sampler is used in the few-shot image classifier , which uses the reinforcement learning method . The reinforcement learning algorithm ( Kaelbling et al. , 1996 ) needs to continuously select the regions of interests from the images , subsequently to recognize the content of the Regions of Interests . Sampling these Regions of Interests can improve the efficiency of reinforcement learning , for the reason of the reduction of the input samples . Muhammad et al . ( Muhammad et al. , 2021 ) proposed a bi-directional long short-term memory ( BiLSTM ) with attention mechanism and a dilated convolutional neural network ( DCNN ) to perform action recognition , which outperformed the state-of-the-art methods . Kwon et al . ( Kwon et al. , 2021 ) proposed a spatio-temporal neighbourhood learning method on action recognition , which performed the state-of-the-art . 3 MATERIALS AND METHODS . This paper is from the perspective of data input , rather than the neural network structure , and study the impact of the memory effect on the temporal sequence neural networks ( such as LSTM ) . The process simulates the method of enhancing the memory process of human brain , repeats the input data in a stepped way . The method is proposed by Hermann Ebbinghaus ( Ebbinghaus , 1913 ) called “ Increasing rate of learning ” in his book . The specific mode we used was the wheel tactic ( Smith , 1994 ) when we recited words , by establishing a novel data sampler in the LSTM model training . The dataset in the experiment is UCF101 ( Soomro et al. , 2012 ) , which is a human action recognition video dataset . The name of each folder indicates the annotation of the video . 3.1 EBBINGHAUS FORGETTING CURVE . Ebbinghaus forgetting curve ( Ebbinghaus , 1913 ) describes the memory effect of human brain over time , which was proposed by Hermann Ebbinghaus in 1885 . This theory reveals the human memory law . It is also the law of human learning . That is , the loss of human memory when learning new knowledge is drop fast first and slow later . Ebbinghaus also pointed out that , timely review and repeated input are the key point to prevent forgetting , consolidate knowledge , and learn better . Figure 1 illustrates Ebbinghaus forgetting curve , and timely review can reduce the knowledge forgetting , which makes the learning better . Based on Ebbinghaus forgetting curve on the human brain , we simulated Ebbinghaus ’ method on machine learning . We believe that the experimental results in Section 4 could prove that there is a certain correlation between human learning and machine learning , since the machine learning method with timely review and spaced repeat has a faster learning effect , compared with the machine learning without the human-like method . Ebbinghaus also found that , making good use of the correlations between knowledge is another key point for enhancing learning . We definite these correlations are temporal information in Appendix A . Thereby enhancing the use of temporal information is the key to video detection , natural language processing ( NLP ) , etc . We believe that , the partly repeated input of the stepped sampler enhances the correlation and the temporal information . 3.2 LSTM The LSTM architecture we used in this paper is to start with a CNN backbone . The CNN backbone has four convolutional layers . The dimension of convolution kernels of each convolutional layer is 32 , 64 , 128 , 256 . The size of each convolution kernel is 5 × 5 , 3 × 3 , 3 × 3 , 3×3 , the stride of the convolution kernel is 2 , and the padding is 0 . Each convolutional layer is followed by a batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) layer and a ReLU layer . The last part of the CNN model is 3 fully connected ( FC ) layers , which use dropout function . The dimensions of the 3 FC layers are 1024 , 768 and 512 respective- ly . The LSTM used in the paper is the model existed in PyTorch . The input dimension of LSTM is 512 , the hidden layer dimension is 512 , and the number of hidden layers is 3 . The next is two fully connected ( FC ) layers followed by dropout function . The dimension of the FC layers is 256 . The dropout rate of the CNN backbone and LSTM are both 0.3 . 3.3 THE STEPPED SAMPLER . Our experiment is compared with the common sampler . Common sampler in PyTorch ( Paszke et al. , 2019 ) include random sampler , weighted random sampler , batch sampler , etc . Batch sampling is nearly the most commonly used . The previous research is to add memory units to deep learning networks , such as RNN , LSTM , etc . Analogous to human learning , an important point is repetition . And the sampler should be an appropriate way to simulate the “ repetition ” , since the data in each batch can be designed to be input repeatedly . We suppose that , “ repetition ” is important , not only for human beings , but also for computers . To make computers better use the “ repetition ” , analogizing the way how we recite words , we propose a “ stepped ” repetition input method , which is the stepped sampler . The structure of the proposed stepped sampler is illustrated in Figure 2 . It is established on the batch sampler . The stepped sampler divides a batch into some sub-batches . Like human memory , this sampler adopts the principle of adjacent repetition ( Crowder , 1968 ) , namely , the back of the previous sub-batch is the same with the front of the next sub-batch . The structure of the stepped sampler shows that , the input data of different batches is partly duplicated . The repeated input seems to increase the redundancy , but the experimental results show that , with our experimental environment , this method can accelerate the convergence of LSTM model . There is a stride between the previous sub-batch and the next sub-batch . The stride size n can be set manually . We believe that this part repetition enhances the correlation of the input frames , thereby enhancing the temporal information of the input frames , according to our definition of the temporal information in Appendix A . Section 4 describes the comparative experiments on the sampler with different stride size .

Summary: This paper proposes a sampler for LSTM video models. The sampler works by repeating frames in a training batch. The method is evaluated for the task of action recognition on the UCF 101 dataset.