A stepped sampling method for video detection using LSTM

Artificial neural networks are considered to simulate the human neural networks , and achieves great progress on object detection , natural language processing ( NLP ) , image generation , etc . Hermann Ebbinghaus proposed the law of human memory and how to improve human learning in 1885 . Inspiring from Ebbinghaus ’ work , we propose a stepped sampler based on the “ repeated input ” , which is Ebbinghaus ’ contribution that how to strengthen the learning . We repeatedly inputted data to the LSTM model stepwise in a batch . The stepped sampler is used to strengthen the ability of fusing the temporal information in LSTM . We tested the stepped sampler on the LSTM offered by PyTorch . Compared with the traditional sampler of PyTorch , such as sequential sampler , batch sampler , the training loss of the proposed stepped sampler converges faster in the model training , and the training loss after convergence is more stable , which means that there is no large jitter after the convergence . Meanwhile , it can maintain a higher test accuracy , compared with the traditional sampler . We quantified the algorithm of the stepped sampler . We assume that , the artificial neural networks may have human-like characteristics , and human learning method could be used for machine learning . Our code will be available online soon . 1 INTRODUCTION . The emergence of convolutional neural networks ( CNN ) ( LeCun et al. , 1989 ) has improved the selflearning ability of artificial neural networks . Recurrent Neural Network ( RNN ) ( Mikolov et al. , 2010 ) is used to process the temporal information data . RNN takes the output of the previous time period as the input of the next time period , effectively using the temporal information of the input sequence . RNN sometimes may have the problem of gradient disappearance or gradient explosion . Hochreiter et al . ( Hochreiter & Schmidhuber , 1997 ) proposed LSTM . LSTM adds gates to RNN , thus it can effectively avoid the problem of gradient disappearance or explosion . These gates include the forgetting gates , the input gates , and the output gates . The forgetting gate seems to be the most important among them . LSTM may simulate the memory process of human brain . Human brain selectively forgets some information for learning better . Consider that one of the principles of neural networks may be learned from biological neural networks , for those artificial neural networks with the memory effects , such as LSTM , learning from the memory method of human , which is the repeated input and timely review , we study the effect of this method with repeated input on LSTM detection results , without considering changing the LSTM network structure . In this study , we learn the effect of the proposed input method on neural networks with memory characteristics , such as LSTM . Specifically , it is to repeatedly input training data by simulating the “ repeated input and timely review ” method of the human memory , and the “ repeated input and timely review ” method is proposed by Hermann Ebbinghaus ( Ebbinghaus , 1913 ) in 1885 , which is the “ Increasing Memory Strength and Rate of Learning ” in his literature . 1.1 OUR CONTRIBUTION . Our views in this paper mainly include the following 3 aspects : a ) A novel sampler is proposed , which implements sampling in a circular and stepwise manner . Compared with the traditional sampler , the loss curve of the LSTM model using this stepped sampler converges faster in training , and is more stable after the convergence , namely there is no large jitter after the convergence . Moreover , its test accuracy curve is more stable either , which has no jitter . When the batch size is 15 , the test accuracy of the stepped sampler LSTM is much higher than that of the traditional sampler with the same parameters . b ) The idea of this sampler comes from the laws of human memory , which was proposed by Ebbinghaus ( Ebbinghaus , 1913 ) . We courageously assume that , other human learning methods can also be applied to machine learning . One example is the proposal of the attention mechanism ( Vaswani et al. , 2017 ) . Moreover , we believe that artificial neural networks have human-like characteristics from the experimental performance . c ) We try to use mathematical language to describe the temporal information of the video frames . We try to apply the mathematical equations to our experimental results , and analyze that the test accuracy in the experiment is the temporal information between video frames . The derivation process is shown in Appendix A and Appendix B . 2 RELATED WORK . Gibbs sampling is one of the earlier data sampling algorithms , which is proposed by Geman et al . ( Geman & Geman , 1984 ) in 1984 . Gibbs sampling is to make the probability of the data sample approximately equal to the required probability distribution via iterations . Gibbs sampling randomly selects data from an initial input sequence , and iterates according to the specified conditional probabilities , which are related to the required probability distribution of the final sampling data . After iterations , Gibbs sampling generates data which is consistent with the required probability distribution . Hu et al . ( Hu et al. , 2018 ) used neural networks to generate a sampler , which transfer the initial data distribution to the target distribution . The method can generate the sampling data at the same time of training . This method works with the un-normalized probability density function . Wang et al . ( Wang et al. , 2018 ) used Generative Adversarial Nets ( GAN ) ( Goodfellow et al. , 2014 ) to generate the negative samples . The approach is the first to combine GAN with the negative sampling method , which improves the training effect of the streaming recommend system . Chu et al . ( Chu et al. , 2019 ) proposed a novel sampler that can sample both the positive and the negative data from the input data sequences , so as to let the classifier utilize the Regions of Interests and the background of the data . The sampler is used in the few-shot image classifier , which uses the reinforcement learning method . The reinforcement learning algorithm ( Kaelbling et al. , 1996 ) needs to continuously select the regions of interests from the images , subsequently to recognize the content of the Regions of Interests . Sampling these Regions of Interests can improve the efficiency of reinforcement learning , for the reason of the reduction of the input samples . Muhammad et al . ( Muhammad et al. , 2021 ) proposed a bi-directional long short-term memory ( BiLSTM ) with attention mechanism and a dilated convolutional neural network ( DCNN ) to perform action recognition , which outperformed the state-of-the-art methods . Kwon et al . ( Kwon et al. , 2021 ) proposed a spatio-temporal neighbourhood learning method on action recognition , which performed the state-of-the-art . 3 MATERIALS AND METHODS . This paper is from the perspective of data input , rather than the neural network structure , and study the impact of the memory effect on the temporal sequence neural networks ( such as LSTM ) . The process simulates the method of enhancing the memory process of human brain , repeats the input data in a stepped way . The method is proposed by Hermann Ebbinghaus ( Ebbinghaus , 1913 ) called “ Increasing rate of learning ” in his book . The specific mode we used was the wheel tactic ( Smith , 1994 ) when we recited words , by establishing a novel data sampler in the LSTM model training . The dataset in the experiment is UCF101 ( Soomro et al. , 2012 ) , which is a human action recognition video dataset . The name of each folder indicates the annotation of the video . 3.1 EBBINGHAUS FORGETTING CURVE . Ebbinghaus forgetting curve ( Ebbinghaus , 1913 ) describes the memory effect of human brain over time , which was proposed by Hermann Ebbinghaus in 1885 . This theory reveals the human memory law . It is also the law of human learning . That is , the loss of human memory when learning new knowledge is drop fast first and slow later . Ebbinghaus also pointed out that , timely review and repeated input are the key point to prevent forgetting , consolidate knowledge , and learn better . Figure 1 illustrates Ebbinghaus forgetting curve , and timely review can reduce the knowledge forgetting , which makes the learning better . Based on Ebbinghaus forgetting curve on the human brain , we simulated Ebbinghaus ’ method on machine learning . We believe that the experimental results in Section 4 could prove that there is a certain correlation between human learning and machine learning , since the machine learning method with timely review and spaced repeat has a faster learning effect , compared with the machine learning without the human-like method . Ebbinghaus also found that , making good use of the correlations between knowledge is another key point for enhancing learning . We definite these correlations are temporal information in Appendix A . Thereby enhancing the use of temporal information is the key to video detection , natural language processing ( NLP ) , etc . We believe that , the partly repeated input of the stepped sampler enhances the correlation and the temporal information . 3.2 LSTM The LSTM architecture we used in this paper is to start with a CNN backbone . The CNN backbone has four convolutional layers . The dimension of convolution kernels of each convolutional layer is 32 , 64 , 128 , 256 . The size of each convolution kernel is 5 × 5 , 3 × 3 , 3 × 3 , 3×3 , the stride of the convolution kernel is 2 , and the padding is 0 . Each convolutional layer is followed by a batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) layer and a ReLU layer . The last part of the CNN model is 3 fully connected ( FC ) layers , which use dropout function . The dimensions of the 3 FC layers are 1024 , 768 and 512 respective- ly . The LSTM used in the paper is the model existed in PyTorch . The input dimension of LSTM is 512 , the hidden layer dimension is 512 , and the number of hidden layers is 3 . The next is two fully connected ( FC ) layers followed by dropout function . The dimension of the FC layers is 256 . The dropout rate of the CNN backbone and LSTM are both 0.3 . 3.3 THE STEPPED SAMPLER . Our experiment is compared with the common sampler . Common sampler in PyTorch ( Paszke et al. , 2019 ) include random sampler , weighted random sampler , batch sampler , etc . Batch sampling is nearly the most commonly used . The previous research is to add memory units to deep learning networks , such as RNN , LSTM , etc . Analogous to human learning , an important point is repetition . And the sampler should be an appropriate way to simulate the “ repetition ” , since the data in each batch can be designed to be input repeatedly . We suppose that , “ repetition ” is important , not only for human beings , but also for computers . To make computers better use the “ repetition ” , analogizing the way how we recite words , we propose a “ stepped ” repetition input method , which is the stepped sampler . The structure of the proposed stepped sampler is illustrated in Figure 2 . It is established on the batch sampler . The stepped sampler divides a batch into some sub-batches . Like human memory , this sampler adopts the principle of adjacent repetition ( Crowder , 1968 ) , namely , the back of the previous sub-batch is the same with the front of the next sub-batch . The structure of the stepped sampler shows that , the input data of different batches is partly duplicated . The repeated input seems to increase the redundancy , but the experimental results show that , with our experimental environment , this method can accelerate the convergence of LSTM model . There is a stride between the previous sub-batch and the next sub-batch . The stride size n can be set manually . We believe that this part repetition enhances the correlation of the input frames , thereby enhancing the temporal information of the input frames , according to our definition of the temporal information in Appendix A . Section 4 describes the comparative experiments on the sampler with different stride size .

Summary: This paper mainly provides a stepped sampling method only for video detection and only adopted in LSTM structure. This idea is inspired from human's repeatedly memory working mechanism. This method can achieve fast convergence during training, smooth the training loss curves and get a good test accuracy when batch size equal to 15.