# Variable Rate Independently Recurrent Neural Network (IndRNN) for Action Recognition
## 4 Proposed Variable Rate IndRNN
### Learning Rate Adjustment
#### 4.2.1 Learning Rate Adjustment for the First \(L-1\) IndRNN Layers  
For the first \(1,2,~{}\ldots,L-1\) IndRNN layers, the gradient consists of two parts: the gradient backpropagated from the previous layer and the gradient backpropagated from the previous time steps. For IndRNN, the neurons are independent from each other at each layer, and the gradient can be calculated for each neuron independently. For the gradient of the \(n\)-th neuron at time step \(t\), it can be obtained according to Equation (2) as  
\[g_{n,t}= \frac{\partial I_{n,t}}{\partial I_{n,t}}+[\frac{\partial I_{n,t+1}}{ \partial I_{n,t+1}}u_{n}^{\dagger}\prod_{k=t}^{t}\sigma^{\prime}{}_{n,k+1}+ \ldots+\frac{\partial I_{n,T}}{\partial I_{n,T}}u_{n}^{T-t}\prod_{k=t}^{T-1} \sigma^{\prime}{}_{n,k+1}] \tag{3}\]  
where \(\frac{\partial I_{n,t}}{\partial I_{n,t}}\) is the gradient of the current step, and \(\frac{\partial I_{n,t+1}}{\partial I_{n,t+1}}u_{n}^{\dagger}\prod_{k=t}^{t} \sigma^{\prime}{}_{n,k+1},\ldots,\)\(\frac{\partial I_{n,T}}{\partial I_{n,T}}u_{n}^{T-t}\prod_{k=t}^{T-1}\sigma^{ \prime}{}_{n,k+1}\) are the gradients of the future time steps. Since here we mainly focus on the gradient backpropagated through time, the small differences among the gradients backpropagated from the previous layer at different time steps are ignored, and the gradients \(\frac{\partial I_{n,t}}{\partial I_{n,t}},\frac{\partial I_{n,t+1}}{ \partial I_{n,t+1}}\ldots,\frac{\partial I_{n,T}}{\partial I_{n,T}}\) are all denoted by \(g_{o}\). In this paper, the ReLU (rectified linear unit) is considered the activation function and its gradient is either 0 or 1 based on whether the neuron is activated. Denoting the probability of a neuron being activated by \(p,\prod_{k=t}^{t-1}\sigma^{\prime}{}_{n,k+1}\) can be statistically simplified to \(p^{(l-t)}\). Thus, the above Equation (3) can be simplified as  
\[g_{n,t}=g_{o}\cdot[1+(u_{n}\cdot p)^{1}+\ldots+(u_{n}\cdot p)^{T-t}] \tag{4}\]  
where \(f_{lr}\) is the learning rate adjustment factor. With \(f_{lr}\), the magnitudes of the changes to the network parameters under gradient backpropagation through time are similar to those of the conventional feedforward network. The basic probability of the neurons activated is assumed to be \(p=0.5\) since half of the input range is activated for the ReLU function. With the recurrent connection, the neuron is more likely to be activated, as the recurrent input is usually no smaller than 0 (with positive recurrent weight). The larger the recurrent weight, the higher the probability of the neuron being activated. In the implementation, the probability of the neurons activated is set as \(0.5+0.4*u_{n}\), where the highest probability reaches 0.9 when \(u_{n}\) equal 1. Figure 4a illustrates the learning rate adjustment factors over different lengths of inputs when the recurrent weight is set to 1. It can be seen that the learning rate adjustment factor drops quickly at first, and then decays steadily around 0.1 when the length of the input is over 30 steps. To avoid the complex calculation for each recurrent neuron, the average learning rate adjustment factor is used for all the neurons. With the recurrent weight uniformly initialized in the range of \([0,1]\), the average learning rate adjustment factor can be obtained as  
\[f_{lr,race}=\sum_{u_{n}=0}^{1}T/[\frac{T-\frac{u_{n}\cdot p\cdot(1-u_{n}\cdot p )^{T}}{1-u_{n}\cdot p}}{1-u_{n}\cdot p}] \tag{7}\]  
It can be numerically obtained as shown in Figure 4b, where 1000 recurrent weights with interval 0.001 is used. Similar behavior as above for the recurrent weight 1 can be  
Figure 4: Illustration of the learning rate adjustment factor over different lengths of inputs for the first 1, 2,..., \(L-1\) IndRNN layers (learning rate adjustment factor versus length of the input). **(a)** Learning rate adjustment factors for neurons with recurrent weight 1; **(b)** average learning rate adjustment factors. observed, which is shown in Figure 4b.