## 3 Proposed Methodology
### Encoder-Decoder with Bi-LSTM and Sequential LSTM  
LSTM [52] is a distinct version from RNN. LSTMs are expressly intended to keep away from the long-term dependency issue. LSTM consists of several gates, such as the input gate, forget gate, output gate, and a constant memory cell. The LSTM based sequential model is followed by those equations where \(i_{t}\), \(f_{t}\), _and_\(o_{t}\) represent the input gate, forget gate, and output gate, respectively, in Equations (1)-(3).  
\[i_{t}=\sigma\left(W_{i}\left[h_{t-1},\ x_{t}\right]+b_{i}\right) \tag{1}\]  
\[f_{t}=\sigma\left(W_{f}\left[h_{t-1},\ x_{t}\right]+b_{f}\right) \tag{2}\]  
\[o_{t}=\sigma\left(W_{o}\left[h_{t-1},\ x_{t}\right]+b_{o}\right) \tag{3}\]  
\[g_{t}=tanh\left(W_{g}\left[h_{t-1},\ x_{t}\right]+b_{g}\right) \tag{4}\]  
\[c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot g_{t} \tag{5}\]  
\[h_{t}=o_{t}\odot\ tanh\ (c_{t}) \tag{6}\]  
Figure 1: A comprehensive framework of our proposed video description generation model. Here, (**1**) the input of the model is video. (**2**) 2D and I3D CNN are used for feature extraction from the input visual frames. (**3**) The model contains attention-based bi-directional LSTM as an encoder. (**4**) A forward pass LSTM is used for sequential sentence generation. (**5**) Lastly, the sentence generation module is used for captioning the video.  
If the input sequence \((x_{1},\ \ldots,\ x_{t})\), is given to the LSTM, it computes the hidden control unit \((h_{1},\ \ldots,\ h_{t})\), the cell memory sequence \((c_{1},\ \ldots,\ c_{t})\), and b denotes the bias vector. Here, \(\odot\) represents the element-wise multiplication, and \(tanh\) is the non-linear hyperbolic function.  
However, the sequential model of LSTM only focuses on present sequences in the temporal order. In our proposed model, we have used bi-LSTM [53] as an encoder, which not only considers the present state but also considers the future state. It improves the standard LSTM network by including another layer. The two layers are opposite in direction and can predict information both from the previous and the forthcoming by applying forward and backward pass. Here, Figure 2 illustrates the traditional LSTM [52] and bi-LSTM [53], and Equation (7) is the equation of bi-LSTM.  
\[h_{i}=\left[\overset{\rightarrow}{h_{i}}\oplus\overset{\leftarrow}{h_{i}}\right] \tag{7}\]  
Here, \(\oplus\) represents the element-wise summation to chain the onward and backward pass LSTM.  
Figure 2: An illustration of LSTM and a bi-directional LSTM model. In our proposed model, we have used bi-LSTM as an encoder and sequential LSTM as a decoder. (**a**) The illustration of an LSTM diagram. Here, the forget gate, input gate, and output gate denote as \(f_{t},\ i_{t}\), and \(o_{t}\), respectively. (**b**) The illustration of a bi-LSTM model. Two layers, such as the forward pass and backward pass LSTM, followed by the sigmoid function.  
The decoder consists of the sequential LSTM [52], which takes input from the encoder. The output \(y_{t}\) and hidden state \(h_{t}\) update based on its earlier position \(y_{t-1}\) and \(h_{t-1}\) based on time step \(t\). If the encoder embedding represents \(V\) as a context vector, then the decoder's equation as Equation (8).  
\[\left[\begin{array}{c}y_{t}\\ h_{t}\end{array}\right]=Dec\left(h_{t-1},\ y_{t-1},\ V\right) \tag{8}\]  
We propose an attention-based bi-LSTM [53] as an encoder in our framework. In the next subsection, we have discussed the attention mechanism of our encoder system.