# Bi-LSTM-Based Neural Source Code Summarization
## 2 General Framework
### Recurrent Neural Networks  
A recurrent neural network (RNN) is a neural-based approach that is effective in processing sequential data. The RNN works recursively on the input and applies computations to all instances of the input. These are represented in a vector of tokens. The main point of the RNN is that it can memorize the result of the previous computation and apply it to the current computation. The RNN is a class of artificial neural networks in which connections between the nodes represent a directed graph such that information flows from a layer to a previous layer and allows the information to flow back into the previous parts. The RNN not only uses the input information but also whatever it is fed during the training of the network [17]. The RNN has problems such as the vanishing gradient, often called the long-term dependency problem; this problem means that the network is not able to hold information for a long time, and it rapidly loses information over time. To solve the vanishing gradient issue, two neural network approaches have been introduced: first, the gated recurrent unit (GRU), and second, LSTM. The two versions provide a memory cell to store information on previous inputs. The gates are used for controlling the flow in the network. These gates can determine what information is important and then store that information in the memory unit. The gates can pass the stored information and use it to make better predictions [18].  
(1) Gated Recurrent Units  
The GRU works like the RNN in terms of workflow but differs in the operations inside the GRU. The GRU has two gates: one is the reset gate, and the other is the update gate. The update gate decides if the current cell state should be updated with the current activation value or not. The reset gate decides if the previous cell state is important or should be removed [19].  
(2) Long-Term Short-Term Memory  
As we mentioned, in RNN, one of the challenges is its short-term memory; it is hard to carry the output from one step to the next steps for long sequences. To solve this issue, we use the LSTM, a kind of RNN that keeps in its memory the information for long periods. LSTM has three steps; first is the input gate, then the forget gate, and lastly, the output gate. The input gate decides what information is to be added to the cell state from the input, the forget gate decides which data to forget and which data to remember, and the output gate decides which parts in the current cell go to the output [20].  
(3) Encoder-Decoder Architecture  
Sequence-to-sequence (seq2 seq) prediction problems often involve predicting the next value in a real-valued sequence or a label for an input. One of the challenges in sequence-to-sequence prediction problems is that the length of the input sequences and the output sequences may vary; this problem is called a many-to-many prediction problem. To solve the problem in seq2 seq, the encoder-decoder architecture has been proposed. The architecture is composed of two components: the encoder, which is responsible for reading the input sequences and encoding them into a fixed length vector, and the decoder, for decoding the fixed length vector back to a variable length vector and then predicting the output sequence [21].