# Ensemble of Networks for Multilabel Classification
## 4 Proposed Approaches
### Long Short-Term Memory (LSTM)  
The LSTM layer in our topologies learns long-term dependencies between the time steps in a time series and sequence data [61]. This layer performs additive interactions, which can help improve gradient flow over long sequences during training.  
LTSM can be defined as follows. Let the output or hidden state be \(\mathrm{h}_{\mathrm{t}}\), and the cell state be \(\mathrm{c}_{\mathrm{t}}\) at time step \(t\). The first LSTM block uses the initial state of the network and the first-time step of the sequence to compute the first output and updated cell state. At time step \(t\), the block uses the current state of the network (\(\mathrm{c}_{\mathrm{t-1}},\mathrm{h}_{\mathrm{t-1}}\)) and the next time step of the sequence to compute the output and the updated cell state ct.  
The state of the layer is the hidden state/output state and the cell state. The hidden state at time step \(t\) contains the output of the LSTM layer for this time step. The cell state contains information learned from previous time steps. At each time step, the layer adds to or removes information from the cell state. The layer controls these updates using gates.  
The basic components of a LSTM are an input gate, forget gate cell candidate, and output gate: the first determines the level of cell state update, the second the level of cell state reset (forget), the third adds information to cell state, and the fourth controls the level of the cell state added to the hidden state.  
Given the above and letting \(\mathrm{x}_{\mathrm{t}}\) be the input sequence with \(\mathrm{h}_{0}=0\), we can then define the input gate \(\mathrm{i}_{\mathrm{t}}\), the forget gate \(\mathrm{f}_{\mathrm{t}}\), the cell candidate \(\mathrm{g}_{\mathrm{t}}\) and the output gate \(\mathrm{o}_{\mathrm{t}}\) as:  
\[\mathrm{i}_{\mathrm{t}} =\mathrm{o}_{\mathrm{g}}(\mathrm{W}_{\mathrm{i}}\mathrm{x}_{ \mathrm{t}}+\mathrm{R}_{\mathrm{i}}\mathrm{h}_{\mathrm{t-1}}+\mathrm{b}_{ \mathrm{i}}) \tag{9}\] \[\mathrm{f}_{\mathrm{t}} =\mathrm{o}_{\mathrm{g}}(\mathrm{W}_{\mathrm{f}}\mathrm{x}_{ \mathrm{t}}+\mathrm{R}_{\mathrm{f}}\mathrm{h}_{\mathrm{t-1}}+\mathrm{b}_{ \mathrm{f}})\] (10) \[\mathrm{g}_{\mathrm{t}} =\mathrm{o}_{\mathrm{c}}\big{(}\mathrm{W}_{\mathrm{g}}\mathrm{x}_{ \mathrm{t}}+\mathrm{R}_{\mathrm{g}}\mathrm{h}_{\mathrm{t-1}}+\mathrm{b}_{ \mathrm{g}}\big{)}\] (11) \[\mathrm{o}_{\mathrm{t}} =\mathrm{o}_{\mathrm{g}}(\mathrm{W}_{\mathrm{o}}\mathrm{x}_{ \mathrm{t}}+\mathrm{R}_{\mathrm{o}}\mathrm{h}_{\mathrm{t-1}}+\mathrm{b}_{ \mathrm{o}}) \tag{12}\]  
where \(\mathrm{W}_{\mathrm{i}}\), \(\mathrm{R}_{\mathrm{i}}\), \(\mathrm{b}_{\mathrm{i}}\), \(\mathrm{W}_{\mathrm{f}}\), \(\mathrm{R}_{\mathrm{f}}\), \(\mathrm{b}_{\mathrm{f}}\), \(\mathrm{W}_{\mathrm{g}}\), \(\mathrm{R}_{\mathrm{g}}\), \(\mathrm{b}_{\mathrm{g}}\), \(\mathrm{W}_{\mathrm{o}}\), \(\mathrm{R}_{\mathrm{o}}\), \(\mathrm{b}_{\mathrm{o}}\) are matrices and vectors and \(\mathrm{o}_{\mathrm{g}}\) denotes the gate activation function and \(\mathrm{o}\)c the state activation function. The LSTM layer function, by default, uses the sigmoid function given by \(\sigma(\mathrm{x})=\left(1+\mathrm{e}^{-\mathrm{x}}\right)^{-1}\) to compute the gate activation function.  
We then define:  
\[\mathrm{c}_{\mathrm{t}}=\mathrm{f}_{\mathrm{t}}\ \odot\ \mathrm{c}_{\mathrm{t-1}}+ \ \mathrm{i}_{\mathrm{t}}\odot\ \mathrm{g}_{\mathrm{t}} \tag{13}\]  
as the cell state, where \(\odot\) is the Hadamard (component-wise) product.  
The output vector is defined as:  
\[\mathrm{h}_{\mathrm{t}}=\mathrm{o}_{\mathrm{t}}\ \odot\ \mathrm{o}_{\mathrm{t}}( \mathrm{c}_{\mathrm{t}}) \tag{14}\]The LSTM layer function, by default, uses the hyperbolic tangent function to compute the state activation function.