# Towards Automatic Depression Detection:
## 3 Proposed Approach and Models
### Bidirectional Long Short-Term Memory with Attention  
The Recurrent Neural Network (RNN) was designed for processing sequence data. The strength of RNN lies in its ability to discover time relevance and logical characteristics by passing the output to the hidden layer again. As a variant of RNN, LSTM adopts the gate mechanism to solve the problem of the  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Layer Name**} & \multicolumn{2}{c}{**Parameter Settings**} & \multirow{2}{*}{**Activation**} \\  & **(Classification/Regression)** & & \\ \hline \multirow{3}{*}{Conv1} & Kernel: [1; 7] & \multirow{3}{*}{ReLU} \\  & Stride: 1 & \\  & Filter size: 32 & \\ \hline \multirow{3}{*}{Max-pool1} & Kernel: [4; 3] & \multirow{3}{*}{ReLU} \\  & Stride: [1; 3] & \\ \hline \multirow{3}{*}{Conv2} & Kernel: [1; 7] & \multirow{3}{*}{ReLU} \\  & Stride: 1/2 & \\  & Filter size: 32 & \\ \hline \multirow{3}{*}{Max-pool2} & Kernel: [1; 3] & \multirow{3}{*}{ReLU} \\  & Stride: [1; 3] & \\ \hline FC1 & Out features: 128 & ReLU \\ \hline FC2 & Out features: 128 & ReLU \\ \hline dropout & 0.5 & \\ \hline FC3 & Out features: 2/1 & \multicolumn{1}{c}{Softmax/Linear} \\ \hline \hline \end{tabular}
\end{table}
Table 1: Parameter settings of the 1D CNN model.  
vanishing gradient caused by the long-term dependency. BiLSTM enables the model to learn not only from the past time steps, but also from the future time steps. Since we wish to extract the characteristics of response sequences in the interview, we utilized BiLSTM and additionally adopted an attention layer to emphasize which sentence contributes most to the depression detection problem.  
Attention is defined in Equation (2), where \(\mathbf{X}\) is the input sentence embeddings with size \((seq\_lens,embed\_dim)\), \(\mathbb{O}\) is the output of BiLSTM and is of size \((seq\_lens,hidden\times layers)\), _hidden_ and _layers_ are the hyperparameters of BiLSTM, \(\mathbb{O}_{f}\) and \(\mathbb{O}_{b}\) are the forward and backward outputs of BiLSTM, respectively, \(\mathbb{O}\) is the concatenation of \(\mathbb{O}_{f}\) and \(\mathbb{O}_{b}\), \(\mathbf{O}\) is the sum of \(\mathbb{O}_{f}\) and \(\mathbb{O}_{b}\) and is of size \(seq\_lens\times hidden\), \(\mathbf{w}\) is the weight vector of length _hidden_ and is generated from a fully connected layer, \(W\) and \(b\) are the parameters of the fully connected layer, \(c\) is the weighted context, and \(y\) is the final output with attention.  
\[\mathbb{O},\mathbf{H} =\textbf{BiLSTM}(\mathbf{X}) \tag{2}\] \[\mathbb{O} =[\mathbb{O}_{f},\mathbb{O}_{b}]\] \[\mathbf{O} =\mathbb{O}_{f}+\mathbb{O}_{b}\] \[\mathbf{w} =W\times\textbf{O}+b\] \[\mathbf{c} =tanh(\mathbf{O})\times\mathbf{w}\] \[y =\mathbf{O}\times\mathbf{c}\]  
Equation (2) defines a simple per-sequence attention mechanism.  
The detailed configuration of the proposed BiLSTM model with an attention layer is summarized in Table 2. The \"Out features\" of FC1 and FC2 in Table 2 represent the number of nodes in the fully connected layers FC1 and FC2. The model consists of two BiLSTM layers, followed by an attention layer to calculate the weighted representations. The output of the attention layer is fed into a two-layer FC network to produce a prediction of whether the participant is depressive or the severity of depression.