# BTCAM: Attention-Based BiLSTM for Imbalanced Classification of Political Inquiry Messages
## 3 Methodology
### Framework Overview of BTCAM
#### 3.2.3 Context-Information-Extraction Layer  
The embedding layer obtains the representation of each token but ignores the meaning of the word and its relationship to other words in the context. Therefore, we use bidirectional long short-term memory (BiLSTM) to extract contextual information, which consists of a series of memory units. Each memory block has three gates and one storage unit. The input gate and output are used to update the cell state and obtain the following hidden state, respectively, while the forget gate employs the sigmoid function to determine whether information should be maintained or deleted. Implementations of LSTM memory cells include the following:  
\[f_{t}=\text{sigmoid}\left(W_{xf}x_{t}+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_{f}\right) \tag{2}\]  
\begin{table}
\begin{tabular}{l l} \hline
**Operation** & **Specific Method** \\ \hline Topic-term insertion (TI) & Randomly select a word from a sentence that belongs to the set of categorical themes and obtain its random synonym to insert anywhere in the sentence. \\ Topic-term replacement (TR) & Randomly select n words from the sentence that belong to the set of categorical themes and replace them by \\ Random swap (RS) & After the positions of two words in a sentence randomly. \\ Insertion punctuation (IP) & Insert n punctuation marks from \(\{\)\(\  
\[i_{t} =\text{sigmoid}(W_{xi}x_{t}+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b_{i}) \tag{3}\] \[c_{t} =f_{t}c_{t-1}+i_{t}\tanh(W_{xc}x_{t}+W_{hc}h_{t-1}+b_{c})\] (4) \[o_{t} =\text{sigmoid}(W_{xo}x_{t}+W_{ho}h_{t-1}+W_{co}c_{t-1}+b_{o})\] (5) \[h_{t} =o_{t}\tanh(c_{t}) \tag{6}\]  
where \(x_{t}\) represents the input at \(t\) seconds. \(h_{t}\) is the hidden layer representation at t seconds. \(c_{t}\) represents the cell state representation at time \(t\) seconds. \(W_{xf}\), \(W_{xi}\), \(W_{xo}\) represent the weight coefficients of the forget gate, input gate, and output gate of \(x_{t}\), respectively. \(W_{hf}\), \(W_{hi}\), \(W_{hc}\), \(W_{ho}\) represent the forgetting gate, input gate, cell state, output gate, and weight coefficient of \(h_{t-1}\), respectively. \(b_{f},b_{i},b_{c},b_{o}\) represent the forgetting gate, the input gate, the cell state, and the bias value of the output gate in the feature extraction process, respectively.  
In this paper, we employ bidirectional LSTM as in [36], which combines forward LSTM with backward LSTM and preserves data information in both directions. The output can be portrayed as follows:  
\[\overrightarrow{h_{t}} =\overrightarrow{LSTM}\Big{(}x_{t},\overrightarrow{h_{t-1}} \Big{)} \tag{7}\] \[\overleftarrow{h_{t}} =\overleftarrow{LSTM}\Big{(}x_{t},\overleftarrow{h_{t-1}} \Big{)} \tag{8}\]  
Concatenate each \(\overrightarrow{h_{t}}\) and \(\overleftarrow{h_{t}}\) to obtain a hidden state \(h_{t}\). Let the hidden unit number for each unidirectional LSTM be \(u\). For simplicity, denote all the \(h_{t}\) as \(H\in R_{n\times 2u}\):  
\[h_{t}=\overrightarrow{h_{t}}\oplus\overleftarrow{h_{t}} \tag{9}\]  
\[H=(h_{1},h_{2},\ldots h_{n}) \tag{10}\]  
where \(\oplus\) denotes the concatenation operation.