# From General Language Understanding to Noisy Text Comprehension
## 3 Methodology
### Sentence Vector Generation
#### 3.1.1 Sentence Representations Using Multi-Layer Pre-Trained Language Models  
An input sentence is represented as a set of input tokens \(T=[t_{0},t_{1},\ldots,t_{n}]\), where \(t_{0}\) is the special \([CLS]\) token that needs to be prepended for the out-of-the-box pooling schema to work. BERT produces a set of hidden layer activations \(H^{0},H^{(1)},\ldots,H^{(L)}\), where \(H^{(l)}=[h^{(l)}_{0},h^{(l)}_{1},\ldots,h^{(l)}_{n}]\) are the activation vectors of the \(l\)th hidden layer. We have ignored the \(H_{0}\), which consists of non-contextual word-piece embeddings, to generate sentence representations.  
To generate a sentence representation based on multiple hidden layers, we propose to generate token representation vector \(w_{i}\) for each token \(t_{i}\) in \(T\), using a _layer pooling_ strategy. A layer pooling strategy combines different representations of the same token across multiple hidden layers. For this, three layer pooling strategies are studied: (i) SUM-layer-strategy, (ii) MEAN-layer-strategy, and (iii) CONCAT-layer-strategy. The SUM-layer-strategy and the MEAN-layer-strategy calculate the sum and mean of all the activation vectors \(h_{i}\in\mathbb{R}^{d}\) of the selected hidden layers, respectively, producing \(w_{i}\in\mathbb{R}^{d}\), where \(d\) is the size of the hidden vector \(h\). Thus, for each sentence, the Mean-layer-strategy and SUM-layer-strategy produce a matrix \(W\in\mathbb{R}^{n\times d}\). On the other hand, the CONCAT-layer-strategy concatenates the corresponding hidden activation vectors \(h_{i}\) in the order of the layer numbers to generate \(w_{i}\in\mathbb{R}^{kd}\), where \(k\) is the number of BERT layers selected to generate the sentence representation. The CONCAT-layer-strategy produces a sentence representation \(W\in\mathbb{R}^{n\times nd}\).  
Then, to derive the sentence vector \(S=[s_{1},s_{2},\ldots,s_{\|w_{i}\|}]\), we apply multiple _token pooling_ strategies for the sentence representation \(W\) (obtained after applying the layer pooling strategy), where each token representation \(w_{i}\) is a row. A token pooling strategy merges all the token embeddings of a sentence into a singe vector. For this, we study two token pooling operations: (i) MEAN-token-strategy, and (ii) MAX-token-strategy. MEAN-token-strategy and MAX-token-strategy are calculated as \(s_{j}=\mathop{\mathbb{E}}_{1\leq j\leq n}W_{ij}\) and \(s_{j}=\mathop{\max}_{1\leq j\leq n}W_{ij}\), respectively. Further, the MEAN-MAX-token-strategy we propose concatenates the MEAN  
Figure 1: The twelve layers (L1 to L12) of the BERTBASE model are partitioned into 3 regions. R_n_-_i_ represents the \(i\)th layer in the \(n\)th region.  
token-strategy output vector and the MAX-token-strategy output to derive a sentence vector twice the size of \(w_{i}\).  
As shown in Figure 1, for each region R_n_ (\(n\in 1,2,3\)), different combinations of four layers are considered to generate sentence embeddings. We apply the layer pooling and token pooling strategy combinations listed in Table 1 across each BERT region _Rn_ to systematically generate a diverse set of sentence embeddings, using the pre-trained BERT model.