# From General Language Understanding to Noisy Text Comprehension
## 5 Results  
This section first analyses the effectiveness of the proposed pooling strategies: layer pooling and token pooling. Next, we analyze the distribution of the language understanding (surface, syntactic and semantic) across the various regions of the BERT model proposed for this study. Finally, we analyze the performance of the sentence vectors generated by combining these findings along with the existing sentence vector generation mechanisms, including the state-of-the-art techniques. Pooling strategy analysis:  
For this study, we consider sentence embeddings derived using all four layers of each BERT region. Table 7 shows the resulting sentence vector sizes for each combination of layer and token pooling strategies when applied to four hidden layers of BERT. The CONCAT-layer-strategy and MEAN-MAX-token-strategy significantly increase the resulting sentence vector size, by four times and two times, respectively. From the results shown in Table 8, we note that the Logistic Regression model achieves the best results with sentence vectors of size 6144, whereas the MLP model achieves the best results, in most cases, with 1536 vector size. From this, it becomes evident that simpler models, such as Logistic Regression, require huge sentence vectors to identify linguistic patterns, while complex models can achieve improved results with significantly lower-sized sentence vectors. Similarly, Table 9 shows that the Logistic Regression model achieves, in most cases, the best accuracy with the CONCAT-layer-strategy. However, one of the syntactic information groups' tasks and the semantic information task obtains the best results with the MEAN-layer-strategy. On the other hand, the MLP model performs satisfactorily with the MEAN-layer-strategy and SUM-layer-strategy. Both logistic regression and the MLP models prefer the MEAN-MAX-token-strategy or MEAN-token-strategy, while MAX-token-strategy performs poorly across all the performing tasks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **LR** & **MLP** \\ \hline nhid & 0 & 200 \\ optimizer & ‘adam’ & ‘adam’ \\ batch\_size & 32 & 32 \\ tenacity & 5 & 5 \\ epoch\_size & 4 & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Parameters for the Classifiers. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer Pooling** & **Token Pooling** & **Embedding Size** \\ \hline \multirow{3}{*}{concat} & max & 3072 \\  & mean & 3072 \\  & mean\_max & 6144 \\ \hline \multirow{3}{*}{mean} & max & 768 \\  & mean & 768 \\  & mean\_max & 1536 \\ \hline \multirow{3}{*}{sum} & max & 768 \\  & mean & 768 \\ \cline{1-1}  & mean\_max & 1536 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Sentence vector sizes derived using different pooling strategies with four hidden layers. In the rest of the analyses, the results derived with the MEAN-layer-strategy and MEAN-token-strategy using the MLP classifier are used. This enables easy comparisons of the BERT based sentence embeddings with vectors derived from static pre-trained models by calculating the average of the word embeddings. Further, Sentence-BERT internally uses the mean of the token embeddings to generate sentence embeddings. Region-wise analysis:  
Figure 3 shows a heat map of the accuracies (darker colors equate to higher accuracy) of each probing task with sentence vectors generated using each hidden layer of the BERT model. The SentLen and the WC tasks in the Surface Information group achieves better accuracy with sentence vectors derived from hidden layers in the first region (R1), and the performance gradually decreases as we move toward the last layers of the BERT model. On the other hand, higher accuracies are obtained for the syntactic information tasks--BShift and TreeDepth--with the sentence vectors generated using the hidden layers from the second region (R2). The initial layers of the R2 show the most contribution to the accuracy, while the hidden layers from the R1 contribute poorly to the syntactic information group tasks.