# From General Language Understanding to Noisy Text Comprehension
## 5 Results
Further, the hidden layers that contribute to increasing the sentence vectors' richness for the semantic information task are found at the border of R2 and R3. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multicolumn{2}{c}{**Pooling**} & \multicolumn{6}{c}{**Logistic Regression**} & \multicolumn{6}{c}{**MLP**} \\ \hline
**Layer** & **Token** & **SL** & **WC** & **BS** & **TD** & **SM** & **SL** & **WC** & **BS** & **TD** & **SM** \\ \hline \multirow{3}{*}{concat} & max & 50.51 & 55.99 & 69.50 & 52.53 & 69.33 & 57.35 & 58.03 & 69.38 & 57.11 & 63.00 \\  & mean & 53.87 & 67.68 & **73.27** & 52.00 & 71.71 & 64.65 & 70.63 & 72.61 & 56.85 & 71.80 \\  & mean & max & **55.83** & **69.21** & 72.27 & 53.72 & 71.85 & 63.66 & **71.85** & 72.30 & 57.91 & 72.17 \\ \hline \multirow{3}{*}{mean} & max & 48.85 & 51.32 & 67.97 & 51.78 & 69.42 & 56.77 & 55.08 & 67.89 & 56.50 & 71.43 \\  & mean & 54.28 & 66.67 & 72.72 & 52.62 & **72.81** & 63.02 & 69.41 & 72.02 & **58.96** & 71.34 \\  & mean & max & 54.63 & 68.50 & 71.77 & **54.07** & 72.26 & 63.02 & 70.33 & 71.74 & 56.80 & **72.31** \\ \hline \multirow{3}{*}{sum} & max & 47.65 & 50.00 & 66.81 & 49.54 & 65.75 & 53.87 & 50.20 & 68.83 & 55.57 & 70.43 \\  & mean & 53.52 & 66.36 & 72.44 & 51.34 & 72.03 & 61.53 & 70.53 & **72.94** & 53.50 & 71.25 \\ \cline{1-1}  & mean & max & 53.28 & 68.19 & 70.47 & 52.66 & 71.62 & **65.06** & 70.94 & 71.33 & 57.42 & 72.03 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Mean classification accuracy for sentence vectors derived using different pooling strategies (SL: SentLen, WC: WC, BS: BShift, TD: TreeDepth, SM: SOMO). \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Vector Size**} & \multirow{2}{*}{**Region**} & \multicolumn{6}{c}{**Logistic Regression**} & \multicolumn{6}{c}{**MLP**} \\ \cline{3-11}  & & **SL** & **WC** & **BS** & **TD** & **SM** & **SL** & **WC** & **BS** & **TD** & **SM** \\ \hline \multirow{3}{*}{768} & 1 & 53.29 & 70.96 & 67.36 & 51.29 & 68.95 & 61.11 & 73.48 & 68.13 & 55.65 & 69.26 \\  & 2 & 52.54 & 58.69 & 71.47 & 51.75 & 70.29 & 59.91 & 60.98 & 71.38 & 57.17 & 70.94 \\  & 3 & 47.40 & 46.11 & 71.13 & 50.93 & 70.77 & 55.37 & 49.47 & 71.75 & 55.58 & 73.14 \\ \hline \multirow{3}{*}{1536} & 1 & 56.75 & 80.18 & 69.45 & 55.42 & 70.15 & **68.41** & 84.15 & 68.99 & 56.61 & 70.09 \\  & 2 & 56.18 & 70.58 & 73.77 & 53.57 & 72.70 & 65.65 & 71.19 & **74.19** & **58.45** & 71.73 \\  & 3 & 48.95 & 54.27 & 70.15 & 51.12 & **72.97** & 58.07 & 56.56 & 71.44 & 56.28 & **74.69** \\ \hline \multirow{3}{*}{3072} & 1 & 54.74 & 73.48 & 69.28 & 51.79 & 68.23 & 63.19 & 74.70 & 69.15 & 56.54 & 59.22 \\  & 2 & 53.42 & 61.74 & 72.53 & 53.10 & 70.77 & 61.49 & 65.40 & 72.40 & 58.39 & 70.70 \\  & 3 & 48.42 & 50.31 & 72.36 & 51.92 & 72.56 &Overall, in the context of noisy texts, the hidden layers in the region R1 contain most of the linguistic characteristics required to address probing tasks in the surface group. In contrast, the syntactic and semantic group tasks are able to identify necessary linguistic patterns from R1 and R2. Nevertheless, the sentence vectors' performance derived from hidden layers in the last region (R3) ranges from low to marginal, indicating their inability to capture linguistic information from noisy texts. Overall accuracy:  
Table 10 presents the classification accuracies for probing tasks with sentence vectors derived from GloVe-based pre-trained models, Sentence-BERT and using different hidden layers from the BERTBASE-uncased model. In the context of BERT-based sentence vectors, we have considered sentence vectors derived based on the last hidden layer, the last four hidden layers, and all 12 layers. [PERSON] et al. [4] achieved comparable results for feature-based by using those layers as input to an artificial recurrent neural network. Based on our findings, we propose two separate approaches for noisy texts. The first is based on BERT's first hidden layer, while the second combines the first hidden layer of each BERT region, i.e., layers 1, 5 and 9 (1-5-9).