# From General Language Understanding to Noisy Text Comprehension
## 3 Methodology
### Noisy Probing Datasets  
Probing datasets have a crucial role in the proposed study, as they validate the model's ability to comprehend linguistic characteristics. Studies reported earlier (e.g., [19]) have focused only on language comprehension of structured and grammatical sentences. Hence, the existing probing datasets [19] contain structured and grammatical sentences and rely on the pre-trained Probabilistic Context-Free Grammar (PCFG) model [38] and part-of-speech, constituency, and dependency parsing information provided by the Stanford Parser. Although the PCFG model reported close to 87% accuracy for regular English sentences, it is poorly suited for noisy texts [39; 40]. Further, the available Twitter-specific dependency parsers reported a low overall accuracy level with further reductions if the test set topics differed from the training dataset. Thus, the use of automatic part-of-speech or automatic dependency parsing as suggested by [19] is not a feasible option for noisy probing datasets. Hence, we propose to use a noisy dataset manually annotated with the required linguistic labels to generate quality probing datasets.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layers** & **No. of Layers** & **Layer Pooling** & **Token Pooling** \\ \hline R_n_-1 & 1 & — & mean, max \\ R_n_-2 & 1 & — & mean, max \\ R_n_-3 & 1 & — & mean, max \\ R_n_-4 & 1 & — & mean, max \\ R_n_-1, R_n_-2 & 2 & sum, mean, concat & mean, max \\ R_n_-3, R_n_-4 & 2 & sum, mean, concat & mean, max \\ R_n_-1 to R_n_-4 & 4 & sum, mean, concat & mean, max \\ \hline \hline \end{tabular}
\end{table}
Table 1: Strategy to generate sentence embeddings from each region (ref. Figure 1) of the BERT model. R_n_-_i_ represents the _i_th layer in the _n_th region. We combine each layer pooling strategy with every token pooling strategy across identified layers to generate multiple sentence embeddings. Layer pooling is not applicable for the sentence embeddings generated using a single vector.  
For this, we use \"Tweebank v2\", a collection of English Tweets [41], annotated in Universal Dependencies [42], as it can be exploited to generate the required noisy probing datasets. Authors of [41] followed a rigorous two-stage process to develop 3550 manually labeled Tweets. They automatically annotated the Tweets, using a parser trained on a sample set of Tweets manually annotated in the first stage. In the second stage, they manually corrected the parsed data. These high-quality labels are crucial to developing gold standard probing datasets for noisy text data. However, this research [41] did not focus on specific aspects of linguistics, such as dependency parsing information. Due to the unavailability of these linguistic labels, we are focusing only on a selected subset of probing tasks out of the ten probing tasks proposed by [19]. Nevertheless, the selected probing tasks continue to cover the three important linguistic categories (i.e., surface, syntactic and semantic), thereby enabling us to analyze the richness of the sentence vectors across all three levels of linguistic information and ensuring the quality of the findings. Further, we introduce additional criteria explained below to adapt the dataset to noisy conditions. The probing tasks that are focused on in this study are explained in the following sections.