BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text
Authors: Aarohi Srivastava
David Chiang
Venue: EMNLP 2023 Findings (2023)
Keywords: BERT
language modeling
dialects
noisy text
fine-tuning
TL;DR: We introduce the novel idea of sandwiching BERT’s encoder stack between additional encoder layers trained to perform masked language modeling on noisy text in order to extend BERT’s capabilities to modeling dialectal and noisy text.

Abstract: Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation or noise. How do we push BERT’s modeling capabilities to encompass nonstandard text? Fine-tuning helps, but it is designed for specializing a model to a task and does not seem to bring about the deeper, more pervasive changes needed to adapt a model to nonstandard language. In this paper, we introduce the novel idea of sandwiching BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. We find that our approach, paired with recent work on including character-level noise in fine-tuning data, can promote zero-shot transfer to dialectal text, as well as reduce the distance in the embedding space between words and their noisy counterparts.

BERTwich:
Extending BERT's Capabilities to Model Dialectal and Noisy Text
Aarohi Srivastava and David Chiang
Computer Science and Engineering University of Notre Dame Notre Dame, IN, USA (asrivas2, dchiang) @nd. edu
Abstract
deteriorates in the face of noisy text and develop novel methods to improve BERT's modeling of noisy text.
Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation noise. How do we push BERT's modeling capabilities to encompass nonstan- dard text? Fine-tuning helps, but it designed for specializing model to task and does not seem to bring about the deeper, more pervasive changes needed to adapt model to nonstan- dard language. In this paper, we introduce the novel idea of sandwiching BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. We find that our approach, paired with re- cent work including character-level noise in fine-tune tine-tunin data, can promote zero-shot transfer dialectal text, well reduce the distance in the embedding space between words and their noisy counterparts.
Studies have shown that humans can easily read text with high amounts of noise (Landauer et al., 1997). It is curious, then, that BERT's accuracy on downstream tasks drops dramatically when noise is added test data (Kumar al., 2020; Yin al., 2020; Aspillaga al., 2020). This seeming contra- diction comes down to the way in which BERT's textual input expressed (subword tokenization) and the resultant units to which BERT assigns contextual embeddings (subwords). For instance, take the sentence and corresponding tokens after applying the BERT-base uncased tokenizer:
I am a student ~ I, am, a, student If there were single typo in the topical cue stu- dent, the corresponding tokens would be as follows:
I am a studebt ~ I, am, a, stud, eb,
1 Introduction
The subword tokenizer's rigidity manifests in two ways: over-segmentation and subword replacement (Kumar et al., 2020; Soper et al., 2021). In the presence of noise, words get over-segmented into shorter-length tokens (stud, eb and above), which may have less meaningful embeddings or even change the meaning of the sentence (stud above). These two roadblocks hinder the model from riving at the correct meaning of noisy sentence, particularly when, as above, the noise falls on a word with important content. In fact, we find that the distance between the vector encodings of words and their noisy counterparts is extremely high BERT's embedding space, and we aim to minimize this distance (Table 5).
Pre-trained language models such as BERT (Devlin et al., 2019) contain large amounts knowledge within their numerous parameters but require rel- atively low computational power to fine-tune on a downstream task, making them the state of the art for wide range of popular natural language processing (NLP) tasks. However, evaluation set- tings and metrics are not always representative of real world use cases. One recurring problem the severe deficit performance when BERRT-based models are used with noisy text.
We define noisy text as any text that deviates from the pre-training data of the model by way of ortho- graphic and/or grammatical variation. The possible diverse and include dialect variation, user errors (typos, misspellings, grammatical errors), and transcription errors (prop- agated from errors optical character recognition or automatic speech recognition systems). We aim to understand how BERT's modeling effectiveness
sources of this variation
In this work, we aim to make BERT-based models more robust the issues of over-segmentation and subword replacement by exposing the models text with induced character-level perturbations various stages of training. Previous efforts (see Section 2) have demonstrated the effectiveness

---

classifier
LF
LF
New BERT Encoder Stock
New BERT Encoder Stack
BERT Encoddr Stock
BERT Encoder Stock
BERT Encoces Slock
continue MLM
fine-tune on task with noised dlataset
pre-training on noisy Wiki fext
LO
LO
Figure I: A sketch of our method, BERTwich. We sandwich BERT's s encoder stack between prepended and appended randomly-initializee encoder layers, LO and LF. We continue the masked language modeling pre-training this expanded encoder stack synthetically-noiee downstream task with noised fine-tuning data to evaluate the model in settings with dialectal variation or noise in text.
Wikipedia text. The BERTwich model can be fine-tuned
introducing noise either during training or fine- tuning. But here we face conundrum. On the one hand, training on noisy data from scratch is not practical with larger language models. On the other hand, fine-tuning is designed for more task-specific learning rather than pervasive changes BERT's language modeling capabilities.
shortcr-length tokens to the existing representation of student.
Related Work
The detrimental effect of noisy test data on BERT's performance is well-documented (Kumar et al., 2020; Yin et al., 2020; Aspillaga et al., 2020; Wu et al., 2022). innthetically-indueedd errors (ty- pos, misspellings, grammatical errors, and lexical changes) significantly decrease performance across medley of tasks and test cases. Moreover, past findings exhibit a large performance gap when eval- uating on standard and nonstandard varieties of language (Aepli and Sennrich, 2022; Srivastava and Chiang. 2023; Held et al., 2023). As discussed above, the source of these issues boils down to the rigid relationship between subword tokenization and resultant subword token embeddings (Kumar et al., 2020; Soper et al., 2021).
In this paper, we propose that the way out this conundrum is to include additional Transformer en- coder layers (prepended and/or appended to BERT's encoder stack) before continuing pre-training the full model for a few more epochs synthetically- perturbed Wikipedia text (Figure 1). We call this method BERTwich.
We evaluate our method nonstandard text (from language varieties dialects unseen during training) as well synthetically-noised text (i.c.. typo simulation). We find that sandwiching BERT between added layers optimized for noisy text mod- eling improves BERT's performance on text from unseen dialects and reduces the distance in the embedding space between words and their noisy counterparts, as well as the distance between sen- tence representations of standard and nonstandard text. Our results indicate that without the additional layers, the model would have to re-learn represen- tations of stud, eb, and that are equivalent to the representation of student, while our method makes it easier for the model to simply learn to map the
Three general approaches have been taken past work to address this issue of noisy text model- ing. Some have tried to tackle it as a text normal- ization problem during preprocessing (Han et al., 2013; Supranovich and Patsepnia, 2015; Benamar et al.. 2021; Demir and Topeu, 2022). Others have attempted get to the root of the problem by improving the tokenization algorithm or using a character-basee model (Hofmann et al., 2021; Lee and Shin, 2021; Tay et al., 2021; Wang et al., 2021).

---

The third approach is inject synthetic noise in training, which we find to be the most flexible and generalizable while requiring a low computational load, and our method fits in this final category of solutions. By exposing the model to noise during some stage of training, the model is expected to be better-equipped to deal with noisy text during infer- ence. For instance, Karpukhin et al. (2019) include synthetic noise while training machine translation systems and find performance improvements in evaluation settings with natural noise; however, the approach depends on training a model from scratch rather than adapting an existing model like BERT. Using this approach in fine-tuning, past work has found augmenting the task training data with noisy or adversarial examples and fine-tuning the model with the augmented dataset to provide improve- ments in performance on noisy text (Vaibhav et al.. 2019; Yin al., 2020).
tuning, this does not speak to truly extending the model's capabilities and allowing it better model noisy text.
Starting with a standard pre-trained model such as BERT, our approach has two stages: 1) the BERTwich method: add untrained layers and con- tinue pre-training on noisy data (Sections 3, to 3.3), and (2) fine-tune on specific task (Section 3.4).
3.1 Model Variations
We propose a set of BERT variants with modified encoder stacks and additional pre-training in or- der to better model noisy text. After pre-training, expand BERT-base by prepending and append- ing additional layers that are identical to the 12 existing layers, but are randomly initialized. Our experiments include four variations (see Figure 1);
1. BERT + Layer 0 (LO + BERT): a blank encoder layer (with randomly initialized parameters) prepended to the bottom of BERT's encoder stack, resulting in 13 encoder layers.
Applying this approach to dialects, Acpli and Sen- mrich (2022) inject synthetic character-lewe noise in continued pre-training and fine-tuning steps for various tasks and find that part-of-speech tagging on dialectal text improves. Followup work by Blaschke al. (2023) develops methods of predicting the effectiveness of cross-lingual transfer based on the amount of noise injected. Srivastava and Chiang (2023) extend the method to strictly zero-shot set- tings and apply it to various sequence classification tasks, finding that the approach works well in tasks
2. BERT + Top Layer (BERT LF): blank encoder layer appended the top of BERT's encoder stack, resulting in encoder layers.
3. BERT in a Sandwich (LO + BERT + LF): two blank encoder layers, one prepended and one appended to BERT's encoder stack, resulting in 14 encoder layers.
that can be solved using surface-level cues, but it is not helpful in more challenging tasks, indicating that more can be done to truly improve the modeling of nonstandard text.
4. Top-Heavy BERT (BERT LFx2): two blank encoder layers appended BERT's encoder stack, resulting in 14 encoder layers.
A related but distinct approach involves adding adapter layers tuned to nonstandard text (Pfeiffer et al., 2020; Held et al., 2023). Our method is simi- lar this work when it comes to adding parameters the model beyond those used in fine-tuning. How- ever, the methods differ in terms of their function- ality; while adapter-basee based approaches are typically specific to a set of language varieties, our approach provides general improvement to the language mod- eling capabilities of a BERT-like model with wider applicability to various dialects and to noisy text.
We follow Devlin et al. (2019)'s original imple- mentation to randomly initialize the new encoder layers.
3.2 Continued Pre-Training
Each of the four model variants then undergoes continued pre-training (CPT) on synthetically per- turbed text. For comparison, we also perform CPT on BERT-base without making any modifications to the architecture. Our continued pre- mre-training ap- proach is as follows: taking small random sample of the Wikipedia corpus (approximately 10%), we introduce synthetic character-level noise in the text (see Section 3.3 and 4.1 for details). We train each model on the synthetically with the masked language modeling objective (with 15% of tokens masked), adopting the same approach
3 Method
As demonstrated by previous work, fine-tuning is a powerful method of domain adaptation and cross- lingual transfer. However, while the model may adapt to performing a task on noisy text via fine-
Wikipedia text

---

was used in the original implementation of BERT pre-training (Devlin al., 2019).
with two copies of the fine-tuning data: one is the original copy, and one is noised (Srivastava and Chiang, 2023). We fine-tune the models five times using one of five distinct noise levels to prepare the noised copy: 0% (baseline), 10%, 20%, 30%, or 40%.
3.3 Noising Technique
Our noising technique draws from past work (Karpukhin et al., 2019; Aepli and Sennrich, 2022; Srivastava and Chiang, 2023). We define a word as a substring comprised only of letters (identified us- ing Python's isalpha function). We use four noise operations, as done by Karpukhin et al. (2019): in- sertion, deletion, replacement, and swapping. Noise is applied at positions that are part of a word in the input string. We leave non-w non-words (for example, numbers, symbols, and punctuation) unchanged, as we expect modeling capabilities to primarily be affected by linguistic content.
Experiments
We conduct experiments sentiment analysis in English with simulated typographical errors, and intent classification in German across multiple di- alects.
4.1 Data and Tasks
Below, we describe the data used for the contin- ued pre-training, fine-tuning, and evaluation of our models. We work with the English and 2erman uncased BERT-base models.
For each letter in word of the input, noise is applied with probability p. We refer to p, expressed percentage, the noise level. When applying noise, all four noise operations have equal proba- bility. For the insertion and replacement operations, we randomly select the additional character from the alphabet of the language of the text (e.g.. umlaut vowels and eszett are included for German). All random selections are uniform within the set of pos- sibilities. In our continued pre-training, the noise level p is set to 5% per preliminary experimentation. We vary the noise level in fine-tuning O, 10, 20, 30, and 40%.
English For continued pre-training, take a randomly sampled subset of Wikicorpus," which consists of text from English Wikipedia articles. For fine-tuning, we use the Stanford Sentiment Treebank (SST) (Socher et al.. 2013), a binary sentiment analysis task that is part of the GLUE benchmark (Wang et al., 201 8). The training data consists of 67,300 examples, and the evaluation data consists of 872 examples. In addition evaluating the models on the SST data itself, we simulate noisy settings in the evaluation data. Specifically, we simulate a one-typo-per-wont test scenario using the typo simulation of Naik et al. (2018): for each word in the test data, we select random character to replace with a letter adjacent to it (left or right) on the standard QWERTY keyboard.
Below, we include few possible results after applying our noising technique at varying noise levels to an example sentence:
0%: colorless green ideas sleep furiously
10%: colorless green ideas sloeep furiously
20%: colorjless geen ideas sleep fruiouszy
German For continued pre-training, we take a randomly-sample subset of the most recent Ger- man Wikipedia dump, 4 which consists of text from German Wikipedia articles. For fine-tuning, the German intent classification subset of xSID (van der Goot et al., 2021), 1221),a a benchmark for cross- lingual slot and intent detection. The xSID dataset was drawn from the English Snips (Coucke et al., 2018) and cross-lingual Facebook (Schuster et al.,
30%: ccozorsless greenidesa sleep urruosly
40%: colorises green izeas jsee furkizusly
3.4 Fine-Tuning
After doing continued pre-training, we add linear fine-tuning layer, and we fine-tune the resultant model on one of the tasks described in Section 4.1. Following Srivastava and Chiang (2023), in effort not only include noise in the fine-tuning, but also expose the model to multiple variations in the spelling and tokenization of the same words, we use a joint composition to set up the fine-tuning data. the joint composition, the model is fine-tuned
2019) datasets and translated to the other lan- guages. The training data (German subset) consists
"https://huggingfest http://wg.w....aaeeeeeeeeeeeeeee no/bert-base-uncased 2h httts::///uggingffcce co/ co/dbndz/ bert- -e---aae--ereen---uneeeeedd
Phttps://hugginggaae
co/datasets/wikine
viewer/raw.en/tee train
"https://dumps, wi wikimedia. http://wwa..///wik////k

---

Fine-Tuning Noise Level
Model
CPT
0%
10%
20%
30%
40%
BERT
no
51.5 +1.7
62.6 =0.6
66.1
42.6
LO BERT
no
53.2 +4.7
63.8 #3.1
65.2
66.1
BERT
no
51.3 +2.0
6.22
#4.1
61.6
LO BERT BERT 1.F22
no
51.4
62.4
64.5
65.8 =2.0
+2.5
no
52.4 43.2
+1.6
64.0 +2.2
66.9 #2.1
65.9 +2.0
BERT
yes
56.4
#3.1
70.0
72.7 #1.
73.8
#1.8
1.0 + BERT
yes
57.1
42.3
70.6
73.3 +2.0
74.2
74.7 +1.8
BERT LF
yes
56.7
+4.8
70.9 +2.0
72.7 #2.2
72.8 =0.8
73.0
+00+10 11
1.0 + BERT BERT LFx2
yes
59.7
+2.6
70.2
72.8
+1.7
73.6 11.9
73.8
+2,
yes
58.2
#1.1
70.7
#1.2
72.6
+2.5
73.9
#0.6
73.1
#2.5
Table 1: English Sentiment Analysis: Sentiment analysis results with 95% confidence interval measured for five
trials, evaluated on one-typo-per-wotd simulation. The highest score by absolute comparison in bold. The models following the BERTwich method, particularly LO+BERT, perform better than the baselines and perform best when more noise is used during fine-tuning.
Fine- Tuning Noise Level
Model
CPT
0%
10%
20%
30%
40%
BERT LO BERT
79.9 +3.0
90.1 =0.9
93.3 =0.7
93.5 =1.1
95.0 =0.7
79.9 +4.7
87.9 +1.8
91.1 +2.2
92.1 =2.7
91.9 #2.2
BERT LF
80.6 #5.5
+1.7
#2.3
=2.2
94.9 #1.2
Lo BERT LF
80.9 +3.4
85.1 #3.2
91.7 #3.2
93.1 =0.9
94.0 20.7
BERT LFx2
80.9 #3.7
88.8 #2.2
92.9 #3.1
92.0
94.7 =2.8
BERT L. BERT BERT LF L. BERT
yes
81.6 11.3
81.7
92.3 #1.4
92.9 #2.1
95.1 20.9
yes
80.7
87.4
95.2 +0.9
yes
79.3 40.6
42.1
93.7
96.0
96.7 #1.3
yes
83.3
+1.2
97.0
96.3 #0.4
97.9 #0.6
BERT LFx2
yes
82.7 +0.9
96.5 #1.7
97.7
=0.9
97.2 =0.3
Table South Tyrolean German Intent Classification: Results for German BERT with 95% confidence interval
measured for five trials, evaluated zero-shot on the South Tyrolean test data. The highest score absolute comparison is in bold. Though simply fine-tunie fine-tuning with noise greatly boosts performance, the BERTwich models, particularly LO BERT LF, perform better than the baselines.
Fine-Tuning Noise Level
Model
CPT
0%
10%
20% 30%
40%
BERT
no
59.7 +3399
81.7 +2.3
88.7 +0.77
90.1 +: 1.44
87.9 +2,8
LO + BERT
62.0 +3.1
80.3 +3.3
87.5
#1.1
88.2 =2.1
87.8 #2.1
BERT LF
60.2 =5.3
79.7 #3.9
88.7 #1.3
88.7
89,9 =2.1
LO
BERT LF
65.1 +6.1
78.4
#1.6
11.2
88.1 =2.5
BERT LFx2
60.3 +5.9
81.7 #3.4
89.1 +2.3
89.7 =2.5
88.8 =2.0
BERT LO BERT
72.4 42.9
43.2
83.4 #2.0
86.6 =2.1
85.0 #2.9
74.5
71.9
86.5 =2.0
85.7 =2.4
85.3 +1.1
BERT LF
yes
70.0 +5.00
77.2 a3.11
82.9 +1.4
83.9 +2.7
85.9 1.3
L. BERT +LF
yes
73.7 +3.4
82.1 +1.6
83.4 +1.
86.1 1..4
85.2 40.8
BERT L.F22
71.9
45.2
77.3
+1.7
83.0
+1.2
85.3
1.0
83.9 +3.3
Table Swiss German Intent Classification: Results for German BERT with 95% confidence interval measured for five trials, evaluated zero-shot on the Swiss German test data. The highest score by absolute comparison is in bold. Though fine-tuning with noise without the added layers or CPT yields the highest performance, note that the BERTwie Twich models perform substantially better than fine-tuning alone without noise.

---

of 10,000 sentences, each labeled with of 18 possible intent classes.
Against these baselines, we prepared four models for each language, as described in Section 3.1: LO BERT, BERT + LF, LO + BERT + LF, and BERT LFx2. We trained all of these models both with CPT (that is, the BERTwich method) and without CPT (that is, the parameters of the new layers remain randomly initialized at the beginning of fine-tuning). We also trained all of these models both with and without added noise in fine-tuning.
We evaluate our models not only on the standard German test set, but also test sets in two dialects of German:
South Tyrolcan, Bavarian dialect spoken in the northernmost province of Italy;
Swiss German, an Alemannic dialect spoken in Switzerland.
4.3 Training and testing details
We include two examples from xSID (van der Goot et al., 2021) translated in each language. Some in- stances are closer in appearance across the language varieties, while others vary more substantially.
In all of our training runs, we use the A optimizer and minibatch size of 8. When doing the continued pre-training of model, we set the learning rate le-4. Continued pre-training takes about 2.9 days on a single NVIDIA A10 Tensor Core GPU. We fine-tune each of the eight models on sequence classification task (English SST or German xSID) with 5 different noise levels (0, 10, 20, 30, or 40%). When fine-tuning model, set the learning rate to le-5, We fine-tune each model under each noise level setting five times with different random initialization each time, and report the average and 95% confidence interval across the five trials.
1. Closer in surface-Ievel appearance:
English: Is cloudy today?
German: Ist es heute bewölkt?
South Tyrolean: heint bewölkt?
Swiss German: Isch hüt bewöukt?
2. Farther in surface-level appearance:
English: Will it be sunny today?
German: Wird es heute sonnig sein?
South Tyrolean: Wearts heint sunnig?
Swiss German: Schynt hüt d' Sunne?
5 Results
We do use any data from the dialects until test time. The test data for each variety consists of 300 sentences.
The results of our experiments are shown in Tables to 3. Models following the BERTwich approach (added layers + CPT) fall in the final four rows of each table.
4.2 Baselines and model variations
We compare against several baselines which, unlike our approach, do not alter the BERT architecture.
5.1 English
We fine-tune our English models the SST sen- timent analysis task. Our new models maintain performance with the standard BERT fine-tuning approach on the original SST test data despite our modifications to the model (Appendix A). In Ta- ble 1, we show the performance of our models on synthetically-noise SST data (simulating one typo per word). Inclusion of noise in fine-tuning provides larger jump in performance, and as the fine-tuning noise level increases, does the performance. For each fine-tuning noise level, our BERTwich models perform the best, and there is compounding effect of adding the noise-primed layers and injecting higher levels of noise in fine-tuning.
BERT models without CPT or added noise in fine-tuning serve as naive baselines (the straightforward approach of fine-tuning BERT on task).
BERT models without CPT but with added noise in fine-tuning follow an approach used to improve performance on noisy and non- standard text in past work (Yin et al., 2020; Srivastava and Chiang, 2023).
BERT models with CPT and added noise in fine-tuning also follow an approach suggested in past work (Aepli and Sennrich, 2022).
5.2 German
For completeness, we also compare against BERT with CPT but without added noise in fine-tuning.
We tine-tune our German models on the German subset the xSID intent classification task. Our

---

models maintain performance on the test data of the standard variety (German), desired (Appendix A). We also evaluate them in zero-shot settings on the two dialects of German, which are unseen during any stage of training.
the experiments, and applying LoRA after CPT yields better performance than simply applying LoRA to BERT. At the same time, these results do not outperform our best BERTwich results (Tables to 3).
Each model fine-tuned with range of noise levels. As shown in Table 2, we find that addition of noise in fine-tuning alone provides boosts in performance on the South Tyrolean test data (a case of zero-shot transfer), and higher noise levels yield the best performance. We attain the highest performance on the South Tyrolean test data with our BERTwich method on the L0 + BERT + L.aar chitecture. This result indicates that our approach can be useful for improving downstream perfor- mance on unseen dialects. There is large jump in performance upon adding noise during fine-tuning, and a steady climb in performance with increasing fine-tuning noise level.
5.4 Embedding Space
To better understand how the embedding space and modeling capabilities change with our approach, we compare the models' representations of standard vs. nonstandard text. We do so in different ways for the English and German models (not fine-tuned), For English (Table 5), we aim to measure the distance in the embedding space between words and their noised counterparts. We do so by applying the teyy--per-worrt simulation to the SST test data and comparing the un-noised vs. noised encodings word by word (space-separated sequences). If a word spans more than one subword token, we take the encoding of the word to be the average of the encodings of the tokens that comprise it. We report the averages over all word pairs in the SST test data. For German (Table 6). we compare the sentence representations (i.e., the CLS encoding) for parallel text standard German vs. one of the two dialcets (South Tyrolean and Swiss German). We obtain the parallel text from the xSID intent classification dataset.
In Table 3, we show the performance of our mod- els on Swiss German test data, another instance of zero-shot cross-lingual transfer. We find that the method of fine-tuning with noise is the most effective in improving intent classification perfor- mance on unseen Swiss German data. At the same time, find that when noise is not present during fine-tuning, the models with CPT perform much better than those without CPT. In fine-tuning tasks where addition of noise may be impractical or un- suitable, continued pre-training with added encoder layers (i.e., the BERTwich method) is a promising approach to greatly boost performance.
We compare the average cosine similarity scores of BERT with those of BERT+CPT to measure the effect of CPT on the embedding space, as well as to the CPT models with added encoder layers to mea- sure the holistic effect of our method. We measure how well the bottom-most layer of each model does this implicit mapping between different versions of the same word, as well the similarity between the final word embeddings (for English typos) or sen- tence embeddings (for German dialects) resulting from the top-most layer of the model. For compar- ison, we also show the average cosine similarity scores for the first and last layer in BERT-base they are modified by CPT in the other models, even if they no longer remain the bottom- top-most layers in the new models due to the added layers.
5.3 Comparison to LoRA
addition to evaluating several variations of our approach, we also compare BERTwich to LoRA, an alternative to fine-tuning that allows for low- rank adaptation of a Transformer-based language model by adapting the attention weights of each encoder layer and freezing all other parameters when fine-tuning on the task (Hu et al., 2021). We use LoRA in place of vanilla fine-tuning our two tasks, English sentiment analysis and German intent classification, and apply the approach two models: BERT and BERT+CPT. We use a learning rate of 5e-4 and set the rank to for our LoRA experiments.
Without any intervention, there is an extremely high distance (i.c., low cosine similarity) in English BERT's embedding space, whether at the bottom the top layer, between the vector encoding of words and their noised counterparts (Table 5). Though continued pre-training reduces the gap, the addition of encoder layers evidently transforms the role that
We report the results, averaged over five tri- als, in Table 4. Just as in the main results, we find that applying more character-level noise to the adaptation/fine-tuni data is beneficial across all

---

Tine-Tuning Noise Level
Task
CPT
10%
20%
30%
40%
51.6 1..4
62.4 +1.1
64.2 11.0
65.0 +2.0
64.6 +0.7
English
yes
58.8 #1.6
67.0 #1.2
69.2 +0.6
69.8 +1.0
69.6 #2.1
75.2 =2.4
87.2 #2.7
90.4 +2.9
92.6 +2.9
94.0 #3.2
South Tyrolcan IC
yes
80.8 +2.0
84.0 #2.3
91.2 +3.7
93.8 +2.0
93.6 #1.4
Swiss German IC
no
=5.7
76.2
85.4 #3.7
86.2 #2.2
86.0
yes
73.2 #1.6
78.8 #3.1
83.6 +1.9
84.0 =2.5
84.6 =4.2
Table 4: LoRA Evaluation: Results for the English sentiment analysis (SA) and German intent classification (IC) tasks using LoRA rather than vamilla fine-tuning with 95% confidence interval measured for five trials. Our BERTwich approach outperforms L for English SA and South Tyrolean IC, and the two approaches perform the same for Swiss German IC.
the bottom layer plays in mapping words exhibiting spelling variation to the standard word, and greatly reduces the gap between representations of words and their noisy counterparts as they would come out from the top of the model. Not only do these results demonstrate the gradual progression mod- eling capability of each version of the model, but they also speak to the effectiveness of our contribu- tion of adding uninitialized encoder layers before conducting the continued pre-training with noise.
Layer
Model
CPT
LO
L1
L12
LF
BERT
0.13
0.24
BERT
yes
0.27
0.54
L0 BERT
yes
0.41
0.44
0.57
BERT LF
yes
0.55
0.48
0.52
I.0 BERT
yes
0.56
0.58
0.56
0.70
BERT LFx2
yes
0.50
0.65
0.77
The results of comparing the distance between the sentence representation (CLS encoding) of a standard German sentence and the parallel dialectal sentence are shown in Table 6, boasting the same finding as for English words that the BERTwich approach brings the internal representation of non- standard text much closer to its standard counterpart than can be found in an undisturbed BERT model. More importantly, this analysis demonstrates the importance of placement and the value of the sand- wich model in particular. While BERT LFx2 contains an equal number of parameters to BERT + LF, we see that it is no better than the undisturbed BERT at bringing standard and non- standard German text closer together in the internal representation. Rather. the combined efforts of LO and LF make L0 + BERT LF superior to the other BERTwich models in its ability model dialectal text the sentence level.
Table 5: Cosine similarity between words with and without typos: While the cosine similarities between noised and noised words in BERT's embedding space are extremely low, they are greatly increased in our BERTwich models, most notably in the lower layers of the model, indicating improved modeling capabilities.
Dialect
South Tyrolean
Swiss
Model
CPT
German
BERT
no
0.70
0.70
BERT
yes
0.78
0.70
L0 BERT
0.78
0.70
BERT + LF
yes
0.79
0.73
L0 BERT LF
yes
0.85
0.80
BERT 1.F22
yes
0.69
0.62
Table 6: Cosine similarity between standard and dialectal sentence embeddings (top layer): While the cosine similarities between top-layer CLS embeddings of standard vs. dialect parallel sentences in BERT"s embedding space are low, they are greatly increased in our BERTwich models (particularly L.0 + BERT + LF), indicating improved modeling capabilities.
Conclusion
In this paper, we explore the issues surrounding language modeling of nonstandard text and extend BERT's language modeling capabilities to com- mon real-world scenarios involving nonstandard text. Such issues could arise for variety of reasons, including inference on text exhibiting dialectal vari- ation, user errors such as misspellings or typos,

---

and propagated errors from optical character recog- nition and automatic speech recognition systems, We introduce the novel idea of adding randomly- initialized layers to the encoder stack of are- pre-trained BERT model before continuing the pre-training of the new model on text with synthetically-indus induuced noise. This way, the newly added layers improve BERT's ability to model nonstandard text. Our approach is called BERTwich.
to report statistical measures across the trials. In an effort to increase the diversity within our experiments, we included results for two languages, English and German, and evaluated our method in two scenarios involving dialectal text and text with typos. Our tasks are both sentence-level clas- sification tasks to allow for a clearer analysis of our methods; however, a more diverse array of lan- guages and tasks would be needed to assess the broader upplicability of our method, particularly as relates to different linguistic features (e.g., types of dialectal variation, morphological structure) and the demands the downstream task.
We find that our approach not only improves downstream task performance on nonstandard text, but also expands BERT's modeling capabilities. For instance, we find that the distance in BERT's embedding space between words and their noisy counterparts (e.g.. sludent vs. studebt) is extremely high, but dramatically reduced in the embedding space of our BERTwich models. Furthermore, we find that the sentential (CLS) representations within BERT's layers are not equipped to represent dialect text, while those extracted from the upper layers of the best BERTwich model are. Our findings indi- cate a transformation in the role the lower layer(s) can play in representing noisy text while BERT would need re-learn representations of the new token sequence for a noised word, our BERTwich method promotes knowledge transfer from standard to nonstandard text modeling, making it casier for the model to perform mappings to standard variants of words.
Ethics Statement
Our work involves the modification of existing mod- els using publicly available data, so our research methods themselves do not inherently evoke ethical concerns. The result of our work is improved ability to model text with features of dialectal vari- ation well as text with user- gerereteereeddddd noise (e.g., misspellings, typos), with the intention to improve the quality of NLP tools for wider range of users. While this can be a valuable change, one can imag- ine scenarios in which text that was once poorly handled by NLP systems can be easily compre- hended, which may be an undesirable change for some users (c.g., unwelcome monitoring of online communication).
Acknowledgements
Limitations
This material is based upon work supported by the US National Science Foundation under Grant No. IIS-2125948.
In this work, we provide a method of extending BERT's language modeling capabilities beyond what can be achieved by fine-tuning alone, without pre-training a new model from scratch. Our method involves adding layers to BERT's encoder stack and continuing the pre-training of the model for fraction of the epochs and data than would be needed when pre-training from scratch. Though continued pre-training is far less expensive, still requires computational power that is available at an institutional level but not always publicly available. As result, we are interested in identifying ways in which the computational load of our method could be further reduced, possibly using machine learning techniques like model distillation teacher learning so that the extra layers could be trained in isolation but still be compatible with the full model when added. Along similar lines, we would have liked to conduct multiple trials of the continued pre-training as we did for the fine-tuning to be able
References
Noëmi Aepli and Rico Sennrich. 2022. Improving zero- shot cross-lingual transfer between closely related languages by injecting character-level noise. In ACL Findings. pages 4074-4083.
Carlos Aspillaga, Andrés Carvallo, and Vladimir Araujo. 2020. Stress test evaluation of Transformer-basee models in natural language understanding tasks. Proc. LREC, pages 1882-1894,
Alexandra Benamar, Meryl Bothua, Cyril Grouin, and Anne Vilnat. 2021. Easy-to-use use combination POS and BERT model for domain-specific and misspelled terms. In NL4IA Workshop Proceedings.
Verena Blaschke, Hinrich Schütze, and Barbara Plank. 2023. Does manipulating tokenization aid cross- lingual transfer? study on POS tagging for non- standardized languages. In Tenth Workshop on NLP

---

for Similar Languages, Varieties and Dialects (Var- Dial 2023). pages 40-54.
Sangah Lee and Hyopil Shin. 2021. The Korean mor- phologically lightffittin tokenizer for noisy user- generated texts. In Proceedings of the Seventh Work- shop on Noisy User-generane Text (W-NUT 2021), pages 410-416.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt- agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private- by-design voice interfaces. In Proc. Workshop on Privacy in MI. and AL.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), Santa Fe, New Mexico, USA.
Seniz Demir and Berkay Topcu. 2022. Graph-based Turkish text normalization and its impact on noisy text processing. Engineering Science and Technology. an International Journal, 35:101192.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. Mad-x: An adapter- based framework for multi-task cross-lingual transfer. In Proc. EMNLP, pages 7654-7673.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing, In Proc. NAACL-HLT, pages 4181-4186.
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proc. NAACL HLT, pages 3795-3805.
Bo Han, Paul Cook, and Timothy Baldwin. 2013. Lexi- cal normalization for social media text. ACM Trans- actions on Intelligent Systems and Technology (TIST), 4(1):1-27.
Richard Socher, Alex Perolygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christo- pher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP, pages 1631-1642.
Will Held, Calcb Ziems, and Diyi Yang. 2023, Tada: Task-agnostic dialect adapters for english. arXiv preprint 2305
Elizabeth Soper, Stanley Fujimoto, and Yen-Yun Yu. 2021. BART for post-correction of OCR newspaper text. Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 284- 290.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Deriva- tional morphology improves BERT's interpretation of complex words. In Proc. ACL-IJCNLP (Volume I: Long Papers), pages 3694,6608, Online. Association for Computational Linguistics.
Aarohi Srivastava and David Chiang. 2023. Fine-tuning BERT with character-level noise for zero-shot transfer to dialects and closely-related languages. In Tenth Workshop on NLP for Similar Languages. Varieties and Dialects (VarDial 2023), pages 152-162.
J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representa- tions.
Dmitry Supranovich and Viachaslau Patsepnia. 2015. IHS RD: Lexical normalization for English tweets. In Proceedings of the Workshop on Noisy User-generated Text, pages 78-8 81.
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise machine translation. In Proceedings of the 5th Workshop on Noisy User-gemerated Text (W-NUT 2019), pages 42-47.
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung. Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient- based subword tokenization. Proc. ICLR.
Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving robustness of machine translation with synthetic noise. In Proc. NAACL HLT, pages 1916-18 19200.
Ankit Kumar, Piyush Makhija, and Anuj Gupta. 2020. Noisy text data: Achilles' heel of BERT. In Proceed- ings of the Sixth Workshop on Noisy Jiser-generated Text (W-NUT 2020), pages 16-21.
Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet Ustün, Marija Stepanovié. Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, and Bar- bara Plank. 2021. From masked language modeling to translation: Non-English auxiliary tasks improve zero-shot spoken language understanding. Proc. NAACL HLT, pages 2479-2497.
Thomas K. Landauer, Darrell Laham, Bob Rehder, and M. Schreiner. 1997. How well can passage meaning he derived without using word order? a comparison of Latent Semantic Analysis and humans. In Pro- ceedings of the 19th Annual Meeting of the Cognitive Science Society, pages 41 412-417.

---

Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUB: A multi-task benchmark and analysis platform for natural language understanding. the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353- 355.
Proceedings of
Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021. Multi-view subword regularization. In Proc. NAACL HLT, pages 473-482.
Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. Oolong: Investigating what makes crosslingual transfer hard with controlled studies. arXiv preprint arXiv:2202.12311 12312.
Fan Yin, Quanyu Long, Tao Meng, and Kai-Wei Chang. 2020. On the robustness of language encoders against grammatical errors. In Proc. ACL.
A Appendix

---

Fine-Tuiee Tine-Tuning Noise Level
Model
CPT
0%
10%
20%
30%
40%
BERT 1.0 BERT
91.0 +0.8
91.0
+1.0
91.3 +0.5
91.6
+0.8
91.2 #0.4
90.3 +0.5
90.7 +1.0
89.2 +3.9
90.8 +0.9
91.4 20.3
BERT LF
90.8 +0.3
90.8 +0.9
90.5 +1.4
91.4 +0.7
91.4 #0.7
LO BERT LF
90.5 #1.3
90.7 #0.5
91.0 =0.8
90.8
86.1
BERT LFx2
90.6 #0.6
90.5 #0.5
91.4 =0.7
90.8 =0.5
91.3 =0.8
BERT Lo BERT BERT Lo BERT LF
yes
88.8 #1.7
#0.4
88.9 #0.6
81.0
88.9 =0.7
yes
88.3
88.0
88.6 #0.9
89.1 #0.6
88.1 =0.8
yes
89.1 40.5
40.6
48.9
88.4
+1.0
yes
90.8 +0.3
90.8 +0.9
90.5
90.7
91.4 #0.7
BERT LFX2
yes
89.1 +0.9
88.9 +1.1
89.0
89.1 +111
89.0 +1.3
Table English Sentiment Analysis: Sentiment analysis results with 95% confidence interval measured for five trials, evaluated the original un-noised SST data. Performance is close to the baseline across all models, desired.
Fine-Tunnig Noise Level
Model
CPT
0%
10%
20%
30%
40%
BERT
no
97.5
+0.8
97.8
=0.8
98.3
98.1 =0.7
98.2 =0.5
L. + BERT BERT
no
97.9
40.4
97.4 +0.7
98.2
98.2 =0.7
98.7 40.6
no 97.8 +0.9
#1.2
98.1
98.8 =0.7
L. + BERT
no 97.8 40.7
97.5 40.7
98.7 40.5
98.4 40.5
98.4 40.6
BERT LFx2
no
98.0 +1.2
97.9 +0.9
99.0 #0.8
98.8 +1.0
98.3 +0.8
BERT
yes
97.5 40.6
98.3 +0.7
98.5 +0.6
98.4
98.7 40.6
Lo + BERT
yes
97.8 +0.4
98.1 +0.2
98.7 #0.5
98.7 =0,2
98.7 =0.5
BERT LF
yes
98.3 +0.5
98.6 +0.6
98.7 =0.7
98.6 #0.4
99.1 =0.4
Lo + BERT LF
yes
98.7
+0.5
98.1 +0.7
98.9 #0.8
98.4 #0.1
98.8 =0.7
BERT LFx2
yes
98.2
+0.6
97.5 +0.3
98.1 #0.5
97.9 =0.5
98.5 #0.4
Table 8: German Intent Classification: Results for German BERT with 95% confidence interval measured for five
trials, evaluated on the standard German test data, All models maintain performance on the standard text, as desired.


Reviews:
[{"review_id": "VwKvJ0SN2q", "reviewer": "Reviewer_x8AR", "summary": "", "questions": "a. Is there a reason why noise levels above 40% were not evaluated?\n\nb. Authors should consider including the performance of BERTwich on standard tasks along with the respective baselines. Additionally, quantifying the additional compute overhead of BERTwich will be helpful.  \n\nc. Results on other variants of BERT and more datasets will help in strengthening the paper further.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and for the opportunity to answer your questions.  \n\na) Is there a reason why noise levels above 40% were not evaluated?\n\nWe didn't try noise levels of 50% or more because it seemed too extreme to disrupt a majority of letters. We could certainly add these experiments in the final version, although even if they perform better, we would be reluctant to recommend such an extreme amount of noise for general use.\n\nb) Authors should consider including the performance of BERTwich on standard tasks along with the respective baselines. Additionally, quantifying the additional compute overhead of BERTwich will be helpful.\n\nAs part of our experimental runs, we also recorded the performance on the standard versions of the task; that is, performance of the English models on the original SST data, and performance of the German models on standard German itself.  However, the scores were all very close (no statistically significant differences) regardless of the model variant used, so we did not include these results in the paper so as to save space.  However, your point is certainly a good one that it would be important to explicitly highlight these results for completeness; we will include them in the appendix of the final version.  \n\nThe additional compute overhead of BERTwich (specifically, doing CPT for L0+BERT+LF) is about 3.4 days on a single NVIDIA Tesla P100 GPU.  We will include this information in the final version of the paper.\n\nc) Results on other variants of BERT and more datasets will help in strengthening the paper further.\n\nWe did preliminary experiments on BERT, RoBERTa, and CharBERT, fine-tuning on a sentiment analysis task and testing on synthetically noised test examples (a baseline experiment in our paper).  We found that all three models had similarly low performance across various synthetic noise settings.  We selected the most “vanilla” of the models when it comes to being specialized for the issue we are investigating - BERT.  This way, we could specifically see the effects of our approach to optimizing the model from scratch, so to speak.  We would like to add results for other variants of the model, such as DistilBERT, RoBERTa, and/or BART, for the final version of the paper for comparison.  \n\nUnfortunately, it is quite difficult to come by data that includes multiple dialects and is labeled for a specific NLP task like sequence classification, rather than masked language modeling or dialect identification.  We spent a lot of time looking for appropriate data and found the German intent classification dataset to be a good candidate.  We also have access to Italian intent classification (with one dialect of Italian, Neapolitan); we can certainly add the results for Neapolitan in the final version of our paper.  In addition, just as we applied synthetic noise to the English SST dataset for testing, we could do the same for a few more tasks (e.g., from the GLUE benchmark) for the final version of the paper.  \n\nTo respond to your comment, “Comparisons to past work are not clearly mentioned in the results”:\n\nWe directly compare to two pieces of past work in our results: 1) BERT with CPT and without fine-tuning noise comes from Aepli and Sennrich (2022), and 2) BERT without CPT and with increasing fine-tuning noise comes from Srivastava and Chiang (2023).  We attempt to explain this in Section 4.2 of our paper.  We can make it clearer in the results tables itself for the final version.\n\nFinally, we would like to mention details of our experimental setup for reproducibility (i.e., hyperparameters): \nIn all of our training runs, we use the AdamW optimizer and a minibatch size of 8.  When doing the continued pre-training of a model, we set the learning rate to 1e-4.  When fine-tuning a model, we set the learning rate to 1e-5.  As described in our Experiments section, we vary the random seed across trials and report the average across trials.  We will certainly include this information in our experiments section (Section 4) in the final version of our paper.  \n\nOnce again, many thanks for your review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and easy to follow\n- The authors perform many ablation experiments across different model variations, with/without continued pre-training and different noise levels to understand the impact of each component\n- BERTwich seems to out-perform the baseline BERT model in most cases when fine-tuned with noisy data. \n- The paper provides interesting insights to understand how the proposed modifications helps in increasing the cosine similarities between words and their noisy counterparts in the embedding space", "reasons_to_reject": "- While the authors claim that performance of BERTwich on standard tasks were not impacted with the proposed modifications, its important to highlight these results in the paper. Currently results on standard versions of different tasks along with the baseline are missing\n- Authors have not quantified the additional computational overhead with continued pre-training and architectural modifications\n- Evaluation on other variations of BERT (BERT-Large, DistilBERT etc)  and other datasets (Italian/Spanish intent classification) are missing\n- Comparisons to past work are not clearly mentioned in the results\n\n\n", "questions_to_authors": "a. Is there a reason why noise levels above 40% were not evaluated?\n\nb. Authors should consider including the performance of BERTwich on standard tasks along with the respective baselines. Additionally, quantifying the additional compute overhead of BERTwich will be helpful.  \n\nc. Results on other variants of BERT and more datasets will help in strengthening the paper further.", "paper_topic_and_main_contributions": "The paper mainly focuses on exploring performance issues of fine-tuned BERT on nonstandard text (eg: dialects, misspellings etc). The paper introduces the idea of adding randomly initialized layers to the encoder stack of pre-trained BERT (BERTwich) trained on noisy data to improve BERT’s performance on nonstandard text. BERTwich was evaluated mainly on 3 tasks across different variations of the proposed approach. The authors also found that BERTwich reduced the distance between the words and their noisy counterparts in the embedding space thereby expanding the modeling capabilities of BERT. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OOH2RQSaI1", "reviewer": "Reviewer_FFsj", "summary": "", "questions": "- A suggestion: I propose experimenting with PEFT methods like adapters, LoRA, etc. as well, which addresses the \"fairness\" in comparison, as the introduced parameters to the models are relatively small.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and thoughtful suggestions.  \n\nTo address your question, \"A suggestion: I propose experimenting with PEFT methods like adapters, LoRA, etc. as well, which addresses the \"fairness\" in comparison, as the introduced parameters to the models are relatively small.\":\n\nWe took your suggestion and ran an experiment in which we use the LoRA approach to adapt BERT to the SST task, and the fine-tuning data we use here is the noisy SST data, as was done in the paper with vanilla fine-tuning.  Our results demonstrate that using the LoRA approach with noisy data performs similarly to vanilla fine-tuning of BERT with noisy data across noise levels.  The highest performance attained using LoRA with BERT and our noisy FT approach was 68%, which is still lower than the highest performance attained by BERTwich (74.3%).  We will perform more thorough experiments with LoRA and include the results in the final version of the paper as another baseline.\n\nWe considered using adapters earlier in the project, but we thought the first step would be to see what types of improvements we can get by making simpler, more modular changes to the architecture in the hopes of having higher interpretability (for example, the type of analyses we do in Table 4 and 5).  \n\nTo respond to your comment, “All the experiments are done with BERT which haven't seen any \"informal\" data during pre-training. A model like RoBERTa, which is also pre-trained on e.g. OpenWebText (with possibly noisy data), would have been a natural candidate for all the experiments instead of adding extra layers to an under-trained model.”:\n\nWe did preliminary experiments on BERT, RoBERTa, and CharBERT, fine-tuning on a sentiment analysis task and testing on synthetically noised test examples (a baseline experiment in our paper).  We found that all three models had similarly low performance across various synthetic noise settings.  While CharBERT is a character-based model and may provide some advantage when dealing with traditionally over-segmented text, and as you rightly said, RoBERTa includes some informal text in its training data and uses BPE tokenization (BPE is often better than WordPiece when dealing with text that does not match the training data), we selected the most “vanilla” of the models when it comes to being specialized for the issue we are investigating - BERT.  This way, we could specifically see the effects of our approach to optimizing the model from scratch, and we are doing zero-shot adaptation.  We would like to add results for RoBERTa and/or BART for the final version of the paper for comparison.\n\nTo respond to your comment, “Although the improvements over baselines seems interesting, comparing BERT with a model that has two more layers, due to considerably increasing the number of parameters doesn't seem to be a fair comparison. After all, the improvement over BERT when CPT+40% is applied is not that significant (even in Table 3 the \"naive\" BERT seems to be the best performing model).”:\n\nThough we do not have enough time to perform the experiment before the end of the author response period, we could try applying the BERTwich method to the 10-layer BERT model released by Google; that way, the number of parameters in this BERTwich model would be equivalent to that of the original 12-layer BERT-base.  However, this would unfairly favor the baseline, because the BERT models from Google have undergone full pre-training, while our extra layers learn only from a fraction of the data during CPT. As such, they serve the purpose of extending the modeling capabilities (much like fine-tuning in this respect), but they are not powerful enough to take over the job of the original encoder stack – that is, without sufficient pre-training, the increased number of parameters alone is not enough to yield the large increases in performance demonstrated by our results.  The experiment with 10-layer BERT can be added to provide another way of interpreting the results.\n\nFinally, we would like to mention details of our experimental setup for reproducibility (i.e., hyperparameters): \nIn all of our training runs, we use the AdamW optimizer and a minibatch size of 8.  When doing the continued pre-training of a model, we set the learning rate to 1e-4.  When fine-tuning a model, we set the learning rate to 1e-5.  As described in our Experiments section, we vary the random seed across trials and report the average across trials.  We will certainly include this information in our experiments section (Section 4) in the final version of our paper.  \n\nOnce again, many thanks for your review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of improving generalisation to close dialects by introducing synthetic noise to pre-training/fine-tuning phase is an interesting approach for dealing with low-resource languages (even though it has been previously studied on e.g. PoS tagging and several sequence classification tasks.\n- The analysis on the similarity between words and their noisy variants is quite interesting, and gives an intuition why pre-training/fine-tuning considerably improves generalisation to noisy/non-standard data.\n", "reasons_to_reject": "- Although the improvements over baselines seems interesting, comparing BERT with a model that has two more layers, due to considerably increasing the number of parameters doesn't seem to be a fair comparison. After all, the improvement over BERT when CPT+40% is applied is not that significant (even in Table 3 the \"naive\" BERT seems to be the best performing model).\n- All the experiments are done with BERT which haven't seen any \"informal\" data during pre-training. A model like RoBERTa, which is also pre-trained on e.g. OpenWebText (with possibly noisy data), would have been a natural candidate for all the experiments instead of adding extra layers to an under-trained model.\n- The main novelty of the paper seems to be proposing adding two layers to the architecture, as the noisy pre-training/fine-tuning seems to be proposed by previous work. \n", "questions_to_authors": "- A suggestion: I propose experimenting with PEFT methods like adapters, LoRA, etc. as well, which addresses the \"fairness\" in comparison, as the introduced parameters to the models are relatively small.", "paper_topic_and_main_contributions": "This paper tries to improve BERT's language modelling when dealing with noisy data. The noisy input could be either due to misspelling/grammatical error, or when a dialect of language is parsed instead of the high-resource language that the model is trained on.\nThe authors propose adding two layers to the BERT's architecture and continue pre-training on character-level noisy data from Wikipedia to help the LM better handle noisy inputs. They show that the proposed method will enhance generalization (compared to baselines like BERT without adding extra layers) to noisy/non-standard data for two tasks: 1) noisy SST test set for English BERT 2) intent classification on Swiss-german and Tyrolean (German dialects) for German BERT.\nThe generalisation especially becomes better when the fine-tuning is performed with higher noise in fine-tuning data (for both the method and baselines). They also show that in the proposed model cosine similarity between a given word and its noisy variant becomes higher compared to baseline models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 226: repetitive text\n"}, "extra_text": {}}, {"review_id": "6nRorM0wJd", "reviewer": "Reviewer_FYgr", "summary": "", "questions": "A) Are there patterns to the cases in German dialects which switch from wrong to right with noisy pretraining? What types of linguistic patterns does this noisy training capture? Maybe this could help explore why the Swiss German performance degrades when noise is introduced.\n\nB) How do you explain the higher similarity at the start of BERT+LF than of L0+BERT?\n\nC) What were the hyperparameters used for experiments? Learning rate, Batch Size, epochs, etc", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review.  We appreciate your thoughtful suggestions and the opportunity to answer your questions.  We also appreciate the missing references you listed, and we will incorporate them into our related work section – and elsewhere when appropriate – in the final version of the paper.\n\nA) Are there patterns to the cases in German dialects which switch from wrong to right with noisy pretraining? What types of linguistic patterns does this noisy training capture? Maybe this could help explore why the Swiss German performance degrades when noise is introduced.\n\nUpon your suggestion, we looked at the examples the model got incorrect for the baseline (BERT without noise) and the best BERTwich model for both South Tyrolean and Swiss German.  The takeaway is one we would have suspected – many of the examples switch from wrong to right (as evidenced by the jump in accuracy); however, in the examples that do not switch, several of the lexical items are very different from those that would appear in the standard version of the sentence.  The mutual comprehensibility for these examples would likely be far lower and would require extra knowledge of new vocabulary items.  We would like to conduct this sort of error analysis in more detail and perhaps include some insights in the final version of the paper.\n\nWe would also note that the Swiss German performance improves when noise is introduced in fine-tuning.  Performance is also much better in the BERTwich models than simply fine-tuning German BERT on standard German.  But it is true that German BERT fine-tuned with 40% noise (with no architectural modifications) yields the highest performance over BERTwich models.   \n\nB) How do you explain the higher similarity at the start of BERT+LF than of L0+BERT?\n\nThe results you reference (Table 5) demonstrate that if it’s between L0 or LF, LF is more powerful; that is, L0 is not as helpful for adaptation to nonstandard text on its own.  Rather, the utility of L0 is unlocked when working in conjunction with LF, and this combination yields the best performance and much higher cosine similarity scores in the embedding space between un-noised and noised words. We suspect that this relates to past work like “BERT Rediscovers the Classical NLP Pipeline” (Tenney et al., 2019), in that L0 is useful for learning lower-level patterns like orthographic changes, but interpretable changes to the embedding space and other measures of language modeling capability do not make sense without including LF for CPT.  This way, higher-level patterns can be captured in the top layer with greater influence on the language modeling head in the architecture.\n\nC) What were the hyperparameters used for experiments? Learning rate, Batch Size, epochs, etc.\n\nWe will certainly include this information in our experiments section (Section 4) in the final version.  In all of our training runs, we use the AdamW optimizer and a minibatch size of 8.  When doing the continued pre-training of a model, we set the learning rate to 1e-4.  When fine-tuning a model, we set the learning rate to 1e-5.  As described in our Experiments section, we vary the random seed across trials and report the average across trials.\n\nTo respond to your comment, “they don't compare to other widely used architecture modifications for transformers such as Adapters, Prefix-Tuning, or LoRA”:\n\nWe took your suggestion and ran an experiment in which we use the LoRA approach to adapt BERT to the SST task, and the fine-tuning data we use here is the noisy SST data, as was done in the paper with vanilla fine-tuning.  Our results demonstrate that using the LoRA approach with noisy data performs similarly to vanilla fine-tuning of BERT with noisy data across noise levels.  The highest performance attained using LoRA with BERT and our noisy FT approach was 68%, which is still lower than the highest performance attained by BERTwich (74.3%).  We will perform more thorough experiments with LoRA and include the results in the final version of the paper as another baseline.\n\nWe considered using adapters earlier in the project, but we thought the first step would be to see what types of improvements we can get by making simpler, more modular changes to the architecture in the hopes of having higher interpretability (for example, the type of analyses we do in Table 4 and 5). \n\nTo respond to your comment, “This work would be much stronger if it incorporated and compared to synthetic noise grounded in actual typo patterns and language variation (e.g., full lexical shift and syntactic variation)”:\n\nWe would like to mention something found in our preliminary experiments that led us to make the decision to include random character-level noise rather than character-level noise based on frequent typos or instances of phonological variation.  We initially considered specific character swaps as our synthetic noise in training, where these swaps were based on the QWERTY keyboard (i.e., only swap X with a letter that is adjacent to X on the keyboard) or common phonological replacements (e.g., ‘r’ and ‘l’).  Anecdotally speaking, we did not find this to change the results, and this finding is echoed in previous work (Aepli & Sennrich, 2022).  We believe this is because our goal in applying character-level noise was to disrupt the subword tokenization, such that over-segmentation and subword replacement are artificially induced, exposing the model to varying tokenized versions of the same words or phrases.  In this respect, the introduction of character-level noise serves more as a means to expose the model to these phenomena during training rather than to mirror expected or potential test examples.  \n\nTo respond to your comment, “‘models maintain performance on Standard’ - Where do you show this? It doesn't appear to be in any of the tables in the paper?”:\n\nAs part of our experimental runs, we also recorded the performance on the standard versions of the task; that is, performance of the English models on the original SST data, and performance of the German models on standard German itself.  However, the scores were all very close (no statistically significant differences) regardless of the model variant used, so we did not include these results in the paper so as to save space.  However, your point is certainly a good one that it would be important to explicitly highlight these results for completeness; we will include them in the appendix of the final version.  \n\nTo respond to your comments, “it seems totally possible to me that the improved results of the sandwich approach are simply from the increased number of parameters” and “unclear that this architecture is a meaningful improvement without more diverse baselines, especially ones which control for parameter count”:\n\nThough we do not have enough time to perform the experiment before the end of the author response period, we could try applying the BERTwich method to the 10-layer BERT model released by Google; that way, the number of parameters in this BERTwich model would be equivalent to that of the original 12-layer BERT-base.  However, this would unfairly favor the baseline, because the BERT models from Google have undergone full pre-training, while our extra layers learn only from a fraction of the data during CPT. As such, they serve the purpose of extending the modeling capabilities (much like fine-tuning in this respect), but they are not powerful enough to take over the job of the original encoder stack – that is, without sufficient pre-training, the increased number of parameters alone is not enough to yield the large increases in performance demonstrated by our results.  The experiment with 10-layer BERT can be added to provide another way of interpreting the results.\n\nOnce again, many thanks for your review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Assessing the benefits of introducing synthetic noise during both pretraining and finetuning is an interesting question. Understanding how much general robustness can be derived from pretraining on synthetically noised data alone might yield good insights into the ways in which models maintain more broad linguistic knowledge.\n\n- Assessing the effects of the procedure on multiple dialects of German is valuable, since it allows experiments which tease apart the benefits depending on the types of variation which occurs in each dialect.", "reasons_to_reject": "- The primary novelty of the work appears to be the architectural change, but they don't compare to other widely used architecture modifications for transformers such as Adapters, Prefix-Tuning, or LoRA. The baselines (L0, LF, and regular BERT) all have strictly fewer tunable parameters than the proposed approach, so it seems totally possible to me that the improved results of the sandwich approach are simply from the increased number of parameters especially since the sandwich approach frequently does not outperform even these baseline architectures by a statistically significant margin. It seems unclear that this architecture is a meaningful improvement without more diverse baselines, especially ones which control for parameter count.\n\n- The approach for synthetic introduction of noise (insertion, deletion, replacement, and swapping) primarily address typos with some coverage of low-edit distance dialectal morphology changes. This is a quite small subset of dialectal shifts compared to full lexical shift and syntactic variation. Furthermore, even for typo noise this model is somewhat naive and assigns even probability to noise categories, despite real noise following patterns e.g. the example \"student\" -> \"studebt\" seems reasonable because B and N are neighbors on the keyboard, but \"student\" -> \"studeqt\" would almost never happen. This work would be much stronger if it incorporated and compared to synthetic noise grounded in actual typo patterns and language variation. \n\n- The model omits many standard details of training which would make it difficult to reproduce.", "questions_to_authors": "A) Are there patterns to the cases in German dialects which switch from wrong to right with noisy pretraining? What types of linguistic patterns does this noisy training capture? Maybe this could help explore why the Swiss German performance degrades when noise is introduced.\n\nB) How do you explain the higher similarity at the start of BERT+LF than of L0+BERT?\n\nC) What were the hyperparameters used for experiments? Learning rate, Batch Size, epochs, etc", "paper_topic_and_main_contributions": "This paper aims to make a Pretrained Language Model more robust to naturally occurring noise and linguistic variation. To do so, they propose adding additional layers to the start and end of a pretrained language model, using continued pretraining on data which has been synthetically noised using character level noise patterns, and finetuning on similarly noised data. They find that this full procedure helps with English typos and with a Bavarian dialect of German, while only finetuning on noisy data benefits performance in Swiss German. Finally, they perform an analysis of token-level alignment throughout the original and modified models and how this correlates with classifiers trained at different layers of the model as well.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Prior works for introducing noise during pretraining and finetuning:\n\nMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.\n\nZiang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 619–628, New Orleans, Louisiana. Association for Computational Linguistics.\n\nWu, Z., Papadimitriou, I., & Tamkin, A. (2022). Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies. ArXiv, abs/2202.12312.\n\nPaul Michel and Graham Neubig. 2018. MTNT: A Testbed for Machine Translation of Noisy Text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 543–553, Brussels, Belgium. Association for Computational Linguistics.\n\nVaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Graham Neubig. 2019. Improving Robustness of Machine Translation with Synthetic Noise. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1916–1920, Minneapolis, Minnesota. Association for Computational Linguistics.\n\nPrior works for modular LLM architectures to handle linguistic variation:\n\nJonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.\n\nWilliam Held, Caleb Ziems, and Diyi Yang. 2023. TADA : Task Agnostic Dialect Adapters for English. In Findings of the Association for Computational Linguistics: ACL 2023, pages 813–824, Toronto, Canada. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1: This figure is quite large, but with not a ton of information. This could be shrunk significantly (stack the 12 internal layers!) to make room for key information such as the settings for both CPT and finetuning training runs.\n\nLine 226: Repeated List of Operations twice\n\nLine 394-396: \"models maintain performance on Standard\" - Where do you show this? It doesn't appear to be in any of the tables in the paper? \n\n"}}]

Author rebuttal:

Meta-review: This paper explores using a sandwich architecture around BERT (a BERTwich) to improve performance for non-standard text. Experiments are performed in English and Swiss German to validate performance, and ablation studies are performed to determine the importance of the two halves of the sandwich.

The discussion period was very active, and while there are some comparison experiments reported in the discussion that perhaps should have been in the original paper, the reviewers generally found the work sound (scores of 3/3/4, note that reviewer FYgr said they would increase their score to 4 but could not successfully edit it, so I reflect that here). Excitement for the approach varied more (2/3/4), especially given the strong performance of prior work on this task when evaluated appropriately (see next paragraph).

The authors reported in the discussion period that there may not be an advantage to this approach over the better-known LoRA. However, I do believe there is still a benefit to publishing this work as it provides an alternative approach that in future experiments or different settings may provide an advantage.

When revising the paper, the authors must add the experiments described in the discussion, and it would be very beneficial to try to find any benefit to this approach over LoRA, either quantitatively or qualitatively. The authors may want to consider whether the focus of the paper should be on methods for success in the nonstandard text tasks they explore (exploring both LoRA and sandwich approaches with equal interest) or whether they still want to focus primarily on the sandwich approach.
Decision: Accept-Findings