Abstract: Pre-trained language models such as BERT are known to perform exceedingly well on various NLP tasks and have even established new State-Of-The-Art (SOTA) benchmarks for many of these tasks. Owing to its success on various tasks and benchmark datasets, industry practitioners have started to explore BERT to build applications solving industry use cases. These use cases are known to have much more noise in the data as compared to benchmark datasets. In this work we systematically show that when the data is noisy, there is a significant degradation in the performance of BERT. Specifically, we performed experiments using BERT on popular tasks such sentiment analysis and textual similarity. For this we work with three well known datasets - IMDB movie reviews, SST-2 and STS-B to measure the performance. Further, we examine the reason behind this performance drop and identify the shortcomings in the BERT pipeline.

Introduction: In recent times, pre-trained contextual language models have led to significant improvement in the performance for many NLP tasks. Among the family of these models, the most popular one is BERT BIBREF0, which is also the focus of this work. The strength of the BERT model FIGREF2 stems from its transformerBIBREF1 based encoder architectureFIGREF1. While it is still not very clear as to why BERT along with its embedding works so well for downstream tasks when it is fine tuned, there has been some work in this direction that that gives some important cluesBIBREF2, BIBREF3. At a high level, BERT’s pipelines looks as follows: given a input sentence, BERT tokenizes it using wordPiece tokenizerBIBREF4. The tokens are then fed as input to the BERT model and it learns contextualized embeddings for each of those tokens. It does so via pre-training on two tasks - Masked Language Model (MLM)BIBREF0 and Next Sentence Prediction (NSP)BIBREF0. The focus of this work is to understand the issues that a practitioner can run into while trying to use BERT for building NLP applications in industrial settings. It is a well known fact that NLP applications in industrial settings often have to deal with the noisy data. There are different kinds of possible noise namely non-canonical text such as spelling mistakes, typographic errors, colloquialisms, abbreviations, slang, internet jargon, emojis, embedded metadata (such as hashtags, URLs, mentions), non standard syntactic constructions and spelling variations, grammatically incorrect text, mixture of two or more languages to name a few. Such noisy data is a hallmark of user generated text content and commonly found on social media, chats, online reviews, web forums to name a few. Owing to this noise a common issue that NLP models have to deal with is Out Of Vocabulary (OOV) words. These are words that are found in test and production data but not part of training data. In this work we highlight how BERT fails to handle Out Of Vocabulary(OOV) words, given its limited vocabulary. We show that this negatively impacts the performance of BERT when working with user generated text data and evaluate the same. This evaluation is motivated from the business use case we are solving where we are building a dialogue system to screen candidates for blue collar jobs. Our candidate user base, coming from underprivileged backgrounds, are often high school graduates. This coupled with ‘fat finger’ problem over a mobile keypad leads to a lot of typos and spelling mistakes in the responses sent to the dialogue system. Hence, for this work we focus on spelling mistakes as the noise in the data. While this work is motivated from our business use case, our findings are applicable across various use cases in industry - be it be sentiment classification on twitter data or topic detection of a web forum. To simulate noise in the data, we begin with a clean dataset and introduce spelling errors in a fraction of words present in it. These words are chosen randomly. We will explain this process in detail later. Spelling mistakes introduced mimic the typographical errors in the text introduced by our users. We then use the BERT model for tasks using both clean and noisy datasets and compare the results. We show that the introduction of noise leads to a significant drop in performance of the BERT model for the task at hand as compared to clean dataset. We further show that as we increase the amount of noise in the data, the performance degrades sharply.

Question: How much is pre-training loss increased in Low/Medium/Hard level of pruning?
Answer: The increase is linearly from lowest on average 2.0 , medium around 3.5, and the largest is 6.0