# Automatic Detection of Sensitive Data Using Transformer-Based Classifiers
## 3 Background
### Bert  
BERT [18] is an advanced word-embedding model that uses a bidirectional Transformer [19] to pre-train a language model on a large corpus and allow for fine-tuningthe pre-trained model on other tasks. As a sentence encoder, it can accurately produce the context representation of a sentence [20]. BERT is a deep bidirectional system that is capable of handling unlabeled text by jointly taking into consideration both the left and right context of a sequence in all layers. A deep bidirectional model is more powerful than a shallow left-to-right and right-to-left model. Given that BERT's goal is to generate a language model without generating a prediction, it uses only the encoder part of the Transformer architecture. The Transformer encoder can read the entire input sequence at once; thus it is considered bidirectional. This peculiarity enables the model to learn the context of the element of a sequence based on its full neighborhood. To overcome the problems affecting previous unidirectional models, such as RNN [21] or n-gram language models [22], BERT takes advantage of both these training strategies, as shown in Figure 2:  
* BERT removes the unidirectional constraint by performing a _Mask Language Model (MLM)_ task, which randomly masks some of the tokens from the input and tries to reconstruct (predict) the full original input. Unlike left-to-right language model pre-training, the objective of MLM enables the word representation to fuse the left and the right context, allowing one to pre-train a deep bidirectional Transformer. MLM is the key feature of BERT that has allowed it to outperform previous embedding methods. * Many important tasks, such as Question Answering (QA) and Natural Language Inference (NLI), involve understanding the relationship between two sentences, denoted as \(\langle\)Sentence \(A\), Sentence \(B\)\(\rangle\), which is not directly captured by language modeling. To train a model that understands sentence relationships, BERT is pre-trained on a binarized _Next Sentence Prediction (NSP)_ task: when sentence pairs \(\langle\)Sentence\(A\), Sentence\(B\rangle\) are selected as pre-training examples, half of the time, \(B\) is the actual sentence following \(A\), while, in the other cases, \(B\) is a random sentence taken from the sentence corpus. The model must then predict for each \(\langle\)Sentence \(A\), Sentence \(B\)\(\rangle\) pair whether Sentence \(B\) is the actual sentence following Sentence \(A\). The pre-training procedure has been based on the BookCorpus (800 M words) [23] and English Wikipedia. The model thus pre-trained can then be used in many NLP tasks, applying a relatively inexpensive (if compared to the pre-training phase) fine-tuning process. Fine-tuning is pretty straightforward: for each task, the task-specific inputs and outputs are plugged into BERT, and all the transformer parameters are then fine-tuned end-to-end. Two BERT models are available, differing in size:  
Figure 2: Overall pre-training and fine-tuning procedures for BERT. Except for the output layers, the same architecture is used in both pre-training and fine-tuning. The same pre-trained model parameters are used to initialize models for different downstream tasks. During fine-tuning, all parameters are tuned. [CLS] is a special symbol added in front of every input example, and [SEP] is a special separator token, e.g., separating questions and answers. (Adapted from [18]). * \(BERT_{base}\): it features 12 blocks of Transformers, a hidden layer with 768 neurons, 12 self-attention heads for a total size of the pre-trained model amounting to 110 million parameters. * \(BERT_{large}\): it features 24 blocks of Transformers, a hidden layer with 1024 neurons, 16 self-attention heads for a total size of the pre-trained model amounting to 340 million parameters. The \(BERT_{large}\) model requires much more memory than \(BERT_{base}\).