# Automatic Detection of Sensitive Data Using Transformer-Based Classifiers
## 3 Background
### Transformers  
Transformers are a recent deep learning paradigm [15] that have become very popular in several research and application fields, such as Computer Vision (CV), natural language processing (NLP) and speech processing. Recent works show that Transformer-based pre-trained models (PTMs) [16] can achieve state-of-the-art performances on various tasks, especially in NLP. The basic Transformer architecture is a sequence-to-sequence mapping model that includes an encoder and a decoder, both consisting of a stack of \(L\) identical blocks. Each encoder block, in turn, consists of a multi-head self-attention module and a position-wisefeed-forward network (FFN). Decoder blocks additionally insert cross-attention modules between the multi-head self-attention modules and the position-wise FFNs. Furthermore, the self-attention modules in the decoder are adapted to prevent one position from attending to subsequent positions in a non-causal way. The overall vanilla Transformer architecture is shown in Figure 1.  
Transformers adopt an attention mechanism based on the Query-Key-Value (QKV) model. Given the packed matrix representations of:  
* The matrix of _queries_\(\mathbf{Q}\in\mathbb{R}^{NxD_{k}}\), which contains the vector representation of a word within a sentence;
* The matrix of _keys_\(\mathbf{K}\in\mathbb{R}^{MxD_{k}}\), which contains the vector representations of all words within a sentence;
* The matrix of _values_\(\mathbf{V}\in\mathbb{R}^{MxD_{x}}\) that are related to \(\mathbf{K}\) and, just like \(\mathbf{K}\), are vector representations of all words within a sentence. Figure 1: Vanilla Transformer architecture (adapted from [15]). The encoder, on the left, and the decoder, on the right, both consist of \(L\) identical blocks. The scaled-dot-product attention function used by the Transformer is given by  
\[Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax\Bigg{(}\frac{\mathbf{Q} \mathbf{K}^{T}}{\sqrt{D_{k}}}\Bigg{)}\mathbf{V}=\mathbf{A}\mathbf{V} \tag{1}\]  
where \(N\) and \(M\) denote the lengths of queries and keys (or values) and \(D_{k}\) and \(D_{v}\) denote the dimensions of keys (or queries) and values, respectively. \(\mathbf{A}=softmax\Bigg{(}\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{D_{k}}}\Bigg{)}\) is often called the _attention mask_; softmax is applied in a row-wise manner. The dot-products of queries and keys are divided by \(\sqrt{D_{k}}\) to alleviate the vanishing-gradient problem affecting the softmax function. Instead of simply applying a single attention function, Transformers use multi-head attention, where the \(D_{m}\)-dimensional original queries, keys and values are projected onto \(D_{k}\), \(D_{k}\) and \(D_{v}\) dimensions, respectively, learning \(H\) (preset number of heads) different sets of projections. For each of the projected queries, keys and values, an attention output is computed according to Equation (1). The model then concatenates all the outputs and projects them back to a \(D_{m}\)-dimensional representation:  
\[MultiHeadAttn(\mathbf{Q},\mathbf{K},\mathbf{V})=Concat(head_{1},\cdots,head_ {H})\mathbf{W}^{O} \tag{2}\]  
\[where\ head_{i}=Attention(\mathbf{Q}\mathbf{W}_{i}^{Q},\mathbf{K}\mathbf{W}_{i} ^{K},\mathbf{V}\mathbf{W}_{i}^{V})\]  
where \(W_{i}^{Q}\in\mathbb{R}^{D_{m}\times D_{k}}\), \(W_{i}^{K}\in\mathbb{R}^{D_{m}\times D_{k}}\), \(W_{i}^{V}\in\mathbb{R}^{D_{m}\times D_{v}}\) and \(W^{O}\in\mathbb{R}^{HD_{v}\times D_{m}}\). In a Transformer, there are three types of attention in terms of query sources and key-value pairs:  
* _Self-attention_. In the Transformer encoder, we set \(\mathbf{Q}=\mathbf{K}=\mathbf{V}=\mathbf{X}\) in Equation (2), where \(\mathbf{X}\) is the output of the previous layer. * _Masked Self-attention_. In the Transformer decoder, self-attention is restricted so that queries at each position can only attend to all key-value pairs up to and including that position to enforce causality.