## 3 Proposed Approach
Therefore, instead of using the maximum similarity of a token with the words in a lexicon, we introduce a parameter \(K\) that alleviates this issue by averaging the top-\(K\) similarities of a word with a lexicon. This technique follows the intuition that for a given lexicon, in the case of \"ass\" (clearly a word to remove), the top similarities are \"ass\": 1.0, \"dumbass\": 0.81, and \"asshole\": 0.80,...so the average top-\(K\) similarity is still high for \"ass\" and the model will correctly delete it from the source text. Still, in the case of \"_portrayed_\", the top similarities are \"_betrayed\": 0.75, \"stoned\": 0.42, and \"demented\": 0.41_, so the top-\(K\) average is reduced compared to the maximum and therefore \"_betrayed_\" is not removed from the source sentence as desired. Furthermore, we found that again due to the nature of word embeddings, very common words like \"_had_\" or \"_they_\" tend to have a high similarity with a lot of words and are, in turn, deleted as if they were toxic style attribute words. To alleviate this phenomenon, we have defined the MultiLexiconGST model. This model extends LexiconGST and utilizes an additional lexicon of common words and stop words to ensure that these types of words are not deleted from the source sentence. The next and last step is to generate target-style sentences, which we approach with a model that capitalizes on the benefits of transfer learning and the Transformer [20] by using an unsupervised large pre-trained language model. In detail, we use a \"decoder-only\" Transformer (where masked attention heads allow the model to look only at tokens to the left), namely Generative Style Transformer (GST) [19], which is based on the GPT [22]. The inspiration to use such a model comes from the recent state-of-the-art performance on different downstream tasks shown by these large pre-trained language models when fine-tuned. Specifically, GST models \(P(\mathbf{y}|\mathbf{c},\mathbf{s}^{\mathbf{tgt}})\) by receiving \(\mathbf{c}\) and \(\mathbf{s}^{\mathbf{tgt}}\) to generate \(\mathbf{y}\), the sentence in the target style. Additionally, we use special tokens in the input as shown in Figure 1 where GST is fed with \"<\(N\)e\(U\)> <\(CON\_START>\)_Stuff is around this <START>\"_. Concretely, we use <\(N\)e\(U\)> (Neutral) or <\(T\)OX> (Toxic) to indicate which style to generate, \(CON\_START\) points where the content starts, and <\(START>\) signals where content ends and begins to generate. Finally, GST uses a softmax layer on the last Transformer block to generate, at each time step \(\mathbf{t}\), a probability distribution over the vocabulary of the form \(P(\mathbf{y}_{\mathbf{t}}|\mathbf{c},\mathbf{s}^{\mathbf{tgt}},\mathbf{y}_{ \mathbf{1}},\mathbf{y}_{\mathbf{2}},\mathbf{y}_{\mathbf{t-1}})\). In the following, we present an example that gives further insight into the LexiconGST and MultiLexiconGST models illustrated in Figure 1 and Algorithm 1, respectively. As described, LexiconGST first strips toxic words, \(\mathbf{a}\), from the original sentence \"Stuff is nuts around this hoe\" with the use of the top-\(K\) similarity of tokens with a toxic lexicon. Please note that this similarity is calculated by averaging the top-\(K\) cosine similarities between each input sentence token and all tokens from the lexicon in use. Next, the associated word is removed from the sentence if this similarity exceeds a predefined th  
Figure 1: LexiconGST architecture. sentence is generated from the toxicity-stripped sentence _\"Stuff is around this\"_, **c**, by using a style-conditional transformer conditioned on neutral style, **s\({}^{\textbf{tgt}}\)**. As explained previously, MultiLexiconGST extends the LexiconGST's _style deletion component_ by using an additional lexicon to preserve common or stop words to alleviate the issues presented below.