## 3 Proposed Methodology
### Data Augmentation through Transfer Learning  
Data augmentation techniques have been used in image processing, visual identification, and text classification projects because they make it easy to collect and generate new data using simple and quick transformations. The goal of augmentation is to increase the quantity of training data samples to reduce the model's overfitting. The text augmentation NLP sector is challenging because of the immense complexity of language. Several techniques are available, including Easy Data Augmentation (EDA), replacement with a synonym, random swap, random insertion, and random deletion. Certain words, however, such as a, an, and the, cannot be filled by others. In addition, often a term does not have a synonym. Even if someone changes a single word, the context changes dramatically. SeqGAN [45], a sequence generation method, is being explored for synthetic text generation, although it is only suitable for generating short text. As we needed news content consisting of a large amount of text to create fake news, this model was unsuitable for our data augmentation technique.  
We had 1300 fake news data instances collected from the \"BanFakeNews\" dataset. To increase the amount of fake news data, we adopted a process that would consume less time and effort than manually collecting the fake news data. We employed transfer learning to facilitate data collection more efficiently. The idea behind transfer learning is that a method trained for one endeavor is re-purposed for a second related task. We used the BERT multilingual uncased model, which is self-supervised and trained on a massive corpus of multilingual data. This means it was trained solely on raw texts, with no human labeling (hence its ability to use a large amount of publicly available data), and then used an automatic strategy with these texts to produce input data and labels. It was explicitly configured with two goals described below in mind.  
Masked Language Modeling (MLM): given a sentence, the method masks 15% of the words in the input and then runs the whole masked text through the model, which must estimate the masked words. This is in contrast to traditional RNNs, which typically see words sequentially, or auto-regressive models such as Generative Pre-trained Transformer (GPT), that internally mask prospective tokens. It enables the model to learn a two-way representation of the sentence. Next Sentence Prediction (NSP): During pre-training, the model merges two masked texts as inputs. They sometimes, though not always, correspond to sentences which were next to one another in the original text. The model must then predict whether or not the two sentences followed each other.  
We performed next word prediction instead of next sentence prediction, which allowed us to perform insertion by prediction, whereas previous word embedding models pick one word randomly for insertion. The same word in sentences is represented by a static vector in traditional word embedding models, which might not work in certain circumstances. To address this issue, contextualized word embedding was introduced to take into account surrounding words when generating a vector in a different context. Thus, for substitution, we considered surrounding words in order to predict the target word. Figure 3 presents the data insertion and substitution performed on a training sample.  
Figure 2: The amount of fake and true news in both the balanced (augmented) and imbalanced (original) datasets. (**a**) Imbalanced dataset; (**b**) Balanced dataset.