## 2 Related Work  
### Pretrained Models  
Various PTMs models have been developed through further optimization: BERT refers to a model specialized for classification by stacking the multiple encoder blocks of the transformer. XLNet applies the factorization order technique, which enables the consideration of all sequences. RoBERTa is an optimized model of BERT, such as training time, batch size, etc. MASS introduces a masking technique that is opposite to encoder-decoder. BART adds various noising methods to the encoder. Finally, MT-DNN enables universal representation by applying to multitask learning to BERT [15; 29; 30; 31; 32; 33]. As a result, pretraining with a huge corpus and fine-tuning to downstream tasks have shown to significantly improve performance for a variety of activities. Existing PTMs, on the other hand, had limitations in that they were unable to complete all NLP tasks, including those involving question answering, summarizing, and machine translation. In addition, a T5 algorithm was developed, which defined all NLP tasks as \"Text-to-Text\" tasks. In various QA task performance evaluation experiments, such as GLUE, SuperGLUE, and SQuAD, T5 demonstrated high performance and accuracy [34; 35; 36]. The structure of the T5 model represents that of the typical transformer, however, performance is improved by suggesting several different learning methods. The first one is a word prediction method, which allows the prediction of words by denoising the noising input as a pretraining objective. Second, overfitting is prevented regardless of the size of the training data by using an unbalanced dataset for training. Third, the target data type is set to a text rather than a numerical value for universal applicable to all NLP tasks. Finally, state-of-the-art (SOTA) performance was demonstrated by training a model with 11 billion parameters on over 1 trillion tokens. Figure 1 depicts the encoder-decoder structure of the transformer, which translates the input sentence. The transformer consists of an encoder block that compresses information regarding the reference sentence structure and, subsequently, sends the information to the decoder block, as well as a decoder block that translates an input sentence utilizing information received from the encoder block as a reference. The transformer may solve the bottleneck of the existing Seq2 Seq model, and high performance has been demonstrated using contextual information as an attention mechanism [14]. Through an embedding process and positional encoding, the input sentence is converted into vectors containing positional information for each word token. Embedded vectors consumed by the encoder undergo multiple self-attention processes as follows: First, each token generates three types of vectors, which include the representation (Query) of each word in the input sentence, Keys, which serve as labels for identifying relevant words, and the Value, which refers to the actual word representation. Second, the query in one token is multiplied by all the keys to calculate a score indicating significance, which then goes through a softmax to output the attention weight. The attention weights are finally multiplied by the values to output the relevance among words within the input sentence. To promote model learning, the input and output vectors are added as a residual connection, and the scores are normalized by layer normalization. Unlike the RNN, the accuracy of the output sequence does not decrease due to the vanishing gradient, since the entire input sentence is repetitively referred to. The decoder block translates the input sentence or transforms the sentence structure by utilizing information from the encoder block regarding the relevance among words. The decoding block consists of two types of self-attention processes: the masked self-attention inside the decoding block and the attention between the output of the encoder and the decoder.