# Empirical Analysis of Parallel Corpora and In-Depth Analysis Using LIWC
## 4 Experiments and Results
### Models Detail  
For verifying the quality of dataset provided by AI Hub, we constructed transformer based NMT model [4] that trained with each dataset. Transformer is an auto-regressive model structure which comprises encoder-decoder architecture, and is widely utilized in many NLP research fields, including NMT, in achieving SOTA performance. Corresponding model refrains recurrence and constructs its encoder-decoder model architecture by mainly applying attention structure. This enables considerable reduction of required training time by allowing significantly more parallelization in training process. Attention based model structure of transformer also can relieve long term dependency problem of RNN and LSTM [57]. Output results of attention structure can be described as Equation (1). \[\text{Attention }(q,k,v)=\text{Softmax}\Bigg{(}\frac{(W_{q}q)(W_{k}k)^{T}}{ \sqrt{d_{k}}}\Bigg{)}(W_{v}v) \tag{1}\]  
In Equation (1), \(W_{q}\), \(W_{k}\) and \(W_{v}\) refers to trainable parameters. Attention structure takes three input; query, key and value, which is denoted as \(q\), \(k\) and \(v\), respectively. Through this structure, transformer can obtain the relational information between input sentence and generating sentence. In such cases, the embedding obtained from input sentence is fed to the attention structure as \(q\) and \(k\), and the embedding from the generating sentence is regarded as \(v\). Attention structure is also leveraged to obtain the bidirectional contextual information of input sentence and generating sentence, through self-attention mechanism which takes identical embedding value as \(q\), \(k\), and \(v\) simultaneously. We construct transformer NMT model trained with each AI Hub dataset. We regard the performance of NMT model as the quality of parallel corpus, by controlling all the training conditions of our experiments to be identical, except the training dataset. Training objective of transformer based NMT model \(\theta\) that trained with parallel corpus \(P\) can be described as Equation (2). \[\max_{\theta}\ \frac{1}{\|D\|}\sum_{(X,Y)\in D}\log\left[\prod_{i=1}^{n}P(y_{i} \mid X,y_{t<i},\theta)\right] \tag{2}\]  
Overall process is similar to the training of sequence to sequence [30] based MT model. In Equation (2), \(X\) and \(Y\) indicate source and target sentence in \(P\), respectively. Target sentence \(Y\) comprises total \(m\) tokens, which are denoted as \(\{y_{i}\}_{1}^{n}\), and through this training process, corresponding model is trained to generate \(Y\) auto-regressively. In our training process, we used adam optimizer with noam decay, and all the batch size is set to be 4096. The transformer NMT model in our experiments consists of six encoder and decoder layers with six attention blocks and eight attention heads, which dimensionality and embedding size is 512. For the pre-processing of our training data, we utilized sentencepiece [58] subword tokenization method, with 32,000 vocab size. We extracted 5000 and 3000 samples randomly from training data for the validation and test set, respectively. The performance evaluation of all the translation results are proceeded with BLEU score by leveraging multi-bleu.perl script ([[https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl)]([https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl)), accessed on 25 May 2022) given by [PERSON].