## 6 Methodology
### Dataset Used for Fine-Tuning  
Using both of the baseline models to translate technical patents from our database was not at a satisfactory level as many technical words and sometimes sentences would be incorrectly interpreted by the models. For example, the output would often contain incomplete sentences where words were cut off from the end of a sentence or incorrect grammar rules were applied. To improve the results outputted by the models, we fine-tuned the models by creating a dataset using approximately 80,000 patents that spanned various domains and used that dataset to train each model. The domains of the patents included human necessities; performing operations and transporting; chemistry and metallurgy; textiles and paper; fixed constructions; mechanical engineering, lighting, heating, weapons, and blasting; physics; and electricity. This was to ensure that each of the models can learn to accurately translate words according to their context. Additionally, from the Japanese Patent Office [55], we used data consisting of a Japanese-English machine translation dictionary to further enlarge our dataset. The dictionary was originally in unified power format and thus had to be extracted using the Accelerator library.  
Originally, the created dataset contained several spelling and grammatical errors and thus reduced the accuracy in the performance of the original model. To rectify this, an additional preprocessing step was performed. We used back translation, which is a technique that translates the target language to the source language and combines the original source sentences and back-translated sentences in an effort to train the model. The model was trained on the training dataset, which consisted of 124,828 rows of English and Japanese sentences, and training took approximately a total of three and a half hours. Then,  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Model** & **Parameters** & **Dataset** & **BLEU** \\ \hline \multirow{4}{*}{NLP-Model-I} & \(\bullet\) Optimizer: [PERSON] & \multirow{4}{*}{Japanese–English} & \multirow{4}{*}{\(<\)0.001} \\  & \(\bullet\) Warmup steps: 4000 & & \\  & \(\bullet\) Dropout: 0.1 & & \\  & \(\bullet\) Label smoothing: 0.1 & & \\ \hline \multirow{4}{*}{NLP-Model-II} & \(\bullet\) Weight decay: 0.01 & \multirow{4}{*}{\(<\)10} \\  & \(\bullet\) Batch size: 16 & & \\  & \(\bullet\) Metric: sacrebleu & & \\  & \(\bullet\) Vocabulary size: 32,001 & & \\  & \(\bullet\) Num of beams: 12 & & \\  & \(\bullet\) Gradient check-pointing: & & \\  & \(\bullet\) False & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Overview of Each Model.  
the model was tested using the validation dataset consisting of 31,207 rows of English and Japanese sentences. Moreover, the batch size chosen for fine-tuning the model was 16, and the metric used to measure accuracy was sacrebleu. The weight decay parameter was set to 0.01, and the number of train epochs was set to 3.  
After implementing various models and completing preprocessing steps, the following step included implementing multi-GPU processing to improve model runtime. Multiple GPU implementation options were considered including Genesis Cloud, DIGITS2, NVIDIA, and Asynchronous Functions. Evaluation of those options and the chosen design is reported in the Results section (Section 7).  
To fine-tune the model further, we have sent an application to ASPEC, a Japanese-based company that creates JP-EN data sets that include millions of parallel sentences extracted from scientific journals. This dataset is very promising as the main problem we faced was translating technical words that are hard for even a layman to grasp.