## 4 Evaluations
### Evaluation on Climate Technology Classification  
To investigate the performance of the PLMs in classifying climate technology in research proposals, we calculated the classification performance for each PLM fine-tuned using the preprocessed text (Table 4). RoBERTa is the only pretrained model in English text; therefore, it can be used as a baseline model to estimate the improvement in the classification performance of the other models. It achieved classification performances of 0.59 and 0.56 for the validation and test datasets, respectively. Meanwhile, XLM-RoBERTa, which is a cross-lingual-based pretrained model, achieved classification performances of 0.63 and 0.61 for the validation and test datasets, respectively. Compared with RoBERTa, which is a model pretrained using English text, XLM-RoBERTa demonstrated an improved performance by approximately 4%. This indicated that the XLM achieved a better classification performance than the model pretrained using English text. Meanwhile, BERT-M-cased pretrained with the multilingual method achieved classification performances of 0.65 and 0.64 for the validation and test datasets, respectively. This indicated that the multilingual model performed better than the XLM. To investigate the effect of pretraining using uncased text in the multilingual model, we calculated the classification performance of BERT-M-uncased, where 0.70 and 0.68 were obtained for the validation and test datasets, respectively. This indicated that the multilingual model pretrained using uncased characters performed better than the cased model in the classification task.  
Additionally, we calculated the classification performance of the models pretrained with Korean text to investigate language-specific effects on pretraining. The classification performance of KoBERT, which was pretrained using the Korean Wikipedia only, was 0.67 for both the validation and test datasets. Meanwhile, KLUE-RoBERTa, which was pretrained with RoBERTa using the KLUE dataset, demonstrated classification levels of 0.72 and 0.70 for the validation and test datasets, respectively. The differences in the classification performance were attributable to the size and detailed refinement of the dataset used for pretraining. Finally, KLUE-BERT achieved the highest classification performance (0.74 and 0.72 for the validation and test datasets, respectively). These findings imply that matching the language, quality, and size of the text for pretraining are key in multiclass classification. In addition, the error range was approximately 0.15 between the lowest and highest values.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Model**} & **Validation** & **Test** \\  & **(Macro-F1)** & **(Macro-F1)** \\ \hline \multicolumn{3}{c}{_English-pretrained baseline (baseline)_} \\ \hline RoBERTa & 0.59 & 0.57 \\ \hline \multicolumn{3}{c}{_Multilingual baseline_} \\ \hline XLM-RoBERTa & 0.63 & 0.61 \\
**Multilingual BERT (uncased)** & **0.71** & **0.68** \\ Multilingual BERT (cased) & 0.65 & 0.64 \\ \hline \multicolumn{3}{c}{_Korean-pretrained baseline_} \\ \hline KoBERT & 0.67 & 0.67 \\ KLUE-RoBERTa & 0.72 & 0.70 \\
**KLUE-BERT** & **0.74** & **0.72** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Classification performance of climate technology classification by the seven language models.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Loss Function**} & **Optimizer** & **Epochs** & **Validation Split Rate** & **Sequence Length** \\ \hline Categorical Cross Entropy & Adam & 10 & 20\% & 256 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Hyperparameters in fine-tuning.