# Automated Assessment of Inferences Using Pre-Trained Language Models
## 2 Materials and Methods
### Pre-Trained Language Models  
In this study, three widely used Transformer-based pre-trained language models [27] were selected to investigate their effectiveness for classifying inference. Due to the relatively small sample size (662) when compared to the huge parameter space of Transformer-based models, we decided to use BERT [24] and its variants in order to avoid overfitting. These models, namely the BERT-base [28], RoBERTa-base [29], and RoBERTa-large [29], have become fundamental in the field of natural language understanding due to their success in capturing complex patterns and representations, achieved through extensive pre-training on large corpora. The selection of these models allows for a comprehensive analysis of their performance and adaptability in fine-tuning scenarios for inference classification. Each model brings its own set of characteristics, such as model size, masking, and training objectives, which are summarized in Table 3. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Inference** & **Inference Type** & **Definition** \\ \hline \multirow{6}{*}{Inference} & elaboration & explanation about the contents of the current sentence using background knowledge \\ \cline{2-3}  & prediction & anticipation of what will occur next in the text \\ \cline{2-3}  & association & concept from background knowledge, brought to mind by the text \\ \cline{2-3}  & bridging & connecting contents of the current sentence with local/near or global/distant text information \\ \cline{2-3}  & metacognitive & reflection of understanding or agreement with the text \\ \cline{2-3}  & exchange & opinion about the text \\ \cline{2-3}  & affective response & emotion related to the contents of the text \\ \hline \multirow{3}{*}{Non-inference} & \multirow{3}{*}{ paraphrase} & putting the current sentence or part of the current sentence into their own words, or restating the text verbatim \\ \cline{2-3}  & meaningless & no response or meaningless response \\ \cline{2-3}  & response & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Inference types and definitions. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Sentence** & **Response** & **Inference** \\ \hline Plants prevent insects from eating them in several ways. & I think plants discourage insects from eating them in some ways, such as thorns or bad smell. & prediction \\ \hline Plants like roses produce thorns to keep insects away. & A rose cannot do anything because it’s a plant, but it has a lot of thorns. When insects come, they get stung and die. So, the insects cannot eat the plant. & elaboration \\ \hline Unusually, there are plants that eat insects. & I think it refers to some plants that eats insects, like the Venus flytrap! & association \\ \hline Plants prevent insects from eating them in several ways. & It means that plants have different ways of preventing insects from trying to eat them. & paraphrase \\ \hline Plants like roses produce thorns to keep insects away. & Hmm... I do not know. & meaningless response \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Examples of sentence–response pairs in which inferences were made. Korean versions of the three base models were pre-trained using the KLUE (Korean Language Understanding Evaluation) benchmark dataset [30]. The benchmark contains eight Korean natural language understanding tasks, including topic classification, semantic textual similarity, natural language inference, named entity recognition, relation extraction, dependency parsing, machine reading comprehension, and dialogue state tracking. The corpora used for this benchmark include news headlines, Wikipedia, Wikinews, policy news, The Korea Economics Daily News, and Acrofan News for formal texts, and ParaKQC, Airbnb reviews, and the NAVER Sentiment Movie Corpus for colloquial texts. The tokenizer for the dataset is a morpheme-based sub-word tokenizer [30], which first divides an input text into morphemes using a morphological analyzer, and then tokenizes them using the byte pair encoding (BPE) technique [31].