## 3 Proposed Method
### Model  
In this study, the BERT [20], RoBERTa [21], and ELECTRA [22] were adopted as the baseline model structures, which were trained with augmented TOEIC data as proposed in Section 3.1. For implementing TOEIC task with transformer encoder-based model structures, conventional multiple-choice pipeline [13; 23; 24; 25; 26] was adopted. In addition, to determine the correct answer among the four answer candidates following that pipeline, four sequences corresponding to each option of the specific TOEIC QA were generated. The problem statement and the option words were concatenated with [SEP] token with a [CLS] token in the head. The [SEP] token was added to specify the problem statement and the option part. Specifically, for a particular problem statement _query_ and the corresponding candidates \(opt^{1},opt^{2},opt^{3},opt^{4}\), four input sequences \(seq^{1},seq^{2},seq^{3},seq^{4}\) were generated as in Equation (1).  
\[seq^{1} =[CLS]\text{ {query }}[SEP]\text{ {opt}}^{1}\ [SEP] \tag{1}\] \[seq^{2} =[CLS]\text{ {query }}[SEP]\text{ {opt}}^{2}\ [SEP]\] \[seq^{3} =[CLS]\text{ {query }}[SEP]\text{ {opt}}^{3}\ [SEP]\] \[seq^{4} =[CLS]\text{ {query }}[SEP]\text{ {opt}}^{4}\ [SEP]\]  
When adopting RoBERTa, where 
 and 
 tokens were utilized instead of [_CLS_] and [_SEP_], they were revised to 
 and 
 respectively, to follow the pre-trained structure.  
Subsequently, through PLM, each input sequence \(seq^{i}\) was encoded to the latent hidden representation. Specifically, the encoded hidden state for the first token of \(seq^{i}\), which is denoted as \(h^{i}_{0}\), was utilized as the encoded hidden representation of the whole sequence \(seq^{i}\). We adopt the same encoding process that was conventionally adopted to several other previous studies [20; 21; 22]. Thereafter, through the trainable pooling layer \(W\) and its bias term \(b\), \(h^{i}_{0}\) was encoded to predict the final pooled constant \(c^{i}\). Eventually, through the combination of \(c^{i}\) for each \(i\) and their subsequent concatenation into the single pooled vector, the pooled output vector was obtained. Consequently, by applying softmax activation to that vector, which was denoted as \(O\), the correct index for the answer can be predicted. These processes can be described as Equation (2).  
\[h^{i}=\text{PLM}(seq^{i})=[h^{i}_{0}\,h^{i}_{1}\,\cdots\,h^{i}_{n}] \tag{2}\]  
\[O=\begin{pmatrix}o^{1}\\ o^{2}\\ o^{3}\\ o^{4}\end{pmatrix}=\text{softmax}\begin{pmatrix}c^{1}\\ c^{2}\\ c^{3}\\ c^{4}\end{pmatrix}=\text{softmax}\begin{pmatrix}Wh^{1}_{0}+b\\ Wh^{2}_{0}+b\\ Wh^{3}_{0}+b\\ Wh^{4}_{0}+b\end{pmatrix} \tag{3}\]  
The predicted output vector \(O\) was utilized to gauge the probability of being selected as the answer for each option candidate. Further, the above process was optimized to minimize the cross-entropy of predicting the correct answer label. Specifically, for a given label \(y\) and model output \(O\) in a whole training dataset \(D\), whole model structures were trained to minimize the loss \(\mathbb{L}\) defined as Equation (4).  
\[\mathbb{L}=\sum D\Big{[}-\sum i=1^{4}t_{i}\log(o^{i})\Big{]}\quad\text{where} \quad t_{i}=\mathbf{1}(y_{i}=i) \tag{4}\]  
In this equation, \(t_{i}\) acts as an index indicator that returns 1 for the answer index and 0 otherwise. Thus, based on the above process, a multiple choice TOEIC classification model that can predict the correct answer, given the input problem statement and candidate options, can be constructed. The overall framework proposed in this study is shown in Figure 2.  
From the unlabeled corpus, the TOEIC data was generated using the proposed augmentation strategies. Thereafter, by following the multiple-choice QA pipeline, the TOEIC multiple choice task was implemented.