## 4 Experiments
### IP-Distillation Scheduling Experiments
#### 4.4.2 F-ALBERT Performance  
F-ALBERT is the result of distilling LR-ALBERT into IP-ALBERT. Based on LR-ALBERT, which reuses half of the layers as before, we proceeded with distillation on IP-ALBERT with the schedule presented above. The performance presented in Figure 8 is normalized as a percentage by dividing F-ALBERT's measurement result by that of the existing LR-ALBERT model. Our experiments confirmed that when the connection length was additionally reduced in reusing the second step, a decrease in the Glue benchmark performance of 1 to 2% occurred. Through this, it was confirmed that the scheduling technique based on the  
Figure 8: Measurement of F-ALBERT’s computing and GLUE average score performance based on different schedule configurations (**left**: Base-size model, **right**: Large-size model). Figure 7: Trend of the change in Euclidean Distance between the final output Relational Knowledge of LR-ALBERT and the Relational Knowledge of each layer. amount of change was valid. We present the final F-ALBERT standard schedule as Y for the base model and N for the large model. Compared to LR-ALBERT, the base model increased in speed by 20% and decreased in memory usage by 15%, and the large model increased in speed by about 40% and decreased in memory usage by 15%. Table 2 is the final GLUE-benchmark result of F-ALBERT. F-ALBERT comprises various hyperparameters that can directly affect the model's generalization performance. We identify crucial parameters that exert a direct influence on F-ALBERT's functionality, proposing hyperparameter settings rooted in previously conducted experiments. Hyperparameters can be divided into two facets: learning progression and model composition. In the learning dimension, the epoch count is vital. We employed the C4 dataset for distillation, observing that when the number of epochs exceeded two, there was a marginal reduction in benchmark performance. Therefore, we suggest training on a range of non-overlapping datasets or configuring training with a lower number of epochs for distillation tasks. Regarding model composition, the careful configuration of LR-ALBERT's constituent layers is essential. As per Tables 3 and 4, if LR-ALBERT is configured with fewer than half of the original layers, performance markedly declines. LR-ALBERT exhibited optimal performance when composed of half the number of the original teacher layers. Moreover, IP-ALBERT's schedule is a key model hyperparameter. In Tables 5 and 6, we have presented schedules for LR-ALBERT according to its sizes. Specifically, Schedules Y and N are those with minimized computational requirements that can ensure generalization performance. Based on empirical findings, establishing a schedule with even less computation led to a generalization performance decrease of more than 15%. Therefore, we recommend employing the proposed Y and N schedules. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **MNLI** & **QNLI** & **QQP** & **RTE** & **SST** & **MRPC** & **CoLA** & **AVG** \\ \hline
**ALBERT-Large +24 Layer** & 87.1 & 91.8 & 90.1 & 80.3 & 92 & 91 & 60.9 & 84.74 \\ \hline
**LR-ALBERT +18 Layer** & 85.3 & 91.76 & 90.3 & 80.5 & 93.6 & 90.2 & 58.3 & 84.28 \\ \hline
**LR-ALBERT +12 Layer** & 85.3 & 91.1 & 89.5 & 78.5 & 92.7 & 90 & 59.3 & 83.77 \\ \hline
**LR-ALBERT +6 Layer** & 81.8 & 88.5 & 89 & 68.5 & 92.3 & 86.9 & 47.6 & 79.23 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: GLUE benchmark for various layers of large LR-ALBERT. The Table 7 compares distilled models possessing the same structure of the Transformer layer as the proposed F-ALBERT, including TinyBERT [33], DistillBERT [34], ALP-KD [19], and LAD [20], all of which are composed of six Transformer layers of similar size to F-ALBERT. Since F-ALBERT reuses parameters, it exhibits approximately six times the parameter efficiency of existing methods. This allows it to operate efficiently in embedded environments where GPU memory resources are limited.