## 4 Experiments
### Computation and Communication Analysis  
We accurately measure various computation and communication times needed to train a configured language model with specific batch sizes and proposed layer sizes. This measurement serves as a benchmark for calculating the total time required for training. Additionally, by aggregating the measured computation and communication times for each block size and the proposed batch size, we demonstrate that the proposed scheduling scheme has almost no GPU idle time due to waiting for context transfers. First of all, forward propagation, backward propagation, and optimization time, which are the main computational tasks in the training process, are measured for the batch size for the major layer size presented above and presented in Figure 7.  
Figure 6: Comparison of maximum configurable batch sizes for proposed layer sizes and layer-based Model Parallelism. In terms of computation, we observed that as the layer size increases, the time for forward computation, backward computation, and optimization all increase for the same input and output capacity. Specifically, the forward and backward computations substantially increase the computation time as the layer size grows. This is because the self-attention mechanism, a significant component of the transformer-based model, causes the computation volume to increase quadratically with the length of the input vector. In other words, if the length of the input vector doubles, the number of relationships that need to be processed increases fourfold. This is due to the necessity of computing the attention score for all pairs of input vectors. Therefore, we can observe that the computational complexity of the transformer models increases quadratically with the length of the input vector. In particular, the backward computation is more computationally intensive than the forward, thus increasing at a greater rate, while optimization is computed using multi-threading on the CPU side, it proceeds faster than computation on the multi-GPU. This is because its computational complexity is significantly lower compared to GPU computation, and there is no computational overhead due to communication. Additionally, we conducted computation experiments to predict the total computation time when composing various batch sizes, and we present the results. Figure 8 shows the calculation time measurement results for each batch size for the major layer sizes. Figure 8: Time measurement for each batch size of the proposed layer size (Base (**top left**), Large (**top right**), XLarge (**bottom left**), and XXLarge (**bottom right**)). Figure 7: Time measurement for each proposed layer size calculation type. The GPU computation task we propose can proceed once the context is prepared from the CPU to the GPU. The context preparation time mainly consists of three tasks: input transfer time, parameter transfer time, and buffer allocation time. We measured the total context preparation time and the times for the individual tasks that compose it. For the input transfer time, because the transfer time is very short in the forward operation, we measured it based on the hidden state transfer time required to construct the backward context. According to Figure 9, the most time-consuming task in preparing the context is Buffer Allocation Time. This is realized using memory allocation operations in CUDA, which take a significant amount of time. Our proposed Management System is designed to reuse the existing memory when conducting the same type of computation as the previous layer. Additionally, the time taken to transfer parameters increases only slightly, even as the model size grows. This is because the bandwidth performance of the DGX-A100 System is high compared to the capacity of the parameters that need to be transferred, and the parameters to be transferred are distributed to multiple GPUs. Consequently, as the model size increases, the time to prepare the context does not significantly increase compared to computation time.