## 3 Methodology
### Pipeline Parallelism
This means that the tasks in stage \(S_{i}\) are assigned to \(P_{max}\) processing units to maximize parallelism and processing efficiency. \(P_{opt}\) denotes the number of optimized parallel computing resources. By dynamically adjusting the degree of parallelism as follows, the algorithm can be made to achieve optimal performance in different hardware environments. \[P_{opt}=optimize(P_{max}), \tag{5}\]where \(optimize(P_{max})\) denotes a function that dynamically adjusts the degree of parallelism according to the hardware environment and task requirements. Based on the above description, a DAPPLE-based improved algorithm can be obtained for training the transformer-based model, as shown in Algorithm 1. ```
1: Initialize base model \(M\), training data \(T_{d}\), iterations \(num\_epochs\), sample batches iterate_batches
2: Initialize the number of pipeline stages \(num\_stages\)
3: Initialize the parallelism \(parallelism=determine\_parallelism()\)
4: Construct the pipeline \(pipeline=construct\_pipeline(num\_stages)\)
5:forepoch in range(\(num\_epochs\)) do
6:forbatch in range(\(iterate\_batches\)) do
7:forstage in range(\(num\_stages\)) do
8: forward_pass(pipeline[stage], batch)
9: synchronize()
10:endfor
11:forstage in reversed(range(\(num\_stages\)) do
12: backward_pass_and_update(pipeline[stage], batch)
13: synchronize()
14:endfor
15:endfor
16: Output algorithm performance evaluation results
17:endfor
18:return Trained model
```  
**Algorithm 1**Pipeline parallel transformer-based model training algorithm  
In Algorithm 1, the initialization phase includes setting up the model, the training dataset, the number of pipeline stages, and parallelism, as depicted in lines 1-3 of Algorithm 1. Then, during the model training phase, samples are initially divided based on sample blocks, followed by parameter calculation and updating via forward and backward propagation, as indicated in lines 4-14 of the algorithm. Eventually, the trained transformer model is obtained. The time complexity of the algorithm primarily revolves around three key factors: data transmission time \(T_{comm}\), task training time \(T_{c}\), and parameter updating time \(T_{update}\). In Algorithm 1, the function \(determine\_parallelism()\) is responsible for deciding the parallelization strategy for the model. It determines how to partition the model and allocate tasks to devices based on factors such as the number of model layers, available hardware resources (e.g., number of GPUs and memory), network topology, and task requirements. The function \(construct\_pipeline()\) constructs the entire pipeline, dividing the model into multiple stages and assigning each stage to a different device for parallel execution. \(forward\_pass()\) carries out forward propagation, starting with input data and processing through each layer of the model until the final output is generated. \(backward\_pass\_and\_update()\) handles the backward propagation and parameter updates. Backward propagation calculates error gradients and propagates them in reverse through the model. It then updates the parameters of each layer. In pipeline parallelism, backward propagation across different stages is performed in parallel, coordinating the exchange of gradients between devices. And the function \(synchronize()\) ensures synchronization between devices in pipeline parallelism, guaranteeing that each device is in a consistent state throughout the training process. Considering these factors, the overall time complexity of the algorithm can be expressed as \(max(K\cdot T_{comm},T_{c},T_{update})\).