## 3 Methodology  
### Pipeline Parallelism  
When applied to the training of the LightChatGLM model, the DAPPLE algorithm optimizes the training efficiency of the Transformer architecture. The DAPPLE algorithm is a hybrid parallel training algorithm designed to address the challenges of distributed training for large-scale models on multi-GPU clusters. By combining the strategies of data parallelism, parameter parallelism, and pipeline parallelism, DAPPLE significantly reduces communication overhead and memory consumption during training, thereby accelerating the overall training process. Data parallelism in the DAPPLE algorithm is implemented by dividing the dataset into smaller mini-batches, which are then distributed across multiple GPUs for parallel processing. Simultaneously, DAPPLE incorporates parameter parallelism, which allocatesmodel parameters across different GPUs, allowing each GPU to store only a subset of the parameters, thus minimizing memory usage. In addition, pipeline parallelism is utilized to further enhance computational efficiency. By assigning different layers of the Transformer model to separate GPUs, DAPPLE enables cross-layer parallel computation, reducing the idle time of computational resources. In contrast to traditional data and parameter parallelism methods, where inter-GPU communication can result in significant overhead, pipeline parallelism overlaps these communications with computations, thereby decreasing the latency of the training process. This advantage is particularly pronounced in the training of large-scale models [59]. By employing a hybrid parallel strategy, DAPPLE efficiently allocates GPU resources, significantly shortening training times. Furthermore, DAPPLE's memory optimization capabilities enable LightChatGLM to achieve superior performance, even when operating with limited hardware resources. Suppose there is a training dataset consisting of \(N\) samples, each denoted as \((X_{i},Y_{i})\), where \(X_{i}\) is the input sequence and \(Y_{i}\) is the target sequence. The initial transformer-based model, denoted as \(M_{init}\), can be represented as follows:  
\[M_{init}=\{\theta_{init}\}, \tag{1}\]  
where \(\theta_{init}\) denotes the initial parameters of the model. In the pipeline construction phase, we construct a pipeline consisting of multiple parallel stages. Each stage executes sequentially in the pipeline and passes the data in a pipelined manner. Suppose there is a pipeline containing \(K\) parallel stages, denoted as  
\[S=\{S_{1},S_{2},\dots,S_{K}\}, \tag{2}\]  
where \(S\) represents the set of all stages in the pipeline, with each \(S_{i}(i=1,2,\dots,K)\) corresponding to the \(i\)-th parallel computation stage. Each stage performs a specific computational task and passes the intermediate results to the subsequent stage, thereby forming a sequential, pipelined computational flow. In the data transfer and update phase, we transfer the intermediate results computed in each phase to the next phase and leverage parallel computing resources to update the model parameters simultaneously in the parameter update phase. By denoting the communication time in the data transfer phase as \(T_{comm}\), the intermediate results computed in each phase \(S_{i}\) will be transferred to the next phase \(S_{i+1}\). \(T_{comm}\), which is represented by  
\[T_{comm}=\sum_{i=1}^{K-1}transfer(S_{i},S_{i+1}), \tag{3}\]  
where \(transfer(S_{i},S_{i+1})\) indicates the data transfer time from stage \(S_{i}\) to stage \(S_{i+1}\). \(T_{update}\) denotes the time of the parameter update phase, which uses parallel computing resources to update the parameters of the model simultaneously as follows:  
\[T_{update}=parallel(S_{i}). \tag{4}\]  
For each stage \(S_{i}\), if we assume that there are \(P_{max}\) parallel processing units that can perform computational tasks simultaneously, then it can be expressed as \(P_{max}\) processing units working simultaneously, i.e., \(parallel(S_{i})=P_{max}\).