SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(f) Time-wise, Bert-Base, batch size  $= 3 \times 64$
Figure 2: Epoch-wise and time-wise convergence speed for training ResNet-50 with 32 samples per GPU, ResNet-50 with 128 samples per GPU, and fine tuning BERT-Base with 3 samples per GPU with 64 GPUs.
Table 1: System throughput and Test Accuracy of SGDM, 1-bit Adam and Birder for training ResNet-50 on ILSVRC2012 from scratch with 8, 16, 32, 64 GPUs.
Optimizer	# GPUs	32 samples per GPU	128 samples per GPU
Throughput (samples / s)	Top-1 Acc. (%)	Throughput (samples / s)	Top-1 Acc.(%)
SGDM	3693 (1.00×)	76.19	5272 (1.00×)	75.05
1-bit Adam	8	3243 (0.83×)	75.55	5229 (0.99×)	75.42
Birder	3462 (0.94×)	75.98	5251 (0.99×)	75.45
SGDM	2959 (1.00×)	75.96	6189 (1.00×)	74.61
1-bit Adam	16	4745 (1.60×)	75.33	8836 (1.42×)	75.05
Birder	6015 (2.03×)	75.53	9633 (1.56×)	75.09
SGDM	4270 (1.00×)	75.47	9909 (1.00×)	74.54
1-bit Adam	32	7268 (1.70×)	75.18	13827 (1.40×)	74.62
Birder	9416 (2.21×)	75.27	15950 (1.61×)	74.82
SGDM	6189 (1.00×)	75.37	16640 (1.00×)	74.22
1-bit Adam	64	5546 (0.89×)	75.54	16426 (0.99×)	74.14
Birder	15253 (2.47×)	75.30	23727 (1.43×)	74.24
Recently, several works [33],[1] have shown that when utilizing the system-level engineered distributed data-parallel framework  $DDP$ , the existing communication-compression optimizers (excluding 1-bit Adam) still perform slower than the uncompressed SGDM/Adam. Therefore, in our evaluation, we focus on assessing the performance of Birder, the uncompressed SGDM/Adam, and the closely related algorithm 1-bit Adam through distributed training experiments using the benchmark models ResNet-50 (CNN) and BERT-Base (Transformer). More extensive experiments can be founded in Section B of the appendix.
Table 2: System throughput and F1-Score / Excat-Match of BertAdam, 1-bit Adam and Birder for fine tuning BERT-base on SQuAD 1.1 with 8, 16, 32, 64 GPUs.
Optimizer	# GPUs	Throughput (samples/s)	F1-Score (%)	Exact Match (%)
BertAdam	8	413 (1.00×)	88.13	80.59
1-bit Adam	358 (0.87×)	88.05	80.06
Birder	412 (1.00×)	88.71	81.18
BertAdam	16	84 (1.00×)	88.47	81.07
1-bit Adam	213 (2.54×)	87.87	80.31
Birder	431 (5.13×)	88.31	80.80
BertAdam	32	119 (1.00×)	88.38	80.94
1-bit Adam	274 (2.30×)	87.78	80.08
Birder	730 (6.13×)	88.08	80.50
BertAdam	64	158 (1.00×)	88.13	80.94
1-bit Adam	252 (1.59×)	87.33	79.67
Birder	990 (6.26×)	88.28	80.75
Our experiments were conducted on a testbed consisting of 1, 2, 4, 8 nodes interconnected via 10Gbps Ethernet. Each node was equipped with 8 Nvidia Tesla A100-80GB GPUs. The hardware and software configurations were identical across all instances, with Ubuntu 20.04.4 LTS serving as the operating system. PyTorch 1.11.0 was used as the primary framework, accompanied by CUDA-11.6, cuDNN-8.2, NCCL-2.10.3, and PyTorch 1.11.0 for other relevant libraries. Notably, to ensure compatibility with PyTorch's DDP, certain components of Birder and our hierarchical communication scheme were implemented within the customized communication hook of DDP.
Training details. For the experiments over ResNet-50, we evaluate the convergence and performance of SGDM, 1-bit Adam and Birder on ILSVRC2012. The batch size per GPU is set to 32 or 128 with the standard input resolution  $224 \times 224$ . When employing SGDM (baseline), the learning rate starts at  $0.1 \times \frac{batch\ size}{256}$  with momentum of 0.9 and weight decay of 0.0001. When employing 1-bit Adam and Birder, the learning rate starts at  $0.001 \times \frac{batch\ size}{256}$  with weight decay of 0.0001, and  $[\beta_{1}, \beta_{2}]$  for 1-bit Adam is set to [0.9, 0.999] and  $\beta$  for Birder is set to 0.95. Then, the learning rate is divided by 10 after 30, 60 and 90 epochs, and training is finally terminated after 100 epochs. Specifically, the first 15 epochs are used as the warmup stage for 1-bit Adam. For the experiments over BERT-Base, we access the convergence and performance of BertAdam (baseline), 1-bit Adam and Birder for SQuAD 1.1 fine-tuning task using a pre-trained BERT-Base model checkpoint from HuggingFace<sup>6</sup>. The batch size per GPU is set to 3. We perform fine-tuning for 2 epochs. The learning rate linearly increases to  $1 \times 10^{-4}$  steps in the early 500 steps and then linearly decreases to 0 in the rest iteration. Specifically, the first  $0.2 \times$  steps are used as the warmup stage for 1-bit Adam.  $[\beta_{1}, \beta_{2}]$  for BertAdam, and 1-bit Adam is set to [0.9, 0.999] and  $\beta$  for Birder is set to 0.9.
5.2 Experimental Results
Figure 2 shows the convergence behaviors of epoch-wise and time-wise training for SGDM / BertAdam (baseline), 1-bit Adam, and Birder using ResNet-50 and BERT-Base models running on 64 GPUs. The experimental results clearly demonstrate that Birder achieves a similar epoch-wise convergence rate compared to the baseline. However, the actual training speed of Birder surpasses both the baseline and 1-bit Adam by a significant margin.
Figure 3 illustrates the system throughput of different optimizers when running ResNet-50 and BERT-Base on 8 GPUs to 64 GPUs (1 node to 8 nodes). When training on 8 GPUs (1 node), where computation takes precedence over communication, the throughput of Birder is slightly lower than that of SGDM and BertAdam. However, as the number of GPUs increases, Birder consistently outperforms its counterparts, and this superiority becomes increasingly evident with more GPUs. Additionally, the system throughput for SGDM, BertAdam, and 1-bit Adam occasionally decreases as the number of GPUs increases, whereas the throughput of Birder steadily grows. This observation indicates that Birder offers better scalability efficiency.
$^{6}$ https://github.com/huggingface/transformers
(a) ResNet-50, 32 samples / GPU
Figure 3: System throughput of optimizers for training (a) ResNet-50 with 32 samples per GPU, (b)ResNet-50 with 128 samples per GPU, and (c) fine tuning BERT-Base with 3 samples per GPU with 8, 16, 32, 64 GPUs.
(b) ResNet-50, 128 samples / GPU
(c) BERT-Base, 3 samples / GPU
(b) ResNet-50, 128 samples / GPU
(c) BERT-Base, 3 samples / GPU
Figure 4: Computation time, communication time and compression/decompression time per iteration of optimizers for training (a) ResNet-50 with 32 samples per GPU, (b)ResNet-50 with 128 samples per GPU, and (c) fine tuning BERT-Base with 3 samples per GPU with 8, 16, 32, 64 GPUs.
In terms of inference performance for ResNet-50, we evaluate the Top-1 accuracy after training on ILSVRC2012 from scratch. For BERT-Base, we measure the F1-score and exact-match score after fine-tuning on SQuAD 1.1. Table 1 shows that when the batch size is set to 32 samples per GPU, the accuracy of Birder is slightly lower than that of SGDM. It has been suggested in some works ([13], [35]) that adaptive optimizers generally yield worse generalization compared to SGDM for CNN architectures. However, as the batch size increases (Table 2), both 1-bit Adam and Birder achieve better accuracy. This can be attributed to the beneficial effect of introducing a certain level of noise for generalization ([25]), which biases the optimizer towards wider valleys. Table 2 demonstrates that Birder achieves similar or higher F1-score and exact-match score compared to BertAdam and 1-bit Adam, validating the effectiveness of Birder for inference tasks.
5.3 Communication Efficiency Analysis
As shown in Figure 4, training on a single node demonstrates that the baseline SGDM and BertAdam algorithms are slightly faster compared to Birder and 1-bit Adam. In this scenario, the inter-GPU bandwidth within a node is extremely high, rendering communication time negligible. However, the newly introduced compression/decompression process by Birder and 1-bit Adam adds extra time due to its implementation. Thanks to light-computation quantization, the compression/decompression time for Birder with ResNet-50 and BERT-Base is significantly reduced to approximately  $15ms$  and  $8ms$  respectively. When conducting distributed training across two nodes, the bandwidth between them is relatively limited (10Gbps in our experiment), making communication time a critical factor. In the case of uncompressed SGDM and BertAdam, the communication time substantially exceeds the computation time for ResNet with 32 samples per GPU and BERT-Base. Consequently, the system throughput is lower compared to a single node (as depicted in Figure 2). However, the extreme 1-bit quantization implemented in Birder effectively reduces communication overhead, resulting in only a marginal increase in the total time required for Birder. As the number of nodes continues to increase, the importance of an efficient communication scheme becomes paramount. By leveraging
our proposed Hierarchical-1-bit-All-Reduce, the overall inter-node communication volume exchanged scales proportionally with the number of nodes. In contrast, the Compressed-All-Reduce method employed by 1-bit Adam [28] results in the overall communication volume exchanged among nodes being proportional to the number of GPUs (eight times larger than the number of nodes in our experiments). Consequently, as the number of nodes increases, the communication time for Birder exhibits a gradual rise, while the communication time for 1-bit Adam experiences a sudden surge.
In this study, we introduce a novel 1-bit adaptive optimizer for distributed training. Our optimizer offers the advantages of being lightweight in terms of computation while employing extreme 1-bit quantization for the communication data. Furthermore, we provide theoretical evidence demonstrating that Birder can achieve convergence rates comparable to the uncompressed Adam. To enhance communication speed, we propose a novel communication scheme tailored specifically for Birder, replacing the inefficient naive All-Gather approach. Through extensive experiments on benchmark models such as ResNet-50 and BERT-Base, we validate the effectiveness and efficiency of Birder in comparison to uncompressed methods like SGDM/Adam as well as the relevant 1-bit Adam.
This work is supported by the National R&D Program of China (Grant No. 2022ZD0115301), the Major Key Project of PCL (Grant No. PCL2023AS7-1), and the National Natural Science Foundation of China (Grant No. 61806128).
[1] Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman, and Dimitris Papailipopoulos. On the utility of gradient compression in distributed training systems. In Proceedings of Machine Learning and Systems, volume 4, pages 652-672, 2022.
[2] Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint arXiv:1704.05021, 2017.
[3] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. In Advances in neural information processing systems, volume 30, 2017.
[4] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. Signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560–569, 2018.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[6] Congliang Chen, Li Shen, Haozhi Huang, and Wei Liu. Quantized adam with error feedback. ACM Transactions on Intelligent Systems and Technology (TIST), 2021.
[7] Jacques Chen, Frederik Kunstner, and Mark Schmidt. Heavy-tailed noise does not explain the gap between sgd and adam on transformers. In 13th Annual Workshop on Optimization for Machine Learning, 2021.
[8] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. arXiv preprint arXiv:1808.02941, 2018.
[9] Minsik Cho, Vinod Muthusamy, Brad Nemanich, and Ruchir Puri. Gradzip: Gradient compression using alternating matrix factorization for large-scale deep learning. In Advances in Neural Information Processing Systems, 2019.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[11] Eduard Gorbunov, Konstantin P Burlachenko, Zhize Li, and Peter Richtárik. Marina: Faster non-convex distributed learning with compression. In International Conference on Machine Learning, 2021.
[33] Hang Xu, Chen-Yu Ho, Ahmed M Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatasenidis, Marco Canini, and Panos Kalnis. Compressed communication for distributed deep learning: Survey and quantitative evaluation. Technical report, 2020.
[34] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems, 2020.
[35] Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285-21296, 2020.
A Theoretical Analysis for Algorithm 1
In practice, we implement Birder in a non-parameter-server model to further reduce the communication overhead, but the data exchange is essentially equivalent to that in a parameter-server prototype. Hence, we provide the theoretical analysis for Birder in a parameter-server model as shown in Algorithm 1.
According to Algorithm 1, the update  $\bar{u}_t$  can be recursively formulated as
$$
\begin{array}{l} \bar {u} _ {t} = \mathcal {Q} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} u _ {t} ^ {(i)} + \bar {e} _ {t}\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} u _ {t} ^ {(i)} + \bar {e} _ {t} - \bar {e} _ {t + 1} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {Q} \left(\frac {m _ {t} ^ {(i)}}{b _ {t} ^ {(i)}} + e _ {t} ^ {(i)}\right) + \bar {e} _ {t} - \bar {e} _ {t + 1} \tag {7} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\frac {m _ {t} ^ {(i)}}{b _ {t} ^ {(i)}} + e _ {t} ^ {(i)} - e _ {t + 1} ^ {(i)}\right) + \bar {e} _ {t} - \bar {e} _ {t + 1} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {m _ {t} ^ {(i)}}{b _ {t} ^ {(i)}} + \frac {1}{n} \sum_ {i = 1} ^ {n} \left(e _ {t} ^ {(i)} - e _ {t + 1} ^ {(i)}\right) + \bar {e} _ {t} - \bar {e} _ {t + 1} \\ \end{array}
$$
Denote
$$
g _ {t} \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} g _ {t} ^ {(i)}, \tag {8}
$$
$$
m _ {t} \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} m _ {t} ^ {(i)} = \beta m _ {t - 1} + (1 - \beta) g _ {t}, \tag {9}
$$
$$
b _ {t} \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} b _ {t} ^ {(i)}, \tag {10}
$$
$$
\delta_ {t} \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {m _ {t} ^ {(i)}}{b _ {t} ^ {(i)}} - \frac {m _ {t}}{b _ {t}}, \tag {11}
$$
$$
e _ {t} \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} e _ {t} ^ {(i)} + \bar {e} _ {t} \tag {13}
$$
Hence, the updating rule can be summarized as
$$
\begin{array}{l} x _ {t + 1} = x _ {t} - \alpha_ {t} \bar {u} _ {t} \\ = x _ {t} - \alpha_ {t} \left(\frac {m _ {t}}{b _ {t}} + \delta_ {t} + e _ {t} - e _ {t + 1}\right) \tag {14} \\ \end{array}
$$
Lemma 1. Let  $u_{t} = \frac{m_{t}}{b_{t}}$ , the element-wise quantization function is defined in Eq.(5) can be reformulated as
$$
\mathcal {Q} \left(\left(u _ {t}\right) _ {j}\right) = \left\{ \begin{array}{l l} 1, & \text {w i t h p r o b a b i l i t y} p = \frac {\left(u _ {t}\right) _ {j} + 1}{2} \\ - 1, & \text {w i t h p r o b a b i l i t y} 1 - p \end{array} \quad (j \in \{1, 2, \dots , d \}, - 1 \leq \left(u _ {t}\right) _ {j} \leq 1). \right. \tag {15}
$$
We have  $e_t = u_t - \mathcal{Q}(u_t)$ , and then the following holds true
$$
\mathbb {E} \left[ e _ {t} \right] = 0, \mathbb {E} \left[ \| e _ {t} \| ^ {2} \right] \leq d. \tag {16}
$$
Proof. From Eq.(15), we know
$$
\begin{array}{l} \mathbb {E} \left[ \left(e _ {t}\right) _ {j} \right] = \mathbb {E} \left[ u _ {t} - \mathcal {Q} \left(u _ {t}\right) \right] \\ = \frac {1}{2} \left(\left(u _ {t}\right) _ {j} + 1\right) \left(\left(u _ {t}\right) _ {j} - 1\right) + \left(1 - \frac {1}{2} \left(\left(u _ {t}\right) _ {j} + 1\right)\right) \left(\left(u _ {t}\right) _ {j} + 1\right) = 0, \tag {17} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ (e _ {t}) _ {j} ^ {2} \right] = \mathbb {E} \left[ ((u _ {t}) _ {j} - \mathcal {Q} ((u _ {t}) _ {j})) ^ {2} \right] \\ = \frac {1}{2} \left(\left(u _ {t}\right) _ {j} + 1\right) \left(\left(u _ {t}\right) _ {j} - 1\right) ^ {2} + \left(1 - \frac {1}{2} \left(\left(u _ {t}\right) _ {j} + 1\right)\right) \left(\left(u _ {t}\right) _ {j} + 1\right) ^ {2} \tag {18} \\ = 1 - \left(\left(u _ {t}\right) _ {j}\right) ^ {2} \leq 1. \\ \end{array}
$$
$$
\mathbb {E} \left[ e _ {t} \right] = 0, \mathbb {E} \left[ \left\| e _ {t} \right\| ^ {2} \right] \leq d. \tag {19}
$$
Lemma 2. Let  $x_0 = x_1$  and  $\alpha_0 = \alpha_1$  in Algorithm 1, defining the sequence
$$
z _ {1} = x _ {1} + \alpha_ {1} \left(\delta_ {1} - e _ {1}\right) \tag {20}
$$
$$
z _ {t} = x _ {t} + \frac {\beta}{1 - \beta} \left(x _ {t} - x _ {t - 1}\right) + \frac {\alpha_ {t - 1}}{1 - \beta} \left(\delta_ {t - 1} + \beta e _ {t - 1} - e _ {t}\right), \forall t \geq 2. \tag {21}
$$
Then the following equality will hold, i.e.,
$$
z _ {t + 1} = z _ {t} + \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} - \alpha_ {t} \frac {g _ {t}}{b _ {t}} - \alpha_ {t - 1} \delta_ {t - 1} - (\alpha_ {t} - \alpha_ {t - 1}) e _ {t}. \tag {22}
$$
Proof. For  $t = 1$ , we have
$$
\begin{array}{l} z _ {2} - z _ {1} = x _ {2} + \frac {\beta}{1 - \beta} \left(x _ {2} - x _ {1}\right) + \frac {\alpha_ {1}}{1 - \beta} \left(\delta_ {1} + \beta e _ {1} - e _ {2}\right) - \left(x _ {1} + \alpha_ {1} \left(\delta_ {1} - e _ {1}\right)\right) \\ = \left(\frac {\beta}{1 - \beta} + 1\right) \left(x _ {2} - x _ {1}\right) + \frac {\alpha_ {1}}{1 - \beta} \left(\delta_ {1} + \beta e _ {1} - e _ {2}\right) - \alpha_ {1} \left(\delta_ {1} - e _ {1}\right) \tag {23} \\ = - \frac {\alpha_ {1}}{1 - \beta} \left(\frac {(1 - \beta) g _ {1}}{b _ {1}} + \delta_ {1} + e _ {1} - e _ {2}\right) + \frac {\alpha_ {1}}{1 - \beta} \left(\delta_ {1} + \beta e _ {1} - e _ {2}\right) - \alpha_ {1} \left(\delta_ {1} - e _ {1}\right) \\ = - \alpha_ {1} \frac {g _ {1}}{b _ {1}} - \alpha_ {0} \delta_ {1} \\ \end{array}
$$
where the second equality follows the updating rule in Eq.(14).
For  $t\geq 2$  , following the updating rule in Eq.(14), we have
$$
\begin{array}{l} x _ {t + 1} - x _ {t} + \alpha_ {t} \left(\delta_ {t} + e _ {t} - e _ {t + 1}\right) = - \alpha_ {t} \frac {m _ {t}}{b _ {t}} \\ = - \alpha_ {t} \frac {\beta m _ {t - 1} + (1 - \beta) g _ {t}}{b _ {t}} \tag {24} \\ = \beta \left(x _ {t} - x _ {t - 1} + \alpha_ {t - 1} \left(\delta_ {t} + e _ {t - 1} - e _ {t}\right)\right) \\ + \beta \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} - (1 - \beta) \alpha_ {t} \frac {g _ {t}}{b _ {t}} \\ \end{array}
$$
We know  $x_{t+1} - x_t + \alpha_t(e_t - e_{t+1}) = (1 - \beta)(x_{t+1} + -\alpha_t(e_{t+1} - \delta_t)) - (1 - \beta)(x_t - \alpha_t e_t) + \beta(x_{t+1} - x_t + \alpha_t(\delta_t + e_t - e_{t+1}))$ , so Eq. (24) can be rearranged as
$$
\begin{array}{l} (1 - \beta) \left(x _ {t + 1} + \alpha_ {t} \left(\delta_ {t} - e _ {t + 1}\right)\right) + \beta \left(x _ {t + 1} - x _ {t} + \alpha_ {t} \left(\delta_ {t} + e _ {t} - e _ {t + 1}\right)\right) \\ = (1 - \beta) \left(x _ {t} - \alpha_ {t} e _ {t}\right) + \beta \left(x _ {t} - x _ {t - 1} + \alpha_ {t - 1} \left(\delta_ {t - 1} + e _ {t - 1} - e _ {t}\right)\right) \tag {25} \\ + \beta \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} - (1 - \beta) \alpha_ {t} \frac {g _ {t}}{b _ {t}} \\ \end{array}
$$
Divided both sides by  $1 - \beta$ , we obtain
$$
\begin{array}{l} x _ {t + 1} + \alpha_ {t} \left(\delta_ {t} - e _ {t + 1}\right) + \frac {\beta}{1 - \beta} \left(x _ {t + 1} - x _ {t} + \alpha_ {t} \left(\delta_ {t} + e _ {t} - e _ {t + 1}\right)\right) \\ = x _ {t} + \alpha_ {t - 1} \left(\delta_ {t - 1} - e _ {t}\right) + \frac {\beta}{1 - \beta} \left(x _ {t} - x _ {t - 1} + \alpha_ {t - 1} \left(\delta_ {t - 1} + e _ {t - 1} - e _ {t}\right)\right) \tag {26} \\ + \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \\ - \alpha_ {t} \frac {g _ {t}}{b _ {t}} - \alpha_ {t - 1} \delta_ {t - 1} - (\alpha_ {t} - \alpha_ {t - 1}) e _ {t} \\ \end{array}
$$
Rearranging Eq. (26), we have
$$
\begin{array}{l} x _ {t + 1} + \frac {\beta}{1 - \beta} \left(x _ {t + 1} - x _ {t}\right) + \frac {\alpha_ {t}}{1 - \beta} \left(\delta_ {t} + \beta e _ {t} - e _ {t + 1}\right) \\ = x _ {t} + \frac {\beta}{1 - \beta} \left(x _ {t} - x _ {t - 1}\right) + \frac {\alpha_ {t - 1}}{1 - \beta} \left(\delta_ {t - 1} + \beta e _ {t - 1} - e _ {t}\right) \tag {27} \\ + \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} \\ - \alpha_ {t} \frac {g _ {t}}{b _ {t}} - \alpha_ {t - 1} \delta_ {t - 1} - (\alpha_ {t} - \alpha_ {t - 1}) e _ {t} \\ \end{array}
$$
Define the sequence
$$
z _ {t} = x _ {t} + \frac {\beta}{1 - \beta} \left(x _ {t} - x _ {t - 1}\right) + \frac {\alpha_ {t - 1}}{1 - \beta} \left(\delta_ {t - 1} + \beta e _ {t - 1} - e _ {t}\right) \tag {28}
$$
We finally obtain
$$
z _ {t + 1} = z _ {t} + \frac {\beta}{1 - \beta} \left(\frac {\alpha_ {t - 1}}{b _ {t - 1}} - \frac {\alpha_ {t}}{b _ {t}}\right) \odot m _ {t - 1} - \alpha_ {t} \frac {g _ {t}}{b _ {t}} - \alpha_ {t - 1} \delta_ {t - 1} - (\alpha_ {t} - \alpha_ {t - 1}) e _ {t}. \tag {29}
$$
Recalling  $x_{1} = x_{0}$  and  $\alpha_{1} = \alpha_{0}$ , we have  $\frac{\alpha_{1}}{b_{1}} = \frac{\alpha_{0}}{b_{0}}$ . Then, combining Eq.(23) and Eq.(29), we obtain the conclusion.
Proof. By the gradient Lipschitz continuous in Assumption 2 and Lemma 2, we obtain