SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C.3 Effectiveness of Block Size in Block-wise Normalization
In Fig. 29,30,31, we show the effect of block size on quantization error for both first and second moments. Fig. 29,30 shows that B2048 normalization quantizes a significant portion of the points to zero, resulting in poor approximation based on the histogram. However, when we utilize a smaller block size of 128, the quantization performance improves. Fig. 31 shows smaller block size improves quantization quality on second moment.
Figure 24: Histogram of second moment of the attention layer  $(\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V)$  in transformer block layer-2 of GPT-2 Medium at epoch 2. A case where rank-1 normalization is worse than block-wise normalization with block size 128. Top: B128/DE-0 quantization. Bottom: Rank-1/DE-0 quantization.
Figure 25: Histogram of second moment of  $\mathbf{W}^2$  in transformer block layer-4 of RoBERTa-Large at epoch 8. A case where rank-1 normalization is better than block-wise normalization with block size 128. Top: B128/DE-0 quantization. Bottom: Rank-1/DE-0 quantization.
Figure 26: Histogram of second moment of  $\mathbf{W}^V$  in transformer block layer-2 of RoBERTa-Large at epoch 8. A case where rank-1 normalization is worse than block-wise normalization with block size 128. Top: B128/DE-0 quantization. Bottom: Rank-1/DE-0 quantization.
Figure 27: Histogram of second moment of  $\mathbf{W}^2$  in transformer block layers.0.blocks.0 of Swin-T at epoch 210. A case where rank-1 normalization is better than block-wise normalization with block size 128. Top: B128/DE-0 quantization. Bottom: Rank-1/DE-0 quantization.
Figure 28: Histogram of second moment of  $\mathbf{W}^{\mathrm{O}}$  in transformer block layers.1 blocs.0 of Swin-T at epoch 210. A case where rank-1 normalization is worse than block-wise normalization with block size 128. Top: B128/DE-0 quantization. Bottom: Rank-1/DE-0 quantization.
Figure 29: Histogram of first moment of  $\mathbf{W}^1$  in transformer block layer-20 of GPT-2 Medium at epoch 2. Top: B128/DE quantization. Bottom: B2048/DE quantization.
Figure 30: Histogram of first moment of  $\mathbf{W}^{\mathrm{O}}$  in transformer block layer-22 of RoBERTa-L at epoch 8. Top: B128/DE quantization. Bottom: B2048/DE quantization.
Figure 31: Histogram of second moment of  $\mathbf{W}^1$  in transformer block layers.0 blocs.0 of Swin-T at epoch 210. Top: B128/DE-0 quantization. Bottom: B2048/DE-0 quantization.
D.1 Quantization
There are several parameters in deep neural networks that play a delicate role without occupying too much memory, such as normalization layers and bias. In this context, we establish a rule to determine which parameters should not be quantized. For all the experiments we conducted, the rule is straightforward: tensors with a size smaller than or equal to 4096 will not be quantized. However, for larger models with a hidden size exceeding 4096, it is advisable to exclude the bias and normalization layers from quantization. Regarding the quantization settings, as stated in Sec. 5, we employ block-wise normalization with a block size of 128, dynamic exponent mapping for first moment and rank-1 normalization, linear mapping for second moment. When we apply factorization on second moment, only tensors with a dimension greater than or equal to 2 will be factorized while 1-dimensional tensors that meet the specified rule will still be quantized.
8-bit Adam [15] also uses the threshold of 4096 about size to determine whether or not to quantize parameters. Additionally, the implementation in huggingface does not quantize the parameters in Embedding layers regardless of the model used. Consequently, we compare our method with the 8-bit Adam that does not quantize Embedding.
D.2 Hyperparameters and Training Details
In each benchmark, unless otherwise specified, we maintain the same hyperparameters for a given optimize across different quantization schemes. Additionally, we use same optimizer hyperparameters across various optimizers, including our 4-bit optimizers, 8-bit Adam [15], SM3 [2], Adafactor [46] and the full precision counterpart AdamW [32]. For Adafactor, we use  $\beta_{1} > 0$  as default setting which is same as the  $\beta_{1}$  value used in AdamW. Also, the case where  $\beta_{1} = 0$  is compared. The other newly introduced hyperparameters in Adafactor are set to their default values and remain fixed throughout the experiments. For SM3, we compare with the  $\beta_{1} > 0$  configuration, same as the  $\beta_{1}$  value used in AdamW.
Table 10: The hyperparameters for RoBERTa-L fine-tuning on GLUE.
Dataset	MNLI	QNLI	QQP	RTE	MRPC	SST-2	CoLA	STS-B
Batch Size	32	32	32	16	16	32	16	16
LR	1e-5	1e-5	1e-5	2e-5	1e-5	1e-5	1e-5	2e-5
Warmup	7432	1986	28318	122	137	1256	320	214
Max Train Steps	123873	33112	113272	2036	2296	20935	5336	3598
Max Seq. Len.	128	128	128	512	512	512	512	512
Table 11: The hyperparameters for RoBERTa-L fine-tuning on SQuAD and SQuAD 2.0.
Dataset	SQuAD & SQuAD 2.0
Batch Size	48
LR	1.5e-5
# Epochs	2
Warmup Ratio	0.06
Max Seq. Len.	384
Table 12: The hyperparameters for GPT-2 on E2E.
Dataset	E2E
Training
Batch Size	8
LR	4e-5
# Epochs	5
Warmup	500
Max Seq. Len.	512
Label Smooth	0.1
Inference
Beam Size	10
Length Penalty	0.8
no repeat ngram size	4
RoBERTa We train all of our RoBERTa-L models with PyTorch Huggingface $^{\dagger\dagger}$ . On GLUE benchmark, we mainly follow the hyperparameters in fairseq [36]. We use  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\epsilon = 1\mathrm{e - }6$ , a weight decay factor of 0.1 and linear learning rate schedule. Other hyperparameters are listed in Tab. 10. On SQuAD benchmark, we mainly follow the reported hyperparameters in RoBERTa paper [30]. We use  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\epsilon = 1\mathrm{e - }6$ , a weight decay factor of 0.01 and linear learning rate schedule. The other hyperparameters are listed in Tab. 11. On both datasets, we report the median and standard deviation results over 5 runs, the result in each run is taken from the best epoch. We utilize single RTX 3090 or 4090 GPU for runs of each task in GLUE datasets and four RTX 3090 or 4090 GPUs for SQuAD and SQuAD 2.0.
$\dagger \dagger$  https://github.com/huggingface/transformers
On SQuAD 2.0, there may be a performance gap observed between the reproduced results using 32-bit AdamW and the original results reported in the original paper. This is because there are some questions without answers in SQuAD 2.0. It is worth noting that the approach employed by Liu et al. [30] to handle unanswered questions may differ from the solutions utilized in the BERT paper [17], which is the reference implementation we are using
GPT-2 We train all of our GPT-2 Medium models with the LoRA codebase $^{\ddagger\ddagger}$ . We mainly follow the hyperparameters in [28] and [24]. We use  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 1\mathrm{e - }6$ , a weight decay factor of 0.01 and linear learning rate schedule. The other hyperparameters used in GPT-2 are listed in Tab. 12. We report the mean and standard deviation results over 3 runs, the result in each run is taken from the best epoch. We utilize fours RTX 3090 or 4090 GPUs for runs of this task.
Transformer We train all of our Transformer-Base models for machine translation with codebase $\S\S$ . We completely follow the hyperparameters in the codebase. We report the mean and standard deviation results over 3 runs, the result in each run is taken from the best epoch. We utilize eight RTX 3090 or 4090 GPUs for runs of this task.
Swin We train all of our Swin-T models with its official codebase $^{\text{III}}$ . We completely follow the hyperparameters in the codebase. We report the mean and standard deviation results over 3 runs, the result in each run is taken from the best epoch. We utilize eight RTX 3090 or 4090 GPUs for runs of this task.
LLaMA We fine-tune LLaMA-7B, LLaMA-13B and LLaMA-33B with Alpaca codebase\*\*. We follow the hyperparameters in the codebase for LLaMA-7B and LLaMA-13B, and the hyperparameters of LLaMA-33B are consistent with LLaMA-13B. We fine-tune LLaMA-7B with two A100 80GB GPUs. The training loss curve is the mean results over 3 runs. For LLaMA-7B, we enable Fully Sharded Data Parallelism (FSDP), which packs parameters into 1-dimensional array. This packing process makes it difficult to apply factorization directly without additional engineering efforts. Consequently, we only compare the performance of 4-bit AdamW with its full precision counterpart.
D.3 Memory and Computing Efficiency
In Tab. 4, we present measurements of memory usage in practical settings, i.e. training configuration described in Sec. D.2. Specifically, we measure the memory usage for LLaMA-7B using 2 A100 80G GPUs, RoBERTa-L using 1 RTX 4090 GPU, and GPT-2 Medium using 4 RTX 4090 GPUs. Additionally, the time measurement for RoBERTa-L is conducted on the RTE task.
E Quantization Formulation Details
E.1 Signed Case
In this section, we discuss quantization function for signed tensors and the differences compared to unsigned case. Regarding the normalization operator, the only difference lies in the fact that the sign of the tensor remains unchanged before and after normalization. Formally, let  $\mathbf{N}$  be the normalization operator for the unsigned cases. For the signed case, the normalization can be defined as
$$
n _ {j} := \operatorname {s i g n} (x _ {j}) \mathbf {N} (| x _ {j} |).
$$
Therefore, the unit interval for signed case is [-1, 1]. Regarding the mapping operator, the difference lies in the values of quantization mappings. See App. E.2 for more details.
E.2 Quantization Mappings
In this work, we mainly consider linear mapping and dynamic exponent mapping [13]. See Fig. 32 for illustration of quantization mappings.
$\text{串}$  https://github.com/microsoft/LoRA
$\S \S$  https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/Transformer
https://github.com/microsoft/Swin-Transformer
*** https://github.com/tatsu-lab/stanford_alpaca
Figure 32: Visualization of the quantization mappings for the linear and dynamic exponent at 4-bit precision. Left: Signed case. Right: Unsigned case.
Linear mapping It is notable that the linear mapping considered in our work does not include zero in both signed case and unsigned case. Actually, we only use linear mapping in unsigned case, which is defined as torch.linspace(0, 1, (2 ** b) + 1) [1:].
Dynamic exponent mapping Let  $b$  be the total bits. In the main text, we mentioned that dynamic exponent takes the form  $\mathbf{T}(i) = 10^{-E(i)}\mathrm{fraction}(i)$ . In following paragraphs, we will define the dynamic exponent mapping formally based on the binary representation.
In unsigned case, dynamic exponent mapping [13] is composed of exponent bits  $E$ , one indicator bit and fraction bits  $F$ , where  $b = 1 + E + F$ . It uses the number of leading zero bits  $E$  represents the exponent with base 10. The first bit, which is one, serves as an indicator bit that separates the exponent and the unsigned linear fraction. The remaining bits  $F$  represent an unsigned linear fraction distributed evenly in (0.1, 1), which is formally defined as
$$
p _ {j} = \frac {1 - 0 . 1}{2 ^ {F}} j + 0. 1, \quad 0 \leq j \leq 2 ^ {F},
$$
$$
\operatorname {f r a c t i o n} [ k ] = \frac {p _ {k} + p _ {k + 1}}{2}, \quad 0 \leq k \leq 2 ^ {F} - 1.
$$
Therefore, a number with  $E$  exponent bits and  $F$  fraction bits valued  $k$  has a value of
$$
1 0 ^ {- E} \times \text {f r a c t i o n} [ k ].
$$
For signed case, the only difference is that dynamic exponent mapping additionally uses the first bit as the sign thus we have  $b = 1 + E + 1 + F$ . Specially, at 8-bit case, we learn from the codebase  $^{\dagger\ddagger\dagger}$  that dynamic exponent mapping assign  $00000000_{2} = 0_{10}$ ,  $00000001_{2} = 1_{10}$  in unsigned case and assign  $10000000_{2}$  and  $00000000_{2}$  with  $1_{10}$  and  $0_{10}$ , respectively. This means  $-1_{10}$  is not defined and the mapping is not symmetric in signed case. Finally, after collecting all the represented numbers and arranging them in a sorted, increasing list, which has a length of  $2^{b}$ , the quantization mapping  $\mathbf{T}(i)$  returns the  $i$ -th element of this list.
The construction of dynamic exponent mapping is unrelated to the number of bits. Therefore, when we say we barely turn the 8-bit optimizer into 4-bit optimizer, it just uses 4 total bits. The corner cases mentioned in last paragraph remain unchanged.
E.3 Stochastic Rounding
Stochastic rounding is only used in Tab. 1. In this section, we talk about how to integrate stochastic rounding into our formulation of quantization. When stochastic rounding is used, the definition of mapping  $\mathbf{M}$  has some minor changes. Specifically,  $\mathbf{M}$  is still an element-wise function and defined as
$$
\mathbf {M} (n _ {j}) = \arg \min _ {0 \leq i <   2 ^ {b}} \left\{n _ {j} - \mathbf {T} (i): n _ {j} - \mathbf {T} (i) \geq 0 \right\} \cup \arg \max _ {0 \leq i <   2 ^ {b}} \left\{n _ {j} - \mathbf {T} (i): n _ {j} - \mathbf {T} (i) \leq 0 \right\}.
$$
$\dagger \ddagger$  https://github.com/TimDettmers/bitsandbytes
In other words,  $\mathbf{M}$  maps each entry  $n_j$  to the maximal index set  $\mathbf{M}(n_j)$  such that for any  $i \in \mathbf{M}(n_j)$  there is no other  $0 \leq k \leq 2^b - 1$  with  $\mathbf{T}(k)$  lying between  $\mathbf{T}(i)$  and  $n_j$ . Actually,  $\mathbf{M}$  acts as a filter of  $\mathbf{T}$  and give a more fine-grained range of quantized output candidates. In this definition,  $\mathbf{M}(n_j)$  has only one or two points since only stochastic rounding is considered in the final step.
Finally, we define stochastic rounding  $\mathbf{R}_s$ . When  $\mathbf{M}(n_j)$  only has one point,  $\mathbf{R}_s$  just output this point. When  $\mathbf{M}(n_j)$  has two points  $q_1$  and  $q_2$  with  $\mathbf{T}(q_1) < n_j < \mathbf{T}(q_2)$ , stochastic rounding  $(\mathbf{R}_s)$  is defined as
$$
\mathbf {R} _ {s} \left(n _ {j}, q _ {1}, q _ {2}\right) = \left\{ \begin{array}{l} q _ {2}, \text {w i t h p r o b a .} \frac {n _ {j} - \mathbf {T} \left(q _ {1}\right)}{\mathbf {T} \left(q _ {2}\right) - \mathbf {T} \left(q _ {1}\right)} \\ q _ {1}, \text {w i t h p r o b a .} \frac {\mathbf {T} \left(q _ {2}\right) - n _ {j}}{\mathbf {T} \left(q _ {2}\right) - \mathbf {T} \left(q _ {1}\right)} \end{array} \right.
$$
F Compression-based Memory Efficient Optimizer Instances
In this section, we present some examples about compression-based memory efficient optimizers. See Compression-based Memory Efficient SGDM in Alg. 2 and Adam in Alg. 3.
Algorithm 2 Compression-based Memory Efficient SGDM
Require: initial parameter  $\theta_0\in \mathbb{R}^p$  , learning rate  $\alpha$  , initial first moment  $\bar{m}_0 = 0$  , total number of iterations  $T$  and momentum parameter  $\beta$    
1: for  $t = 1,2,\ldots ,T$  do   
2: Sample a minibatch  $\zeta_t$  and get stochastic gradient  $g_{t} = \nabla_{\theta}f(\theta_{t - 1},\zeta_{t})$    
3:  $m_{t - 1}\gets \mathrm{decompress}(\bar{m}_{t - 1})$    
4:  $m_t\gets \beta \cdot m_{t - 1} + g_t$    
5:  $\theta_t\gets \theta_{t - 1} - \alpha \cdot m_t$    
6:  $\bar{m}_t\gets \mathrm{compress}(m_t)$    
7: end for   
8: return  $\theta_T$
Algorithm 3 Compression-based Memory Efficient Adam
Require: initial parameter  $\theta_0\in \mathbb{R}^p$  , learning rate  $\alpha$  , initial moments  $\bar{m}_0 = 0,\bar{v}_0 = 0$  , total number of iterations  $T$  and hyperparameters  $\beta_{1},\beta_{2},\epsilon$    
1: for  $t = 1,2,\ldots ,T$  do   
2: Sample a minibatch  $\zeta_t$  and get stochastic gradient  $g_{t} = \nabla_{\theta}f(\theta_{t - 1},\zeta_{t})$    
3:  $m_{t - 1},v_{t - 1}\gets \mathrm{decompress}(\bar{m}_{t - 1}),\mathrm{decompress}(\bar{v}_{t - 1})$    
4:  $m_t\gets \beta_1\cdot m_{t - 1} + (1 - \beta_1)\cdot g_t$    
5:  $v_{t}\gets \beta_{2}\cdot v_{t - 1} + (1 - \beta_{1})\cdot g_{t}^{2}$    
6:  $\hat{m}_t\gets m_t / (1 - \beta_1^t)$    
7:  $\hat{v}_t\gets v_t / (1 - \beta_2^t)$    
8:  $\theta_t\gets \theta_{t - 1} - \alpha \cdot \hat{m}_t / (\sqrt{\hat{v}_t} +\epsilon)$    
9:  $\bar{m}_t,\bar{v}_t\gets \mathrm{compress}(m_t),\mathrm{compress}(v_t)$    
10: end for   
11: return  $\theta_T$
G Rank-1 Normalization
In this section, we present the detailed formulation of rank-1 normalization in Alg. 4.
H Theoretical Analysis
The convergence of low-bit optimizers can be guaranteed if their fp32 counterparts converge. Here we provide a theorem about the convergence of quantized SGDM (Alg. 2) under some assumptions. We believe the convergence of low-bit AdamW could be inferred from the convergence of AdamW.
Algorithm 4 Rank-1 Normalization
Require: tensor  $x \in \mathbb{R}^{d_1 \times \dots \times d_p}$ ; statistics  $\mu_r \in \mathbb{R}^{d_r}$  for  $1 \leq r \leq p$ ; permutation function  $\Phi$  mapping  $\{1, \ldots, d\}$  to indices of tensor  $x$ , where  $d = d_1 \times \dots \times d_p$ .
1: for  $r = 1,2,\ldots ,p$  do
2: for  $j = 1,2,\ldots ,d_r$  do
3:  $\mu_{r,j} = \max_{i_1,\dots ,i_{r - 1},i_{r + 1},\dots ,i_p}\left|x_{[i_1,\dots ,i_{r - 1},j,i_{r + 1},\dots ,i_p]}\right|$
4: end for
5: end for
6: for  $i = 1,2,\ldots ,d$  do
7:  $M_{i} = \min_{1\leq r\leq p}\mu_{r,\Phi (i)_{r}}$
8: end for
9: reshape 1-dimensional array  $M$  to the same shape as  $x$
10: return  $x / M$
First, we make some assumptions. The first three are rather standard in stochastic optimization literature, while last two depict properties of stochastic quantizers.
1. (Convexity) The objective function is convex and has an unique global minimum  $f(\theta^{*})$ .
2. (Smoothness) The objective  $f(\theta)$  is continuous differentiable and  $L$ -smooth;
3. (Moments of stochastic gradient) The stochastic gradient  $g$  is unbiased, i.e.,  $\mathbb{E}[g(\theta)] = \nabla f(\theta)$ , and has bounded variance, i.e.,  $\mathbb{E}\left[\| g(\theta) - \nabla f(\theta)\| ^2\right] < \sigma^2$ ,  $\forall \theta \in \mathbb{R}^d$ .
4. (Unbiased quantizer)  $\forall x\in \mathbb{R}^d$ $\mathbb{E}[Q(x)] = x$
5. (Bounded quantization variance)  $\forall x\in \mathbb{R}^d$ $\mathbb{E}\left[\| Q_m(x) - x\| ^2\right]\leq \sigma_m^2.$
Then, we have following theorem:
Theorem 1. Consider the Algorithm 2 with Assumptions 1-5. Let  $\alpha \in (0, \frac{1 - \beta}{L}]$ , then for all  $T > 0$  we have
$$
\begin{array}{l} \mathbb {E} \left[ f (\bar {\theta} _ {T}) - f _ {*} \right] \leq \frac {1}{2 T} \left(\frac {L \beta}{1 - \beta} + \frac {1 - \beta}{\alpha}\right) \| \theta_ {0} - \theta_ {*} \| ^ {2} \\ + \frac {\alpha \sigma^ {2}}{(1 - \beta)} + \frac {\alpha \sigma_ {m} ^ {2}}{(1 - \beta)}. \tag {2} \\ \end{array}
$$
where  $\bar{\theta}_T = \frac{1}{T}\sum_{i = 0}^{T - 1}\theta_i$
H.1 Proof of Theorem 1
To prove Theorem 1, we need some useful lemmas.
Lemma 1. In Algorithm 2, The conditional first and second moments of  $g_{t}$  satisfies
$$
\mathbb {E} \left[ g _ {t} \mid \theta_ {t - 1} \right] = \nabla f \left(\theta_ {t - 1}\right) \tag {3}
$$
$$
\mathbb {E} \left[ \| g _ {t} \| ^ {2} | \theta_ {t - 1} \right] \leq \| \nabla f (\theta_ {t - 1}) \| ^ {2} + \sigma^ {2} \tag {4}
$$
Proof. By assumption, we easily have
$$
\mathbb {E} \left[ g _ {t} \mid \theta_ {t - 1} \right] = \nabla f (\theta_ {t - 1}).
$$
With Assumption 3, it holds true that
$$
\begin{array}{l} \mathbb {E} \left[ \| g (\theta) \| ^ {2} \right] = \mathbb {E} \left[ \| g (\theta) - \nabla f (\theta) + \nabla f (\theta) \| ^ {2} \right] \\ = \mathbb {E} \left[ \| g (\theta) - \nabla f (\theta) \| ^ {2} \right] + \mathbb {E} \left[ \| \nabla f (\theta) \| ^ {2} \right] + 2 \mathbb {E} [ \langle g (\theta) - \nabla f (\theta), \nabla f (\theta) \rangle ] \\ = \mathbb {E} \left[ \| g (\theta) - \nabla f (\theta) \| ^ {2} \right] + \mathbb {E} \left[ \| \nabla f (\theta) \| ^ {2} \right] \\ \leq \sigma^ {2} + \| \nabla f (\theta) \| ^ {2}, \\ \end{array}
$$
which implies the second part.
Lemma 2. If Assumptions 3-5 hold, then sequence  $\{z_t\}$  satisfies
$$
z _ {t + 1} - z _ {t} = \frac {1}{1 - \beta} \left(\theta_ {t + 1} - \theta_ {t}\right) - \frac {\beta}{1 - \beta} \left(\theta_ {t} - \theta_ {t - 1}\right) \tag {5}
$$
$$
\mathbb {E} \left[ z _ {t + 1} - z _ {t} \right] = \frac {- \alpha}{1 - \beta} \nabla f \left(\theta_ {t}\right) \tag {6}
$$
$$
\mathbb {E} [ \| z _ {t + 1} - z _ {t} \| ^ {2} ] \leq 2 \left(\frac {\alpha}{1 - \beta}\right) ^ {2} \left(\mathbb {E} [ \| g _ {t + 1} \| ^ {2} ] + \sigma_ {m} ^ {2}\right). \tag {7}
$$
Proof. By definition of  $z_{t}$ , we have the first equation immediately. Take expectation on the first equation and we get
$$
\mathbb {E} \left[ z _ {t + 1} - z _ {t} \right] = \frac {1}{1 - \beta} \mathbb {E} \left[ \theta_ {t + 1} - \theta_ {t} \right] - \frac {\beta}{1 - \beta} \mathbb {E} \left[ \theta_ {t} - \theta_ {t - 1} \right].
$$
$$
\begin{array}{l} \mathbb {E} \left[ \theta_ {t + 1} - \theta_ {t} \right] = \mathbb {E} \left[ \theta_ {t + 1} - \left(\theta_ {t} - \alpha m _ {t + 1}\right) \right] - \mathbb {E} \left[ \alpha m _ {t + 1} \right] \\ = - \alpha \mathbb {E} [ m _ {t + 1} ] \\ = - \alpha \mathbb {E} [ \beta m _ {t} + g _ {t + 1} ] \\ = - \alpha \beta \mathbb {E} [ m _ {t} ] - \alpha \nabla f (\theta_ {t}), \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \theta_ {t} - \theta_ {t - 1} \right] = \mathbb {E} \left[ \theta_ {t} - \left(\theta_ {t - 1} - \alpha m _ {t}\right) \right] - \mathbb {E} \left[ \alpha m _ {t} \right] \\ = - \alpha \mathbb {E} [ m _ {t} ], \\ \end{array}
$$
which gives the second equation.
$$
\mathbb {E} \left[ z _ {t + 1} - z _ {t} \right] = \frac {- \alpha}{1 - \beta} \nabla f \left(\theta_ {t}\right)
$$
For the last equation, since
$$
\begin{array}{l} z _ {t + 1} - z _ {t} = \frac {1}{1 - \beta} \left(\theta_ {t + 1} - \theta_ {t}\right) - \frac {\beta}{1 - \beta} \left(\theta_ {t} - \theta_ {t - 1}\right) \\ = - \frac {\alpha}{1 - \beta} \left(m _ {t + 1} - \beta m _ {t}\right) \\ \end{array}
$$
Take expectation and we have
$$
\begin{array}{l} \mathbb {E} \left[ \| z _ {t + 1} - z _ {t} \| ^ {2} \right] = \left(\frac {\alpha}{1 - \beta}\right) ^ {2} \mathbb {E} \left[ \| m _ {t + 1} - \beta m _ {t} \| ^ {2} \right] \\ \leq 2 \left(\frac {\alpha}{1 - \beta}\right) ^ {2} \left(\mathbb {E} \left[ \| m _ {t + 1} - (\beta m _ {t} + g _ {t + 1}) \| ^ {2} \right] + \mathbb {E} \left[ \| g _ {t + 1} \| ^ {2} \right]\right) \\ \leq 2 \left(\frac {\alpha}{1 - \beta}\right) ^ {2} \left(\mathbb {E} \left[ \| g _ {t + 1} \| ^ {2} \right] + \sigma_ {m} ^ {2}\right). \\ \end{array}
$$
Proof of Theorem 1. From Lemma 2, we have
$$
\mathbb {E} \left[ \| z _ {t + 1} - z _ {t} \| ^ {2} \right] \leq 2 \left(\frac {\alpha}{1 - \beta}\right) ^ {2} \left(\mathbb {E} \left[ \| g _ {t + 1} \| ^ {2} \right] + \sigma_ {m} ^ {2}\right).
$$
Substituting Lemma 1 gives
$$
\mathbb {E} \left[ \| z _ {t + 1} - z _ {t} \| ^ {2} \right] \leq 2 \left(\frac {\alpha}{1 - \beta}\right) ^ {2} \left(\| \nabla f \left(\theta_ {t}\right) \| ^ {2} + \sigma^ {2} + \sigma_ {m} ^ {2}\right). \tag {8}
$$
Suppose  $\theta_{*}$  is the optimal parameter and  $f_{*} = f(\theta_{*})$  is the minimal objective value. First, we have
$$
\left\| z _ {t + 1} - \theta_ {*} \right\| ^ {2} = \left\| z _ {t} - \theta_ {*} \right\| ^ {2} + 2 \left\langle z _ {t} - \theta_ {*}, z _ {t + 1} - z _ {t} \right\rangle + \left\| z _ {t + 1} - z _ {t} \right\| ^ {2}
$$
Take expectation over the randomness in the  $(t + 1)$  -th step, we have
$$
\begin{array}{l} \mathbb {E} [ \| z _ {t + 1} - \theta_ {*} \| ^ {2} ] = \| z _ {t} - \theta_ {*} \| ^ {2} - \frac {2 \alpha}{1 - \beta} \left\langle z _ {t} - \theta_ {*}, \nabla f (\theta_ {t}) \right\rangle + \mathbb {E} [ \| z _ {t + 1} - z _ {t} \| ^ {2} ] \\ = \left\| z _ {t} - \theta_ {*} \right\| ^ {2} - \frac {2 \alpha}{1 - \beta} \left\langle \theta_ {t} - \theta_ {*}, \nabla f (\theta_ {t}) \right\rangle \\ - \frac {2 \alpha \beta}{(1 - \beta) ^ {2}} \left\langle \theta_ {t} - \theta_ {t - 1}, \nabla f (\theta_ {t}) \right\rangle + \mathbb {E} [ \| z _ {t + 1} - z _ {t} \| ^ {2} ] \\ \end{array}
$$
Since  $f$  is continuously differentiable and L-smooth, we have the following inequalities. [34]