SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[53] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated backpropagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1723-1732. PMLR, 2019.
[54] Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107-194, 2012.
[55] Chenggen Shi, Jie Lu, and Guangquan Zhang. An extended kuhn-tucker approach for linear bilevel programming. Applied Mathematics and Computation, 162(1):51-63, 2005.
[56] Daouda Sow, Kaiyi Ji, Ziwei Guan, and Yingbin Liang. A constrained optimization approach to bilevel optimization with multiple inner minima. arXiv preprint arXiv:2203.01123, 2022.
[57] Bradly Stadie, Lunjun Zhang, and Jimmy Ba. Learning intrinsic rewards as a bi-level optimization problem. In Conference on Uncertainty in Artificial Intelligence, pages 111-120. PMLR, 2020.
[58] Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory, pages 845–861. PMLR, 2020.
[59] Davoud Ataee Tarzanagh and Laura Balzano. Online bilevel optimization: Regret analysis of online alternating gradient methods. arXiv preprint arXiv:2207.02829, 2022.
[60] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020.
[61] Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In International Conference on Machine Learning, pages 449-457. PMLR, 2016.
[62] Zhuoran Yang, Zuyue Fu, Kaiqing Zhang, and Zhaoran Wang. Convergent reinforcement learning with function approximation: A bilevel optimization perspective. 2018.
[63] Hongyuan Zhan, Gabriel Gomes, Xiaoye S Li, Kamesh Madduri, and Kesheng Wu. Efficient online hyperparameter optimization for kernel ridge regression with applications to traffic time series prediction. arXiv preprint arXiv:1811.00620, 2018.
[64] Lijun Zhang, Tianbao Yang, Jinfeng Yi, Rong Jin, and Zhi-Hua Zhou. Improved dynamic regret for non-degenerate functions. Advances in Neural Information Processing Systems, 30, 2017.
[65] Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning, pages 26693-26712. PMLR, 2022.
[66] Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Learning for Dynamics and Control, pages 48-59. PMLR, 2021.
[67] Zhenxun Zhuang, Yunlong Wang, Kezi Yu, and Songtao Lu. No-regret non-convex online meta-learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3942-3946. IEEE, 2020.
[68] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pages 928–936, 2003.
A Discussion about practical applications of OBO
(1) In the traffic flow prediction problem [63], since the data collected by the traffic sensors arrive at the controller continuously and frequently, the hyperparameters need to be optimized (in the outer level) quickly in an online manner in order to guarantee the performance of the prediction model (in the inner level). Keeping the hyperparameters static for the prediction model may result in sub-optimal performance, because the distribution of the traffic flow can change gradually. Further, the controller may not have global information of the traffic flows, because it is exorbitantly expensive to deploy sensors to cover all traffic flows.
(2) In the wireless network control problem [37], the controller allocates resources, e.g., wireless channel bandwidth, to the users, where each user can have its own utility function depending on the wireless channel conditions given the allocated resources. Since wireless channels are usually time-varying, the controller has to continuously update the resource allocation quickly to maximize the network performance. The rate allocation decisions (deciding what packet rate a user transmits at any given time) and scheduling decisions (which users transmit) are done at a fast time-scale (in the inner level), while determining the utility functions of the users could be done at a slower time scale (in the outer level). Further these decisions need to be made in a distributed fashion, so under local knowledge of the channel conditions and interference levels.
In these applications, the current offline bilevel optimization framework cannot be directly applied because of the streaming data, the time-varying functions and possibly the limited information about the system. In contrast, online bilevel optimization has great potential for these online applications. Moreover, the decision making in these online applications also needs to be fast and efficient without the need of knowing all previous functions. Thus, the regret captures the performance of the learning model over the sequential process rather than just the performance of a final output model in the offline setting. How to design such algorithms with a sublinear regret guarantee is very important for making online bilevel optimization more practical in real applications.
B Applications of OBO in meta-learning and adversarial training
Online meta-learning In online meta-learning, learning tasks arrive one at a time, and the agent aims to learn a good meta-model  $\theta$  based on the past tasks in a sequential manner, which can be quickly adapted to a good task-specific model  $\phi$  for the current task. Specifically, each task  $t$  has a training dataset  $\mathcal{D}_t^{tr}$  and a testing dataset  $\mathcal{D}_t^{te}$ , and given a meta-model  $\theta$ , the optimal task model is defined as
$$
\phi_ {t} ^ {*} (\theta) \in \arg \min _ {\phi} g _ {t} (\theta , \phi) = \mathcal {L} _ {t} (\phi , \mathcal {D} _ {t} ^ {t r}) + \lambda \| \theta - \phi \| ^ {2}.
$$
In the OBO framework of online meta-learning, the meta-model  $\theta_t$  is the outer-level decision variable at round  $t$ , and the task model  $\phi_t$  is the inner-level decision variable. At round  $t$ , the task model  $\phi_{t+1}$  is first obtained based on  $g_t(\theta_t, \phi)$  as an estimation of  $\phi_t^*(\theta_t)$ ; and then given  $\phi_{t+1}$ , the meta-model will be further updated w.r.t.
$$
f _ {t} (\theta , \phi_ {t + 1}) = \mathcal {L} _ {t} \left(\phi_ {t + 1}, \mathcal {D} _ {t} ^ {t e}\right) + \lambda \| \theta - \phi_ {t + 1} \| ^ {2}.
$$
Online adversarial training Adversarial training [46, 60, 65] is usually formulated as a min-max optimization problem. The defender learns a robust model to minimize the worst-case training loss against an attacker, where the attacker aims to maximize the loss by perturbing the training data. A static setting is often considered with full access to the target dataset at all times. Nevertheless, many real-world applications involve streaming data that arrive in an online manner, e.g., the financial markets or real-time sensor networks. A continuously robust model update is more desirable in these applications against potential attacks on the streaming data.
This online adversarial training problem can also be addressed by the OBO framework. Here the model parameters  $\theta$  is the outer-level decision variable and the adversarial perturbations  $\delta$  to the data point  $(x,y)$  is the inner-level decision variable. At each time  $t$ , the defender updates the estimate  $\delta_{t+1}$  of the worst adversarial perturbations, given her knowledge about the inner-level objection function  $g_t(\theta_t, \delta_t) = -\mathcal{L}(\theta_t, x_t + \delta, y_t)$  subject to the perturbation constraint. Based
on  $\delta_{t + 1}$ , the defender updates the model  $\theta_{t + 1}$  robustly w.r.t. the outer-level objective function  $f_{t}(\theta ,\delta_{t + 1}) = \mathcal{L}(\theta ,x_{t} + \delta_{t + 1},y_{t})$
We use a grid of values between  $10^{-5}$  and 10 to set the step sizes and did not find the algorithm particularly sensitive to them for the experiments considered. For example, we achieve best performance by setting both the inner and outer step sizes to  $10^{-4}$  for the online hyper-representation learning experiments and small values around that scale yield the same performance. For the dynamic OHO experiments, only the outer step size is set manually to 0.01. The inner step size is optimized along with the other regularization hyperparameters.
OAGD needs to store the previous objective functions, which requires all the previous data points to compute the function values and additional resources to store the knowledge of the function structures. In contrast, our method only stores the previous hypergradient estimates averaged over previous data points. For example, when  $n$  data points are used to evaluate the function at each round, the memory requirement for OAGD is  $O(nK)$ , compared to  $O(K)$  for our method. Here,  $K$  is the window size. Therefore, the memory cost of our method can be lower, especially when the number of data points is large.
D Comparison between the regret definitions
For a clear comparison, we restate the definitions of bilevel local regret in our work and OAGD here: Our definition:
$$
B L R = \sum_ {t = 1} ^ {T} \left\| \frac {1}{W} \sum_ {i = 1} ^ {K - 1} \eta^ {i} \nabla f _ {t - i} (x _ {t - i}, y _ {t - i} ^ {*} (x _ {t - i})) \right\| ^ {2}.
$$
In OAGD:
$$
B L R = \sum_ {t = 1} ^ {T} \left\| \frac {1}{W} \sum_ {i = 1} ^ {K - 1} \eta^ {i} \nabla f _ {t - i} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) \right\| ^ {2}.
$$
The key difference is that we evaluate the past loss  $f_{t - i}$  using the variable updates  $x_{t - i}$  and  $y_{t - i}^{*}(x_{t - i})$  at exactly same time  $t - i$ , while in OAGD the past loss  $f_{t - i}$  is evaluated using the most recent updates  $x_{t}$  and  $y_{t}^{*}(x_{t})$ . As shown in [2], the static regret in OAGD can cause problems for time-varying loss functions. Intuitively, evaluating the objective at time slot  $i$  using variable updates at different time slot  $j$  can be misleading, because it does not properly characterize the online learning performance of the model update at time slot  $i$ , especially when the objective functions vary a lot.
In Figure 2 we compare our algorithm and OAGD using the regret notion proposed in OAGD. The results show that our algorithm still achieves similar regret performance compared to OAGD, but with a much shorter runtime.
Using path-length regularization to capture the variation of optimal decision variables is very common in the literature of dynamic online learning, e.g., [68, 48, 61, 64, 66]. Note that because this variation of optimal decision variables is not controllable, we do not use this term in the design of the algorithm. Rather, the variation term is only used in the theoretical analysis to understand which factors in the system lead to a tighter bound on the regret. However, we mention here that it is also possible to explicitly analyze the regret in terms of the function variations directly.
Theorem E.1. Suppose that Assumptions 5.1-5.4 hold. Let  $V_{g} = \sum_{t=1}^{T} \sup |g_{t+1}(x,y) - g_{t}(x,y)|$  and  $V_{f} = \sum_{t=1}^{T} \sup [f_{t+1}(x,y) - f_{t}(x,y)]$ . Under the same conditions on  $\lambda, \alpha, Q_{t}, \eta$  and  $\beta$  with Theorem 5.7, we can have  $BLR_{w}(T) \leq O\left(\frac{T}{\beta W} + \frac{V_{f}}{\beta} + V_{g} + \frac{\sqrt{TV_{g}}}{\beta}\right)$ .
The main proof idea is as follows:
Figure 2: Comparison between our SOBOW and OAGD using the regret notion proposed in [59] on the dynamic online HR problem.
(1) For the term  $H_{2,T}$ , based on the strong convexity of  $g_t$ , we can show that  $\| y_{t+1}^*(x) - y_t^*(x) \|^2 \leq \frac{2}{\mu_g} \sup |g_{t+1}(x,y) - g_t(x,y)|$ ;
(2) For the term  $V_{1,T}$ , we can show that  $f_{t+1}(x, y_{t+1}^{*}(x)) - f_{t}(x, y_{t}^{*}(x)) \leq L_{0} \| y_{t+1}^{*}(x) - y_{t}^{*}(x) \| + \sup [f_{t+1}(x, y) - f_{t}(x, y)]$ , such that  $\sum_{t=1}^{T} [F_{t,\eta}(x_{t}, y_{t}^{*}(x_{t})) - F_{t,\eta}(x_{t+1}, y_{t}^{*}(x_{t+1}))]$  in Lemma H.3 can be upper bounded by  $\frac{2MT}{W} + L_{0} \sqrt{\frac{2T}{\mu_{g}}} \sqrt{\sum_{t=1}^{T} \sup |g_{t+1}(x, y) - g_{t}(x, y)|} + \sum_{t=1}^{T} \sup [f_{t+1}(x, y) - f_{t}(x, y)]$ ;
(3) Based on the above, if we denote  $V_{g} = \sum_{t=1}^{T} \sup |g_{t+1}(x,y) - g_{t}(x,y)|$  and  $V_{f} = \sum_{t=1}^{T} \sup [f_{t+1}(x,y) - f_{t}(x,y)]$  to capture the function variations, we can have the overall regret as  $O\left(\frac{T}{\beta W} + \frac{V_{f}}{\beta} + V_{g} + \frac{\sqrt{T V_{g}}}{\beta}\right)$ . In this case, a sublinear regret will be achieved if both  $V_{g}$  and  $V_{f}$  are  $o(T)$  for suitably selected  $W$ . As mentioned in our previous response, the condition on the variation of  $y_{t}^{*}(x)$  is weaker compared to the condition on the variation of  $g_{t}$  in order to achieve a small regret. For example, suppose  $W = \omega(T)$  and the function variation of  $f_{t}$  is very small, to achieve a regret of  $O(T^{3/4})$ ,  $V_{1,T} = O(T^{3/4})$  is sufficient, while we need a stricter condition on the function variation of  $g_{t}$ , i.e.,  $V_{g} = O(T^{1/2})$ .
F Proof of Lemma 5.5
To prove Lemma 5.5, we first have the following lemma about  $v_{t}^{*}$ :
Lemma F.1. Suppose that Assumption 5.2, Assumption 5.3 and Assumption 5.4 hold. We can have the following upper bound on  $\| v_t^*\|$ :
$$
\left\| v _ {t} ^ {*} \right\| \leq \frac {\rho \mu_ {g} + D L _ {1} ^ {2} + D L _ {1} \mu_ {g}}{\mu_ {g} ^ {2}} \triangleq M _ {v}.
$$
Proof. Based on Lemma 2.2 in [14], it can be shown that  $y_{t}^{*}(x)$  is  $\frac{L_1}{\mu_g}$ -Lipschitz continuous in  $x$ , i.e.,
$$
\left\| y _ {t} ^ {*} (x) - y _ {t} ^ {*} \left(x ^ {\prime}\right) \right\| \leq \frac {L _ {1}}{\mu_ {g}} \| x - x ^ {\prime} \| \tag {8}
$$
for any  $x$  and  $x'$ . According to Assumption 5.4, let  $\hat{x} \in \mathcal{X}$  such that  $\| \nabla_y f_t(\hat{x}, y_t^*(\hat{x})) \| \leq \rho$ . Then it follows that
$$
\begin{array}{l} \left\| \nabla_ {y} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) \right\| = \left\| \nabla_ {y} f _ {t} \left(\hat {x}, y _ {t} ^ {*} (\hat {x})\right) + \nabla_ {y} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \nabla_ {y} f _ {t} \left(\hat {x}, y _ {t} ^ {*} (\hat {x})\right) \right\| \\ \leq \left\| \nabla_ {y} f _ {t} \left(\hat {x}, y _ {t} ^ {*} (\hat {x})\right) \right\| + \left\| \nabla_ {y} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \nabla_ {y} f _ {t} \left(\hat {x}, y _ {t} ^ {*} (\hat {x})\right) \right\| \\ \leq \rho + \left(L _ {1} + \frac {L _ {1} ^ {2}}{\mu_ {g}}\right) \| x _ {t} - \hat {x} \| \\ \leq \frac {\rho \mu_ {g} + D L _ {1} ^ {2} + D L _ {1} \mu_ {g}}{\mu_ {g}}. \\ \end{array}
$$
$$
\begin{array}{l} \left\| v _ {t} ^ {*} \right\| = \left\| \left(\nabla_ {y} ^ {2} g _ {t} \left(x _ {t}, y _ {t} ^ {*} (x _ {t})\right)\right) ^ {- 1} \nabla_ {y} f _ {t} \left(x _ {t}, y _ {t} ^ {*} (x _ {t})\right) \right\| \\ \leq \| \left(\nabla_ {y} ^ {2} g _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right)\right) ^ {- 1} \| \| \nabla_ {y} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) \| \\ \leq \frac {1}{\mu_ {g}} \| \nabla_ {y} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) \| \\ \leq \frac {\rho \mu_ {g} + D L _ {1} ^ {2} + D L _ {1} \mu_ {g}}{\mu_ {g} ^ {2}}. \\ \end{array}
$$
Based on Lemma 1 in [30], we can have that
$$
\begin{array}{l} \left\| v _ {t} ^ {Q} - v _ {t} ^ {*} \right\| \\ \leq \left[ Q _ {t} \left(1 - \lambda \mu_ {g}\right) ^ {Q _ {t} - 1} L _ {2} \lambda M _ {v} + \frac {1 - \left(1 - \lambda \mu_ {g}\right) ^ {Q _ {t}} \left(1 + \lambda Q _ {t} \mu_ {g}\right)}{\mu_ {g}} L _ {2} M _ {v} \right] \| y _ {t} ^ {*} - y _ {t + 1} \| \\ + \left(1 - \left(1 - \lambda \mu_ {g}\right) ^ {Q _ {t}}\right) \frac {L _ {1}}{\mu_ {g}} \left\| y _ {t} ^ {*} - y _ {t + 1} \right\| + \left(1 - \lambda \mu_ {g}\right) ^ {Q _ {t}} \left\| v _ {t} ^ {0} - v _ {t} ^ {*} \right\| \\ \leq \left[ \frac {1}{1 - \lambda \mu_ {g}} \frac {Q _ {t}}{1 + Q _ {t} \log \frac {1}{1 - \lambda \mu_ {g}}} L _ {2} \lambda M _ {v} + \frac {1}{\mu_ {g}} L _ {2} M _ {v} + \frac {L _ {1}}{\mu_ {g}} \right] \| y _ {t} ^ {*} - y _ {t + 1} \| + (1 - \lambda \mu_ {g}) ^ {Q _ {t}} \| v _ {t} ^ {0} - v _ {t} ^ {*} \| \\ = \left(\frac {L _ {2} \lambda M _ {v}}{\left(1 - \lambda \mu_ {g}\right) \log \frac {1}{1 - \lambda \mu_ {g}}} + \frac {L _ {2} M _ {v} + L _ {1}}{\mu_ {g}}\right) \| y _ {t} ^ {*} - y _ {t + 1} \| + \left(1 - \lambda \mu_ {g}\right) ^ {Q _ {t}} \| v _ {t} ^ {0} - v _ {t} ^ {*} \| \\ \end{array}
$$
$$
\triangleq C _ {Q} \left\| y _ {t} ^ {*} - y _ {t + 1} \right\| + \left(1 - \lambda \mu_ {g}\right) ^ {Q _ {t}} \left\| v ^ {0} - v _ {t} ^ {*} \right\|
$$
where  $C_Q = \frac{L_2\lambda M_v}{(1 - \lambda\mu_g)\log\frac{1}{1 - \lambda\mu_g}} + \frac{L_2M_v + L_1}{\mu_g}$ . By using the Young's inequality and Lemma F.1, it follows that
$$
\begin{array}{l} \left\| v _ {t} ^ {Q} - v _ {t} ^ {*} \right\| ^ {2} \leq \left(1 + \frac {1}{\lambda \mu_ {g}}\right) C _ {Q} ^ {2} \left\| y _ {t} ^ {*} - y _ {t + 1} \right\| ^ {2} + (1 + \lambda \mu_ {g}) (1 - \lambda \mu_ {g}) ^ {2 Q _ {t}} \left\| v ^ {0} - v _ {t} ^ {*} \right\| ^ {2} \\ \leq \left(1 + \frac {1}{\lambda \mu_ {g}}\right) C _ {Q} ^ {2} \left\| y _ {t} ^ {*} - y _ {t + 1} \right\| ^ {2} + (1 + \lambda \mu_ {g}) \left(1 - \lambda \mu_ {g}\right) ^ {2 Q _ {t}} \left(2 \| v ^ {0} \| ^ {2} + 2 M _ {v}\right) \\ \triangleq c _ {2} \left\| y _ {t} ^ {*} - y _ {t + 1} \right\| ^ {2} + \epsilon_ {t} ^ {2} \\ \end{array}
$$
$$
\text {w h e r e} c _ {2} = \left(1 + \frac {1}{\lambda \mu_ {g}}\right) C _ {Q} ^ {2}, \text {a n d} \epsilon_ {t} ^ {2} = (1 + \lambda \mu_ {g}) (1 - \lambda \mu_ {g}) ^ {2 Q _ {t}} (2 \| v ^ {0} \| ^ {2} + 2 M _ {v}).
$$
G Proof of Theorem 5.6
Based on Equation (3) and Equation (4), it follows that
$$
\begin{array}{l} \left\| \nabla f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \widehat {\nabla} f _ {t} \left(x _ {t}, y _ {t + 1}\right) \right\| ^ {2} \\ = \| \nabla_ {x} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) v _ {t} ^ {*} - \nabla_ {x} f _ {t} \left(x _ {t}, y _ {t + 1}\right) + \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t + 1}\right) v _ {t} ^ {Q} \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} = \left\| \nabla_ {x} f _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \nabla_ {x} f _ {t} \left(x _ {t}, y _ {t + 1}\right) + \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t + 1}\right) v _ {t} ^ {Q} - \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t + 1}\right) v _ {t} ^ {*} \right. \\ + \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t + 1}\right) v _ {t} ^ {*} - \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) v _ {t} ^ {*} \| ^ {2} \\ \leq 3 \| \nabla_ {x} f _ {t} (x _ {t}, y _ {t + 1}) - \nabla_ {x} f _ {t} (x _ {t}, y _ {t} ^ {*} (x _ {t})) \| ^ {2} + 3 \| \nabla_ {x} \nabla_ {y} g _ {t} (x _ {t}, y _ {t + 1}) \| ^ {2} \| v _ {t} ^ {*} - v _ {t} ^ {Q} \| ^ {2} \\ + 3 \| \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t} ^ {*} \left(x _ {t}\right)\right) - \nabla_ {x} \nabla_ {y} g _ {t} \left(x _ {t}, y _ {t + 1}\right) \| ^ {2} \| v _ {t} ^ {*} \| ^ {2} \\ \stackrel {(a)} {\leq} 3 L _ {1} ^ {2} \| y _ {t + 1} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} + 3 L _ {1} ^ {2} \| v _ {t} ^ {*} - v _ {t} ^ {Q} \| ^ {2} + 3 L _ {2} ^ {2} \| y _ {t + 1} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} \| v _ {t} ^ {*} \| ^ {2} \\ \stackrel {(b)} {\leq} \left(3 L _ {1} ^ {2} + \frac {3 L _ {2} ^ {2} (\rho \mu_ {g} + D L _ {1} ^ {2} + D L _ {1} \mu_ {g}) ^ {2}}{\mu_ {g} ^ {4}}\right) \| y _ {t + 1} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} + 3 L _ {1} ^ {2} \| v _ {t} ^ {*} - v _ {t} ^ {Q} \| ^ {2} \\ \stackrel {(c)} {\leq} \left(3 L _ {1} ^ {2} (1 + c _ {2}) + \frac {3 L _ {2} ^ {2} (\rho \mu_ {g} + D L _ {1} ^ {2} + D L _ {1} \mu_ {g}) ^ {2}}{\mu_ {g} ^ {4}}\right) \| y _ {t + 1} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} + 3 L _ {1} ^ {2} \epsilon_ {t} ^ {2} \\ = 3 L _ {1} ^ {2} \left[ \left(1 + c _ {2} + \frac {L _ {2} ^ {2} \left(\rho \mu_ {g} + D L _ {1} ^ {2} + D L _ {1} \mu_ {g}\right) ^ {2}}{L _ {1} ^ {2} \mu_ {g} ^ {4}}\right) \| y _ {t + 1} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} + \epsilon_ {t} ^ {2} \right] \tag {9} \\ \end{array}
$$
where (a) is true because of Assumption 5.2, (b) is true because of Lemma F.1 and (c) holds due to Lemma 5.5.
Next, based on Assumption 5.1 and Assumption 5.2, for any  $t$  and  $\alpha < \frac{1}{L_1}$  we can have
$$
\begin{array}{l} \left\| y _ {t + 1} - y _ {t} ^ {*} \left(x _ {t}\right) \right\| ^ {2} \\ \leq \left(1 - \alpha \mu_ {g}\right) \| y _ {t} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} \\ = (1 - \alpha \mu_ {g}) \| y _ {t} - y _ {t - 1} ^ {*} (x _ {t - 1}) + y _ {t - 1} ^ {*} (x _ {t - 1}) - y _ {t} ^ {*} (x _ {t}) \| ^ {2} \\ \stackrel {(a)} {\leq} (1 + \lambda) (1 - \alpha \mu_ {g}) \| y _ {t} - y _ {t - 1} ^ {*} (x _ {t - 1}) \| ^ {2} + (1 + \frac {1}{\lambda}) (1 - \alpha \mu_ {g}) \| y _ {t - 1} ^ {*} (x _ {t - 1}) - y _ {t} ^ {*} (x _ {t}) \| ^ {2} \\ \leq (1 + \lambda) (1 - \alpha \mu_ {g}) \| y _ {t} - y _ {t - 1} ^ {*} (x _ {t - 1}) \| ^ {2} + 2 (1 + \frac {1}{\lambda}) (1 - \alpha \mu_ {g}) \| y _ {t - 1} ^ {*} (x _ {t - 1}) - y _ {t} ^ {*} (x _ {t - 1}) \| ^ {2} \\ + 2 \left(1 + \frac {1}{\lambda}\right) \left(1 - \alpha \mu_ {g}\right) \left\| y _ {t} ^ {*} \left(x _ {t - 1}\right) - y _ {t} ^ {*} \left(x _ {t}\right) \right\| ^ {2} \\ \stackrel {(b)} {\leq} (1 + \lambda) (1 - \alpha \mu_ {g}) \| y _ {t} - y _ {t - 1} ^ {*} (x _ {t - 1}) \| ^ {2} + 2 (1 + \frac {1}{\lambda}) (1 - \alpha \mu_ {g}) \| y _ {t - 1} ^ {*} (x _ {t - 1}) - y _ {t} ^ {*} (x _ {t - 1}) \| ^ {2} \\ + \frac {2 L _ {1} ^ {2}}{\mu_ {g} ^ {2}} (1 + \frac {1}{\lambda}) (1 - \alpha \mu_ {g}) \| x _ {t - 1} - x _ {t} \| ^ {2}, \\ \end{array}
$$
where (a) is based on the Young's inequality and (b) is due to Equation (8).
For  $\lambda = \frac{\alpha\mu_g}{2}$ , it follows that
$$
\begin{array}{l} (1 + \lambda) (1 - \alpha \mu_ {g}) = (1 + \frac {\alpha \mu_ {g}}{2}) (1 - \alpha \mu_ {g}) \\ = 1 - \frac {\alpha \mu_ {g}}{2} - \frac {\alpha^ {2} \mu_ {g} ^ {2}}{2} \\ <   1 - \frac {\alpha \mu_ {g}}{2}. \\ \end{array}
$$
Let  $G_{1} = 1 + c_{2} + \frac{L_{2}^{2}(\rho\mu_{g} + DL_{1}^{2} + DL_{1}\mu_{g})^{2}}{L_{1}^{2}\mu_{g}^{4}}$  and  $\delta_t = G_1\| y_{t+1} - y_t^*(x_t)\|^2 + \epsilon_t^2$ . Based on Lemma 5.5, we can have that
$$
\begin{array}{l} \delta_ {t} = G _ {1} \| y _ {t + 1} - y _ {t} ^ {*} (x _ {t}) \| ^ {2} + \epsilon_ {t} ^ {2} \\ \leq G _ {1} (1 + \lambda) \left(1 - \alpha \mu_ {g}\right) \left\| y _ {t} - y _ {t - 1} ^ {*} \left(x _ {t - 1}\right) \right\| ^ {2} + 2 G _ {1} \left(1 + \frac {1}{\lambda}\right) \left(1 - \alpha \mu_ {g}\right) \left\| y _ {t - 1} ^ {*} \left(x _ {t - 1}\right) - y _ {t} ^ {*} \left(x _ {t - 1}\right) \right\| ^ {2} \\ + \frac {2 L _ {1} ^ {2} G _ {1}}{\mu_ {g} ^ {2}} (1 + \frac {1}{\lambda}) (1 - \alpha \mu_ {g}) \| x _ {t - 1} - x _ {t} \| ^ {2} + \epsilon_ {t} ^ {2} \\ \leq G _ {1} \left(1 - \frac {\alpha \mu_ {g}}{2}\right) \left\| y _ {t} - y _ {t - 1} ^ {*} (x _ {t - 1}) \right\| ^ {2} + G _ {2} \left\| y _ {t - 1} ^ {*} (x _ {t - 1}) - y _ {t} ^ {*} (x _ {t - 1}) \right\| ^ {2} \\ \end{array}
$$