SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu-Hu Yan, Peng Zhao, Zhi-Hua Zhou
Nanjing University, Nanjing 210023, China
{yanyh, zhaop, zhouzh}@lamda.nju.edu.cn
In this paper, we propose an online convex optimization approach with two different levels of adaptivity. On a higher level, our approach is agnostic to the unknown types and curvatures of the online functions, while at a lower level, it can exploit the unknown niceness of the environments and attain problem-dependent guarantees. Specifically, we obtain  $\mathcal{O}(\log V_T)$ ,  $\mathcal{O}(d\log V_T)$  and  $\hat{\mathcal{O}} (\sqrt{V_T})$  regret bounds for strongly convex, exp-concave and convex loss functions, respectively, where  $d$  is the dimension,  $V_{T}$  denotes problem-dependent gradient variations and the  $\hat{\mathcal{O}} (\cdot)$ -notation omits log  $V_{T}$  factors. Our result not only safeguards the worst-case guarantees but also directly implies the small-loss bounds in analysis. Moreover, when applied to adversarial/stochastic convex optimization and game theory problems, our result enhances the existing universal guarantees. Our approach is based on a multi-layer online ensemble framework incorporating novel ingredients, including a carefully designed optimism for unifying diverse function types and cascaded corrections for algorithmic stability. Notably, despite its multi-layer structure, our algorithm necessitates only one gradient query per round, making it favorable when the gradient evaluation is time-consuming. This is facilitated by a novel regret decomposition equipped with carefully designed surrogate losses.
Online convex optimization (OCO) is a versatile model that depicts the interaction between a learner and the environments over time [Hazan, 2016, Orabona, 2019]. In each round  $t \in [T]$ , the learner selects a decision  $\mathbf{x}_t$  from a convex compact set  $\mathcal{X} \subseteq \mathbb{R}^d$ , and simultaneously the environments choose a convex loss function  $f_t: \mathcal{X} \mapsto \mathbb{R}$ . Subsequently, the learner incurs a loss  $f_t(\mathbf{x}_t)$ , obtains information about the online function, and updates the decision to  $\mathbf{x}_{t+1}$ , aiming to optimize the game-theoretical performance measure known as regret [Cesa-Bianchi and Lugosi, 2006]:
$$
\operatorname {R E G} _ {T} \triangleq \sum_ {t = 1} ^ {T} f _ {t} \left(\mathbf {x} _ {t}\right) - \min  _ {\mathbf {x} \in \mathcal {X}} \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x}), \tag {1.1}
$$
which represents the learner's excess loss compared to the best fixed comparator in hindsight.
In OCO, the type and curvature of online functions significantly impact the minimax regret bounds. Specifically, for convex functions, online gradient descent (OGD) can achieve an  $\mathcal{O}(\sqrt{T})$  regret guarantee [Zinkevich, 2003]. For  $\alpha$ -exp-concave functions, online Newton step (ONS), with prior knowledge of the curvature coefficient  $\alpha$ , attains an  $\mathcal{O}(d\log T)$  regret [Hazan et al., 2007]. For  $\lambda$ -strongly convex functions, OGD with prior knowledge of the curvature coefficient  $\lambda$  and a different parameter configuration enjoys an  $\mathcal{O}(\log T)$  regret [Hazan et al., 2007]. Note that the above
*Correspondence: Peng Zhao <zhaop@lamda.nju.edu.cn>
Table 1: Comparison with existing results. The second column presents the regret bounds for various kinds of functions, where  $V_{T}$  and  $F_{T}$  are problem-dependent quantities that are at most  $\mathcal{O}(T)$  and can be much smaller in nice environments. The  $\hat{\mathcal{O}}(\cdot)$ -notation omits logarithmic factors on  $V_{T}$  and  $F_{T}$ . The third column shows the gradient query complexity. All problem-dependent bounds require the smoothness of online functions.
Works	Regret Bounds	Gradient Query
Strongly Convex	Exp-concave	Convex
van Erven and Koolen [2016]	O(d log T)	O(d log T)	O(√T)	1
Wang et al. [2019]	O(log T)	O(d log T)	O(√T)	1
Zhang et al. [2022a]	O(min{log VT, log FT})	O(d min{log VT, log FT})	O(√FT)	O(log T)
Ours	O(min{log VT, log FT})	O(d min{log VT, log FT})	O(min{√VT,√FT})	1
algorithms require the function type and curvature beforehand and does not consider the niceness of environments. Recent studies further strengthen the algorithms and results with two levels of adaptivity. The higher-level adaptivity requires an algorithm to be agnostic to the unknown types and curvatures of the online functions. And the lower-level adaptivity requires an algorithm to exploit the unknown niceness of the environments within a specific function family. In the following, we delve into an extensive discussion on these two levels of adaptivity.
1.1 High Level: Adaptive to Unknown Curvature of Online Functions
Traditionally, the learner needs to know the function type and curvature in advance to select suitable algorithms (and parameter configurations), which can be burdensome in practice. Universal online learning [van Erven and Koolen, 2016, Wang et al., 2019, Zhang et al., 2022a] aims to develop a single algorithm agnostic to the specific function type and curvature while achieving the same regret guarantees as if they were known. The pioneering work of van Erven and Koolen [2016] proposed a single algorithm called MetaGrad that achieves an  $\mathcal{O}(\sqrt{T})$  regret for convex functions and an  $\mathcal{O}(d\log T)$  regret for exp-concave functions. Later, Wang et al. [2019] further obtained the optimal  $\mathcal{O}(\log T)$  regret for strongly convex functions. Notably, these approaches are efficient regarding the gradient query complexity, by using only one gradient within each round.
However, the above approaches are not flexible enough since they have to optimize a group of heterogeneous and carefully-designed surrogate loss functions, which can be cumbersome and challenging. To this end, Zhang et al. [2022a] introduced a simple framework that operates on the original online functions with the same optimal results at the expense of  $\mathcal{O}(\log T)$  gradient queries.
1.2 Low Level: Adaptive to Unknown Niceness of Online Environments
Within a specific function family, the algorithm's performance is also substantially influenced by the niceness of environments. This concept is usually captured through problem-dependent quantities in the literature. Therefore, it becomes essential to develop adaptive algorithms with problem-dependent regret guarantees. Specifically, we consider the following problem-dependent quantities:
$$
F _ {T} \triangleq \min  _ {\mathbf {x} \in \mathcal {X}} \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x}), \text {a n d} V _ {T} \triangleq \sum_ {t = 2} ^ {T} \sup  _ {\mathbf {x} \in \mathcal {X}} \| \nabla f _ {t} (\mathbf {x}) - \nabla f _ {t - 1} (\mathbf {x}) \| ^ {2},
$$
where the small loss  $F_{T}$  represents the cumulative loss of the best comparator [Srebro et al., 2010, Orabona et al., 2012] and the gradient variation  $V_{T}$  characterizes the variation of the function gradients [Chiang et al., 2012]. In particular, the gradient-variation bound demonstrates its fundamental importance in modern online learning from the following three aspects: (i) it safeguards the worst-case guarantees in terms of  $T$  and implies the small-loss bounds in analysis directly; (ii) it draws a profound connection between adversarial and stochastic convex optimization; and (iii) it is crucial for fast convergence rates in game theory. We will explain the three aspects in detail in the next part.
1.3 Our Contributions and Techniques
In this paper, we consider the two levels of adaptivity simultaneously and propose a novel universal approach that achieves  $\mathcal{O}(\log V_T)$ ,  $\mathcal{O}(d\log V_T)$  and  $\hat{\mathcal{O}} (\sqrt{V_T})$  regret bounds for strongly convex,
exp-concave and convex loss functions, respectively, using only one gradient query per round, where  $\widehat{\mathcal{O}} (\cdot)$ -notation omits factors on  $\log V_{T}$ . Table 1 compares our results with existing ones. In summary, relying on the basic idea of online ensemble [Zhao et al., 2021], our approach primarily admits a multi-layer online ensemble structure with several important novel ingredients. Specifically, we propose a carefully designed optimism, a hyper-parameter encoding historical information, to handle different kinds of functions universally, particularly exp-concave functions. Nevertheless, it necessitates careful management of the stability of final decisions, which is complicated in the multi-layer structure. To this end, we analyze the negative stability terms in the algorithm and propose cascaded correction terms to realize effective collaboration among layers, thus enhancing the algorithmic stability. Moreover, we facilitate a novel regret decomposition equipped with carefully designed surrogate losses to achieve only one gradient query per round, making our algorithm as efficient as van Erven and Koolen [2016] regarding the gradient complexity. Our result resolves an open problem proposed by Zhang et al. [2022a], who have obtained partial results for exp-concave and strongly convex functions and asked whether it is possible to design designing a single algorithm with universal gradient-variation bounds. Among them, the convex case is particularly important because the improvement from  $T$  to  $V_{T}$  is polynomial, whereas logarithmic in the other cases.
Next, we shed light on some applications of our approach. First, it safeguards the worst-case guarantees [van Erven and Koolen, 2016, Wang et al., 2019] and directly implies the small-loss bounds of Zhang et al. [2022a] in analysis. Second, gradient variation is shown to play an essential role in the stochastically extended adversarial (SEA) model [Sachs et al., 2022, Chen et al., 2023b], an interpolation between stochastic and adversarial OCO. Our approach resolves a major open problem left in Chen et al. [2023b] on whether it is possible to develop a single algorithm with universal guarantees for strongly convex, exp-concave, and convex functions in the SEA model. Third, in game theory, gradient variation encodes the changes in other players' actions and can thus lead to fast convergence rates [Rakhlin and Sridharan, 2013b, Syrgkanis et al., 2015, Zhang et al., 2022b]. We demonstrate the universality of our approach by taking two-player zero-sum games as an example.
Technical Contributions. Our first contribution is proposing a multi-layer online ensemble approach with effective collaboration among layers, which is achieved by a carefully-designed optimism to unify different kinds of functions and cascaded correction terms to improve the algorithmic stability within the multi-layer structure. The second contribution arises from efficiency. Although there are multiple layers, our algorithm only requires one gradient query per round, which is achieved by a novel regret decomposition equipped with carefully designed surrogate losses. Two interesting byproducts rises in our approach. The first one is the negative stability term in the analysis of MSMwC [Chen et al., 2021], which serves as an important building block of our algorithm. And the second byproduct contains a simple approach and analysis for the optimal worst-case universal guarantees, using one gradient query within each round.
Organization. The rest of the paper is structured as follows. Section 2 provides preliminaries. Section 3 proposes our multi-layer online ensemble approach for universal gradient-variation bounds. Section 4 further improves the gradient query complexity. Due to page limits, the applications of our proposed algorithm are deferred to Appendix A. All the proofs can be found in the appendices.
In this section, we introduce some preliminary knowledge, including our assumptions, the definitions, the formal problem setup, and a review of the latest progress of Zhang et al. [2022a].
To begin with, we list some notations. Specifically, we use  $\| \cdot \|$  for  $\| \cdot \|_2$  in default and use  $\sum_{t}, \sum_{k}, \sum_{i}$  as abbreviations for  $\sum_{t \in [T]}, \sum_{k \in [K]}$  and  $\sum_{i \in [N]}$ .  $a \lesssim b$  represents  $a \leq \mathcal{O}(b)$ .  $\widehat{\mathcal{O}}(\cdot)$ -notation omits logarithmic factors on leading terms. For example,  $\widehat{\mathcal{O}}(\sqrt{V})$  omits the dependence of  $\log V$ .
Assumption 1 (Boundedness). For any  $\mathbf{x},\mathbf{y}\in \mathcal{X}$  and  $t\in [T]$ , the domain diameter satisfies  $\| \mathbf{x} - \mathbf{y}\| \leq D$ , and the gradient norm of the online functions is bounded by  $\| \nabla f_t(\mathbf{x})\| \leq G$ .
Assumption 2 (Smoothness). All online functions are  $L$ -smooth:  $\| \nabla f_t(\mathbf{x}) - \nabla f_t(\mathbf{y}) \| \leq L \| \mathbf{x} - \mathbf{y} \|$  for any  $\mathbf{x}, \mathbf{y} \in \mathcal{X}$  and  $t \in [T]$ .
Both assumptions are common in the literature. Specifically, the boundedness assumption is common in OCO [Hazan, 2016]. The smoothness assumption is essential for first-order algorithms to
achieve gradient-variation bounds [Chiang et al., 2012]. Strong convexity and exp-concavity are defined as follows. For any  $\mathbf{x},\mathbf{y}\in \mathcal{X}$ , a function  $f$  is  $\lambda$ -strongly convex if  $f(\mathbf{x}) - f(\mathbf{y})\leq \langle \nabla f(\mathbf{x}),\mathbf{x} - \mathbf{y}\rangle -\frac{\lambda}{2}\| \mathbf{x} - \mathbf{y}\|^{2}$ , and is  $\alpha$ -exp-concave if  $f(\mathbf{x}) - f(\mathbf{y})\leq \langle \nabla f(\mathbf{x}),\mathbf{x} - \mathbf{y}\rangle -\frac{\alpha}{2}\langle \nabla f(\mathbf{x}),\mathbf{x} - \mathbf{y}\rangle^{2}$ . Note that the formal definition of  $\beta$ -exp-concavity states that  $\exp (-\beta f(\cdot))$  is concave. Under Assumption 1,  $\beta$ -exp-concavity leads to our definition with  $\alpha = \frac{1}{2}\min \left\{\frac{1}{4GD},\beta \right\}$  [Hazan, 2016, Lemma 4.3]. For simplicity, we use it as an alternative definition for exp-concavity.
In the following, we formally describe the problem setup and briefly review the key insight of Zhang et al. [2022a]. Concretely, we consider the problem where the learner has no prior knowledge about the function type (strongly convex, exp-concave, or convex) or the curvature coefficient  $(\alpha$  or  $\lambda)$ . Without loss of generality, we study the case where the curvature coefficients  $\alpha, \lambda \in [1/T, 1]$ . This requirement is natural because if  $\alpha$  (or  $\lambda$ )  $< 1/T$ , even the optimal regret is  $\Omega(T)$  [Hazan et al., 2007], which is vacuous. Conversely, functions with  $\alpha$  (or  $\lambda$ )  $> 1$  are also 1-exp-concave (or 1-strongly convex). Thus using  $\alpha$  (or  $\lambda$ )  $= 1$  will only worsen the regret by a constant factor, which can be omitted. This condition is also used in previous works [Zhang et al., 2022a].
A Brief Review of Zhang et al. [2022a]. A general solution to handle the uncertainty is to leverage a two-layer framework, which consists of a group of base learners exploring the environments and a meta learner tracking the best base learner on the fly. To handle the unknown curvature coefficients  $\alpha$  and  $\lambda$ , the authors discretize them into the following candidate pool:
$$
\mathcal {H} \triangleq \left\{1 / T, ^ {2} / T, ^ {4} / T, \dots , 1 \right\}, \tag {2.1}
$$
where  $|\mathcal{H}| \approx \log T$ . Consequently, they design three groups of base learners:
(i) about  $\log T$  base learners, each of which runs the algorithm for strongly convex functions with a guess  $\lambda_{i}\in \mathcal{H}$  of the strong convexity coefficient  $\lambda$ ;
(ii) about  $\log T$  base learners, each of which runs the algorithm for exp-concave functions with a guess  $\alpha_{i}\in \mathcal{H}$  of the exp-concavity coefficient  $\alpha$ ;
(iii) 1 base learner that runs the algorithm for convex functions.
Overall, they maintain  $N \approx \log T + \log T + 1$  base learners. Denoting by  $\pmb{p}_t \triangleq (p_{t,1},\dots,p_{t,N})$  the meta learner's weights and  $\mathbf{x}_{t,i}$  the  $i$ -th base learner's decision, the learner submits  $\mathbf{x}_t = \sum_i p_{t,i} \mathbf{x}_{t,i}$ .
In the two-layer framework, the regret (1.1) can be decomposed into two terms:
$$
\operatorname {R E G} _ {T} = \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(\mathbf {x} _ {t}\right) - \sum_ {t = 1} ^ {T} f _ {t} \left(\mathbf {x} _ {t, i ^ {\star}}\right) \right] + \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(\mathbf {x} _ {t, i ^ {\star}}\right) - \min  _ {\mathbf {x} \in \mathcal {X}} \sum_ {t = 1} ^ {T} f _ {t} (\mathbf {x}) \right], \tag {2.2}
$$
where the meta regret (first term) assesses how well the algorithm tracks the best base learner, and the base regret (second term) measures the performance of it. The best base learner is the one which runs the algorithm matching the ground-truth function type with the most accurate guess of the curvature — taking  $\alpha$ -exp-concave functions as an example, there must exist a base learner indexed by  $i^{\star}$ , whose coefficient  $\alpha_{i^{\star}} \in \mathcal{H}$  satisfies  $\alpha_{i^{\star}} \leq \alpha \leq 2\alpha_{i^{\star}}$ .
A direct benefit of the above decomposition is that the meta regret can be bounded by a constant  $\mathcal{O}(1)$  for exp-concave and strongly convex functions, allowing the algorithm to perfectly inherit the gradient-variation bound from the base learner. Taking  $\alpha$ -exp-concave functions as an example, by definition, the meta regret can be bounded by  $\sum_{t}r_{t,i^{\star}} - \frac{\alpha}{2}\sum_{t}r_{t,i^{\star}}^{2}$ , where the first term  $r_{t,i}\triangleq \langle \nabla f_t(\mathbf{x}_t),\mathbf{x}_t - \mathbf{x}_{t,i}\rangle$  denotes the linearized regret, and the second one is a negative term from exp-concavity. Choosing ADAPT-ML-PROD [Gaillard et al., 2014] as the meta algorithm bounds the first term  $\sum_{t}r_{t,i^{\star}}$  by  $\mathcal{O}(\sqrt{\sum_{t}(r_{t,i^{\star}})^{2}})$ , which can be canceled by the negative term, leading to an  $\mathcal{O}(1)$  meta regret. Due to the meta algorithm's benefits, their approach can inherit the gradient-variation guarantees from the base learner. Similar derivation also applies to strongly convex functions. However, their approach is not favorable enough in the convex case and is not efficient enough in terms of the gradient query complexity. We will give more discussions about the above issues in Section 3.1 and Section 4.
This section presents our multi-layer online ensemble approach with universal gradient-variation bounds. Specifically, in Section 3.1, we provide a novel optimism to unify different kinds of functions. In Section 3.2, we exploit two types of negative terms to cancel the positive term caused by
the optimism design. We summarize the overall algorithm in Section 3.3. Finally, in Section 3.4, we present the main results and list several applications of our approach.
3.1 Universal Optimism Design
In this part, we propose a novel optimism that simultaneously unifies various kinds of functions. We start by observing that Zhang et al. [2022a] does not enjoy gradient-variation bounds for convex functions, where the main challenge lies in obtaining an  $\mathcal{O}(\sqrt{V_T})$  meta regret for convex functions while simultaneously maintaining an  $\mathcal{O}(1)$  regret for exp-concave and strongly convex functions. In the following, we focus on the meta regret because the base regret optimization is straightforward by employing the optimistic online learning technique [Rakhlin and Sridharan, 2013a,b] in a black-box fashion. Optimistic online learning is essential in our problem since it can utilize the historical information, e.g.,  $\nabla f_{t - 1}(\cdot)$  for our purpose due to the definition of the gradient variation.
Shifting our focus to the meta regret, we consider upper-bounding it by a second-order bound of  $\mathcal{O}(\sqrt{\sum_t(r_{t,i^{\star}} - m_{t,i^{\star}})^2})$ , where  $r_{t,i} = \langle \nabla f_t(\mathbf{x}_t),\mathbf{x}_t - \mathbf{x}_{t,i}\rangle$  and the optimism  $m_{t,i^{\star}}$  can encode historical information. Such a second-order bound can be easily obtained using existing prediction with expert advice algorithms, e.g., ADAPT-ML-PROD [Wei et al., 2016]. Nevertheless, as we will demonstrate in the following, designing an optimism  $m_{t,i}$  that effectively unifies various function types is not straightforward, thereby requiring novel ideas in the optimism design.
To begin with, a natural impulse is to choose the optimism as  $m_{t,i} = \langle \nabla f_{t - 1}(\mathbf{x}_{t - 1}),\mathbf{x}_{t,i} - \mathbf{x}_t\rangle ,^2$  which yields the following second-order bound:
$$
\sum_ {t = 1} ^ {T} \left(r _ {t, i ^ {\star}} - m _ {t, i ^ {\star}}\right) ^ {2} \lesssim \left\{ \begin{array}{l l} \sum_ {t} \left\| \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\| ^ {2}, & (s t r o n g l y c o n v e x) \\ \sum_ {t} \left\| \nabla f _ {t} (\mathbf {x} _ {t}) - \nabla f _ {t - 1} (\mathbf {x} _ {t - 1}) \right\| ^ {2}, & (c o n v e x) \end{array} \right.
$$
where the inequality is due to the boundedness assumption (Assumption 1). This optimism design handles the strongly convex functions well since the bound can be canceled by the negative term imported by strong convexity (i.e.,  $-\|\mathbf{x}_t - \mathbf{x}_{t,i^*}\|^2$ ). Moreover, it is quite promising for convex functions because the bound essentially consists of the desired gradient variation and a positive term of  $\|\mathbf{x}_t - \mathbf{x}_{t-1}\|^2$  (we will deal with it later). However, it fails for exp-concave functions because the negative term imported by exp-concavity (i.e.,  $-\langle\nabla f_t(\mathbf{x}_t), \mathbf{x}_t - \mathbf{x}_{t,i^*}\rangle^2$ ) cannot be used for cancellation due to the mismatch of the formulation.
To unify various kinds of functions, we propose a novel optimism design defined by  $m_{t,i} = r_{t - 1,i}$ . This design aims to secure a second-order bound of  $\mathcal{O}(\sqrt{\sum_t(r_{t,i^*} - r_{t - 1,i^*})^2})$ , which is sufficient to achieve an  $\mathcal{O}(1)$  meta regret for exp-concave functions (with strong convexity being a subcategory thereof) while maintaining an  $\mathcal{O}(\sqrt{V_T})$  meta regret for convex functions. The high-level intuition behind this approach is as follows: although the bound cannot be canceled exactly by the negative term imported by exp-concavity (i.e.,  $-r_{t,i^*}^2$ ) within each round, it becomes manageable when aggregated across the whole time horizon as  $\sum_t(r_{t,i^*} - r_{t - 1,i^*})^2 \lesssim 4\sum_t r_{t,i^*}^2$ , because  $r_{t,i^*}$  and  $r_{t - 1,i^*}$  differs by merely a single time step. In the following, we propose the key lemma of the universal optimism design and defer the proof to Appendix B.1.
Lemma 1 (Key Lemma). Under Assumptions 1 and 2, if the optimism is chosen as  $m_{t,i} = r_{t - 1,i} = \langle \nabla f_{t - 1}(\mathbf{x}_{t - 1}),\mathbf{x}_{t - 1} - \mathbf{x}_{t - 1,i}\rangle$ , it holds that
$$
\sum_ {t = 1} ^ {T} \left(r _ {t, i ^ {\star}} - m _ {t, i ^ {\star}}\right) ^ {2} \lesssim \left\{ \begin{array}{l l} \sum_ {t = 1} ^ {T} \left\langle \nabla f _ {t} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} _ {t, i ^ {\star}} \right\rangle^ {2}, & (e x p - c o n c a v e) \\ V _ {T} + \sum_ {t = 2} ^ {T} \left\| \mathbf {x} _ {t, i ^ {\star}} - \mathbf {x} _ {t - 1, i ^ {\star}} \right\| ^ {2} + \sum_ {t = 2} ^ {T} \left\| \mathbf {x} _ {t} - \mathbf {x} _ {t - 1} \right\| ^ {2}. & (c o n v e x) \end{array} \right.
$$
Moreover, the second part of Lemma 1 shows that the bound for convex functions is also controllable by being further decomposed into three terms. The first term is the desired gradient variation. The
2. Although  $\mathbf{x}_t$  is unknown when using  $m_{t,i}$ , we only need the scalar value of  $\langle \nabla f_{t-1}(\mathbf{x}_{t-1}), \mathbf{x}_t \rangle$ , which can be efficiently solved via a fixed-point problem of  $\langle \nabla f_{t-1}(\mathbf{x}_{t-1}), \mathbf{x}_t(z) \rangle = z$ .  $\mathbf{x}_t$  relies on  $z$  since  $m_{t,i}$  relies on  $z$  and  $\mathbf{x}_t$  relies on  $m_{t,i}$ . Interested readers can refer to Section 3.3 of Wei et al. [2016] for details.