SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lisha Chen†
Troy, NY, United States
chenl21@rpi.edu
Yiming Ying
Camperdown, Australia
yiming.ying@sydney.edu.au
Heshan Fernando†
fernah@rpi.edu
Multi-objective learning (MOL) often arises in emerging machine learning problems when multiple learning criteria or tasks need to be addressed. Recent works have developed various dynamic weighting algorithms for MOL, including MGDA and its variants, whose central idea is to find an update direction that avoids conflicts among objectives. Albeit its appealing intuition, empirical studies show that dynamic weighting methods may not always outperform static alternatives. To bridge this gap between theory and practice, we focus on a new variant of stochastic MGDA - the Multi-objective gradient with Double sampling (MoDo) algorithm and study its generalization performance and the interplay with optimization through the lens of algorithm stability. We find that the rationale behind MGDA - updating along conflict-avoidant direction - may impede dynamic weighting algorithms from achieving the optimal  $\mathcal{O}(1 / \sqrt{n})$  population risk, where  $n$  is the number of training samples. We further highlight the variability of dynamic weights and their impact on the three-way trade-off among optimization, generalization, and conflict avoidance that is unique in MOL. Code is available at https://github.com/heshandevaka/Trade-Off-MOL.
Multi-objective learning (MOL) emerges frequently in recent machine learning problems such as learning under fairness and safety constraints [49]; learning across multiple tasks, including multi-task learning [39] and meta-learning [47]; and, learning across multiple agents that may not share a global utility including federated learning [40] and multi-agent reinforcement learning [33].
This work considers solving the empirical version of MOL defined on the training dataset as  $S = \{z_1, \ldots, z_n\}$ . The performance of a model  $x \in \mathbb{R}^d$  on a datum  $z$  for the  $m$ -th objective is denoted as  $f_{z,m}: \mathbb{R}^d \mapsto \mathbb{R}$ , and its performance on the entire training dataset  $S$  is measured by the  $m$ -th empirical objective  $f_{S,m}(x)$  for  $m \in [M]$ . MOL optimizes the vector-valued objective, given by
$$
\min  _ {x \in \mathbb {R} ^ {d}} F _ {S} (x) := \left[ f _ {S, 1} (x), \dots , f _ {S, M} (x) \right]. \tag {1.1}
$$
The work of L. Chen, H. Fernando, and T. Chen was supported by the National Science Foundation (NSF) MoDL-SCALE project 2134168 and the RPI-IBM Artificial Intelligence Research Collaboration (AIRC). The work of Y. Ying was partially supported by NSF (DMS-2110836, IIS-2103450, and IIS-2110546).
(a) Objective 1
(b) Objective 2
(c) MGDA
Figure 1: An example from [29] with two objectives (1a and 1b) to show the three-way trade-off in MOL. Figures 1c-1e show the optimization trajectories, where the black  $\clubsuit$  marks initializations of the trajectories, colored from red (start) to yellow (end). The background solid/dotted contours display the landscape of the average empirical/population objectives. The gray/green bar marks empirical/population Pareto front, and the black  $\star$ /green  $\star$  marks solution to the average objectives.
(d) Static
(e) MoDo (dynamic)
One natural method for solving (1.1) is to optimize the (weighted) average of the multiple objectives, also known as static or unitary weighting [18, 45]. However, this method may face challenges due to potential conflicts among multiple objectives during the optimization process; e.g., conflicting gradient directions  $\langle \nabla f_{S,m}(x),\nabla f_{S,m'}(x)\rangle < 0$ . A popular alternative is thus to dynamically weight gradients from different objectives to avoid conflicts and obtain a direction  $d(x)$  that optimizes all objective functions jointly that we call a conflict-avoidant (CA) direction. Algorithms in this category include the multi-gradient descent algorithm (MGDA) [9], its stochastic variants [30, 10, 52]. While the idea of finding CA direction in dynamic weighting-based approaches is very appealing, recent empirical studies reveal that dynamic weighting methods may not outperform static weighting in some MOL benchmarks [18, 45], especially when it involves stochastic updates and deep models. Specifically, observed by [18], the vanilla stochastic MGDA can be under-optimized, leading to larger training errors than static weighting. The reason behind this training performance degradation has been studied in [52, 10], which suggest the vanilla stochastic MGDA has biased updates, and propose momentum-based methods to address this issue. Nevertheless, in [45], it is demonstrated that the training errors of MGDA and static weighting are similar, while their main difference lies in the generalization performance. Unfortunately, the reason behind this testing performance degradation is not fully understood and remains an open question.
To gain a deeper understanding of the dynamic weighting-based algorithms, a natural question is
Q1: What are the major sources of errors in dynamic weighting-based MOL methods?
To answer this question theoretically, we first introduce a proper measure of testing performance in MOL – the Pareto stationary measure in terms of the population objectives, which will immediately imply stronger measures such as Pareto optimality under strongly convex objectives. We then decompose this measure into generalization error and optimization error and further introduce a new metric on the distance to CA directions that is unique to MOL; see Sections 2.1 and 2.2.
To characterize the performance of MOL methods in a unified manner, we introduce a generic dynamic weighting-based MOL method that we term stochastic Multi-Objective gradient with DOuble sampling algorithm (MoDo), which uses a step size  $\gamma$  to control the change of dynamic weights. Roughly speaking, by controlling  $\gamma$ , MoDo approximates MGDA (large  $\gamma$ ) and static weighting algorithm ( $\gamma = 0$ ) as two special cases; see Section 2.3. We first analyze the generalization error of the model learned by MoDo through the lens of algorithmic stability [3, 14, 24] in the framework of statistical learning theory. To our best knowledge, this is the first-ever-known stability analysis for MOL algorithms. Here the key contributions lie in defining a new notion of stability - MOL uniform stability and then establishing a tight upper bound (matching lower bound) on the MOL uniform stability for MoDo algorithm that involves two coupled sequences; see Section 3.1. We then analyze the optimization error of MoDo and its distance to CA directions, where the key contributions lie in relaxing the bounded function value/gradient assumptions and significantly improving the convergence rate of state-of-the-art dynamic weighting-based MOL methods [10]; see Section 3.2.
Different from the stability analysis for single-objective learning [14], the techniques used in our generalization and optimization analysis allow to remove conflicting assumptions and use larger step sizes to ensure both small generalization and optimization errors, which are of independent interest.
Given the holistic analysis of dynamic weighting methods provided in Q1, a follow-up question is
Q2: What may cause the empirical performance degradation of dynamic weighting methods?
Figure 2: An illustration of three-way trade-off among optimization, generalization, and conflict avoidance in the strongly convex case;  $\alpha$  is the step size for  $x$ ,  $\gamma$  is the step size for weights  $\lambda$ , where  $o(\cdot)$  denotes a strictly slower growth rate,  $\omega(\cdot)$  denotes a strictly faster growth rate, and  $\Theta(\cdot)$  denotes the same growth rate. Arrows  $\downarrow$  and  $\uparrow$  respectively represent diminishing in an optimal rate and growing in a fast rate w.r.t.  $n$ , while  $\searrow$  represents diminishing w.r.t.  $n$ , but not in an optimal rate.
Region	Opt.	Gen.	Conflict	Step sizes α & γ	Iterations T & data size n
I	↑	↓	↓	γ = ω(α)	T = o(n)
II	↓	↑	↓	γ = ω(α)	T = ω(n)
III	↓	↓	↑	γ = o(T-1)	T = ω(n)
IV	↓	↓	↓	γ = Θ(α1/2) = Θ(T-1/4)	T = Θ(n4/5)
Visualizing MOL solution concepts. To reason the root cause for this, we first compare different MOL algorithms in a toy example shown in Figure 1. We find MGDA can navigate along CA directions and converge to the empirical Pareto front under all initializations, while static weighting gets stuck in some initializations; at the same time, the empirical Pareto solution obtained by MGDA may incur a larger population risk than the suboptimal empirical solution obtained by the static weighting method; finally, if the step size  $\gamma$  of dynamic weights is carefully tuned, MoDo can converge along CA directions to the empirical Pareto optimal solution that also generalizes well.
Aligned with this toy example, our theoretical results suggest a novel three-way trade-off in the performance of dynamic weighting-based MOL algorithm; see Section 3.3. Specifically, it suggests that the step size for dynamic weighting  $\gamma$  plays a central role in the trade-off among convergence to CA direction, convergence to empirical Pareto stationarity, and generalization error; see Figure 2. In this sense, MGDA has an edge in convergence to the CA direction but it could sacrifice generalization; the static weighting method cannot converge to the CA direction but guarantees convergence to the empirical Pareto solutions and their generalization. Our analysis also suggests that MoDo achieves a small population risk under a proper combination of step sizes and the number of iterations.
2 Problem Formulation and Target of Analysis
In this section, we first introduce the problem formulation of MOL, the target of analysis, the metric to measure its generalization, and then present the MGDA algorithm and its stochastic variant.
2.1 Preliminaries of MOL
Denote the vector-valued objective function on datum  $z$  as  $F_{z}(x) = [f_{z,1}(x),\ldots ,f_{z,M}(x)]$ . The training and testing performance of  $x$  can then be measured by the empirical objective  $F_{S}(x)$  and the population objective  $F(x)$  which are, respectively, defined as  $F_{S}(x)\coloneqq \frac{1}{n}\sum_{i = 1}^{n}F_{z_{i}}(x)$  and  $F(x)\coloneqq \mathbb{E}_{z\sim \mathcal{D}}[F_z(x)]$ . Their corresponding gradients are denoted as  $\nabla F_S(x)$  and  $\nabla F(x)\in \mathbb{R}^{d\times M}$ .
Analogous to the stationary solution and optimal solution in single-objective learning, we define the Pareto stationary point and Pareto optimal solution for MOL problem  $\min_{x\in \mathbb{R}^d}F(x)$  as follows.
Definition 1 (Pareto stationary and Pareto optimal). If there exists a convex combination of the gradient vectors that equals to zero, i.e., there exists  $\lambda \in \Delta^M$  such that  $\nabla F(x)\lambda = 0$ , then  $x \in \mathbb{R}^d$  is Pareto stationary. If there is no  $x \in \mathbb{R}^d$  and  $x \neq x^*$  such that, for all  $m \in [M]$ $f_m(x) \leq f_m(x^*)$  with  $f_{m'}(x) < f_{m'}(x^*)$  for at least one  $m' \in [M]$ , then  $x^*$  is Pareto optimal. If there is no  $x \in \mathbb{R}^d$  such that for all  $m \in [M]$ ,  $f_m(x) < f_m(x^*)$ , then  $x^*$  is weakly Pareto optimal.
By definition, at a Pareto stationary solution, there is no common descent direction for all objectives. A necessary and sufficient condition for  $x$  being Pareto stationary for smooth objectives is that  $\min_{\lambda \in \Delta^M} \| \nabla F(x)\lambda \| = 0$ . Therefore,  $\min_{\lambda \in \Delta^M} \| \nabla F(x)\lambda \|$  can be used as a measure of Pareto stationarity (PS) [9, 11, 42, 30, 10]. We will refer to the aforementioned quantity as the PS population risk henceforth and its empirical version as PS empirical risk or PS optimization error. We next introduce the target of our analysis based on the above definitions.
2.2 Target of analysis and error decomposition
In existing generalization analysis for MOL, measures based on function values have been used to derive generalization guarantees in terms of Pareto optimality [7, 41]. However, for general
nonconvex smooth MOL problems, it can only be guaranteed for an algorithm to converge to Pareto stationarity of the empirical objective, i.e., a small  $\min_{\lambda \in \Delta^M} \| \nabla F_S(x)\lambda \|$  [9, 11, 30, 10]. Thus, it is not reasonable to measure population risk in terms of Pareto optimality in this case. Furthermore, when all the objectives are convex or strongly convex, Pareto stationarity is a sufficient condition for weak Pareto optimality or Pareto optimality, respectively, as stated in Proposition 1.
Proposition 1 ([42, Lemma 2.2]). If  $f_{m}(x)$  are convex or strongly-convex for all  $m \in [M]$ , and  $x \in \mathbb{R}^{d}$  is a Pareto stationary point of  $F(x)$ , then  $x$  is weakly Pareto optimal or Pareto optimal.
Next, we proceed to decompose the PS population risk.
Error Decomposition. Given a model  $x$ , the PS population risk can be decomposed into
$$
\underbrace {\min  _ {\lambda \in \Delta^ {M}} \| \nabla F (x) \lambda \|} _ {\text {P S p o p u l a t i o n r i s k} R _ {\mathrm {p o p}} (x)} = \underbrace {\min  _ {\lambda \in \Delta^ {M}} \| \nabla F (x) \lambda \| - \min  _ {\lambda \in \Delta^ {M}} \| \nabla F _ {S} (x) \lambda \|} _ {\text {P S g e n e r a l i z a t i o n e r r o r} R _ {\mathrm {g e n}} (x)} + \underbrace {\min  _ {\lambda \in \Delta^ {M}} \| \nabla F _ {S} (x) \lambda \|} _ {\text {P S o p e r m i z a t i o n e r r o r} R _ {\mathrm {o p t}} (x)} \tag {2.1}
$$
where the optimization error quantifies the training performance, i.e., how well does model  $x$  perform on the training data; and the generalization error (gap) quantifies the difference between the testing performance on new data sampled from  $\mathcal{D}$  and the training performance, i.e., how well the model  $x$  performs on unseen testing data compared to the training data.
Let  $A:\mathcal{Z}^n\mapsto \mathbb{R}^d$  denote a randomized MOL algorithm. Given training data  $S$ , we are interested in the expected performance of the output model  $x = A(S)$ , which is measured by  $\mathbb{E}_{A,S}[R_{\mathrm{pop}}(A(S))]$ . From (2.1) and linearity of expectation, it holds that
$$
\mathbb {E} _ {A, S} \left[ R _ {\text {p o p}} (A (S)) \right] = \mathbb {E} _ {A, S} \left[ R _ {\text {g e n}} (A (S)) \right] + \mathbb {E} _ {A, S} \left[ R _ {\text {o p t}} (A (S)) \right]. \tag {2.2}
$$
Distance to CA direction. As demonstrated in Figure 1, the key merit of dynamic weighting over static weighting algorithms lies in its ability to navigate through conflicting gradients. Consider an update direction  $d = -\nabla F_S(x)\lambda$ , where  $\lambda$  is the dynamic weights from a simplex  $\lambda \in \Delta^M \coloneqq \{\lambda \in \mathbb{R}^M \mid \mathbf{1}^\top \lambda = 1, \lambda \geq 0\}$ . To obtain such a steepest CA direction in unconstrained learning that maximizes the minimum descent of all objectives, we can solve the following problem [11]
$$
\operatorname {C A} \quad d (x) = \underset {d \in \mathbb {R} ^ {d}} {\arg \min } \underset {m \in [ M ]} {\max } \left\{\langle \nabla f _ {S, m} (x), d \rangle + \frac {1}{2} \| d \| ^ {2} \right\} \tag {2.3a}
$$
$$
\stackrel {\text {e q u i v a l e n t} \quad \text {t o}} {\Longleftrightarrow} d (x) = - \nabla F _ {S} (x) \lambda^ {*} (x) \text {s . t .} \lambda^ {*} (x) \in \underset {\lambda \in \Delta^ {M}} {\arg \min } \| \nabla F _ {S} (x) \lambda \| ^ {2}. \tag {2.3b}
$$
Defining  $d_{\lambda}(x) = -\nabla F_S(x)\lambda$  given  $x \in \mathbb{R}^d$  and  $\lambda \in \Delta^M$ , we measure the distance to  $d(x)$  via [10]
$$
\text {C A} \quad \text {d i r e c t i o n e r r o r} \quad \mathcal {E} _ {\mathrm {c a}} (x, \lambda) := \| d _ {\lambda} (x) - d (x) \| ^ {2}. \tag {2.4}
$$
With the above definitions of measures that quantify the performance of algorithms in different aspects, we then introduce a stochastic gradient algorithm for MOL that is analyzed in this work.
2.3 A stochastic algorithm for MOL
MGDA finds  $\lambda^{*}(x)$  in (2.3b) using the full-batch gradient  $\nabla F_{S}(x)$ , and then constructs  $d(x) = -\nabla F_{S}(x)\lambda^{*}(x)$ , a CA direction for all empirical objectives  $f_{S,m}(x)$ . However, in practical statistical learning settings, the full-batch gradient  $\nabla F_{S}(x)$  may be costly to obtain, and thus one may resort to a stochastic estimate of  $\nabla F_{S}(x)$  instead. The direct stochastic counterpart of MGDA, referred to as the stochastic multi-gradient algorithm in [30], replaces the full-batch gradients  $\nabla f_{S,m}(x)$  in (2.3b) with their
Algorithm 1 Stochastic MGDA - MoDo algorithm
1: input Training data  $S$ , initial model  $x_0$ , weighting coefficient  $\lambda_0$ , and their learning rates  $\{\alpha_t\}_{t=0}^T, \{\gamma_t\}_{t=0}^T$ .
2: for  $t = 0, \dots, T - 1$  do
3: for objective  $m = 1,\ldots ,M$  do
4: Independent gradients  $\nabla f_{m,z_t,s}(x_t)$ $s\in [3]$
5: end for
6: Compute dynamic weight  $\lambda_{t + 1}$  following (2.5a)
7: Update model parameter  $x_{t + 1}$  following (2.5b)
8: end for
9: output  $x_{T}$
stochastic approximations  $\nabla f_{z,m}(x)$  for  $z\in S$ , which, however, introduces a biased stochastic estimate of  $\lambda_{t + 1}^{*}$ , thus a biased CA direction; see [10, Section 2.3].
Table 1: Comparison of optimization error, generalization error, and population risk under different assumptions for static and dynamic weighting. Use "NC", "SC" to represent nonconvex and strongly convex, and "Lip-C", "S" to represent Lipschitz continuous and smooth, respectively.
Assumption	Method	Optimization	Generalization	Risk	CA Distance
NC,Lip-C,S	Static	(αT)-1/2+α1/2	T1/2n-1/2	n-1/6	X
Dynamic	(αT)-1/2+α1/2+γ1/2	T1/2n-1/2	n-1/6	(γT)-1+α1/2γ-1/2+γ
SC,S	Static	(αT)-1/2+α1/2	n-1/2	n-1/2	X
Dynamic	(αT)-1/2+α1/2+γ1/2	{n-1/2, γ=O(T-1)}T1/2n-1/2, o.w.	{n-1/2}n-1/6	(γT)-1+α1/2γ-1/2+γ
To provide a tight analysis, we introduce a simple yet theoretically grounded stochastic variant of MGDA - stochastic Multi-Objective gradient with DOuble sampling algorithm (MoDo). MoDo obtains an unbiased stochastic estimate of the gradient of problem (2.3b) through double (independent) sampling and iteratively updates  $\lambda$ , because  $\mathbb{E}_{z_{t,1},z_{t,2}}[\nabla F_{z_{t,1}}(x_t)^\top \nabla F_{z_{t,2}}(x_t)\lambda_t] = \nabla F_S(x_t)^\top \nabla F_S(x_t)\lambda_t$ . At each iteration  $t$ , denote  $z_{t,s}$  as an independent sample from  $S$  with  $s \in [3]$ , and  $\nabla F_{z_{t,s}}(x_t)$  as a stochastic estimate of  $\nabla F_S(x_t)$ . MoDo updates  $x_t$  and  $\lambda_t$  as
$$
\lambda_ {t + 1} = \Pi_ {\Delta^ {M}} \left(\lambda_ {t} - \gamma_ {t} \nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t}) \lambda_ {t}\right) \tag {2.5a}
$$
$$
x _ {t + 1} = x _ {t} - \alpha_ {t} \nabla F _ {z _ {t, 3}} \left(x _ {t}\right) \lambda_ {t + 1} \tag {2.5b}
$$
where  $\alpha_{t},\gamma_{t}$  are step sizes, and  $\Pi_{\Delta^M}(\cdot)$  denotes Euclidean projection to the simplex  $\Delta^M$ . We have summarized the MoDo algorithm in Algorithm 1 and will focus on MoDo in the subsequent analysis.
3 Optimization, Generalization and Three-Way Trade-Off
This section presents the theoretical analysis of the PS population risk associated with the MoDo algorithm, where the analysis of generalization error is in Section 3.1 and that of optimization error is in Section 3.2. A summary of our main results is given in Table 1.
3.1 Multi-objective generalization and uniform stability
We first bound the expected PS generalization error by the generalization in gradients in Proposition 2, then introduce the MOL uniform stability and establish its connection to the generalization in gradients. Finally, we bound the MOL uniform stability.
Proposition 2. With  $\| \cdot \|_{\mathrm{F}}$  denoting the Frobenius norm,  $R_{\mathrm{gen}}(A(S))$  in (2.2) can be bounded by
$$
\mathbb {E} _ {A, S} \left[ R _ {\text {g e n}} (A (S)) \right] \leq \mathbb {E} _ {A, S} \left[ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} \right]. \tag {3.1}
$$
With Proposition 2, next we introduce the concept of MOL uniform stability tailored for MOL problems and show that PS generalization error in MOL can be bounded by the MOL uniform stability. Then we analyze their bound in general nonconvex case and strongly convex case, respectively.
Definition 2 (MOL uniform stability). A randomized algorithm  $A: \mathcal{Z}^n \mapsto \mathbb{R}^d$ , is MOL-uniformly stable with  $\epsilon_{\mathrm{F}}$  if for all neighboring datasets  $S, S'$  that differ in at most one sample, we have
$$
\sup  _ {z} \mathbb {E} _ {A} \left[ \| \nabla F _ {z} (A (S)) - \nabla F _ {z} (A \left(S ^ {\prime}\right)) \| _ {\mathrm {F}} ^ {2} \right] \leq \epsilon_ {\mathrm {F}} ^ {2}. \tag {3.2}
$$
Next we show the relation between the upper bound of PS generalization error in (3.1) and MOL uniform stability in Proposition 3.
Proposition 3 (MOL uniform stability and generalization). Assume for any  $z$ , the function  $F_{z}(x)$  is differentiable. If a randomized algorithm  $A: \mathcal{Z}^{n} \mapsto \mathbb{R}^{d}$  is MOL-uniformly stable with  $\epsilon_{\mathrm{F}}$ , then
$$
\mathbb {E} _ {A, S} [ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} ] \leq 4 \epsilon_ {\mathrm {F}} + \sqrt {n ^ {- 1} \mathbb {E} _ {S} \left[ \mathbb {V} _ {z \sim \mathcal {D}} (\nabla F _ {z} (A (S))) \right]} \tag {3.3}
$$
where the variance is defined as  $\mathbb{V}_{z\sim \mathcal{D}}(\nabla F_z(A(S))) = \mathbb{E}_{z\sim \mathcal{D}}\big[\| \nabla F_z(A(S)) - \mathbb{E}_{z\sim \mathcal{D}}[\nabla F_z(A(S))]\|_{\mathbf{F}}^2\big]$ .
Proposition 3 establishes a connection between the upper bound of the PS generalization error and the MOL uniform stability, where the former can be bounded above by the latter plus the variance of the stochastic gradient over the population data distribution. It is worth noting that the standard arguments of bounding the generalization error measured in function values by the uniform stability measured in function values [14, Theorem 2.2] is not applicable here as the summation and norm operators are not exchangeable. More explanations are given in the proof in Appendix B.1.
Theorem 1 (PS generalization error of MoDo in nonconvex case). If  $\sup_{z}\mathbb{E}_{A}\left[\| \nabla F_{z}(A(S))\|_{\mathrm{F}}^{2}\right]\leq G^{2}$  for any  $S$ , then the MOL uniform stability, i.e.,  $\epsilon_{\mathrm{F}}^{2}$  in Definition 2 is bounded by  $\epsilon_{\mathrm{F}}^{2}\leq 4G^{2}T / n$ . And the PS generalization error  $\mathbb{E}_{A,S}[R_{\mathrm{gen}}(A(S))] = \mathcal{O}(T^{\frac{1}{2}}n^{-\frac{1}{2}})$ .
Compared to the function value uniform stability upper bound in [14, Theorem 3.12] for nonconvex single-objective learning, Theorem 1 does not require a step size decay  $\alpha_{t} = \mathcal{O}(1 / t)$ , thus can enjoy at least a polynomial convergence rate of optimization errors w.r.t.  $T$ . Combining Theorem 1 with Proposition 3, to ensure the generalization error is diminishing with  $n$ , one needs to choose  $T = o(n)$  which lies in the "early stopping" regime and results in potentially large optimization error. We then provide a tighter bound in the strongly convex case that allows a larger choice of  $T$ . Below we list the standard assumptions used to derive the introduced MOL stability.
Assumption 1 (Lipschitz continuity of  $\nabla F_z(x)$ ). For all  $m \in [M]$ ,  $\nabla f_{z,m}(x)$  is  $\ell_{f,1}$ -Lipschitz continuous for all  $z$ . And  $\nabla F_z(x)$  is  $\ell_{F,1}$ -Lipschitz continuous in Frobenius norm for all  $z$ .
Assumption 2. For all  $m \in [M]$ ,  $z \in \mathcal{Z}$ ,  $f_{z,m}(x)$  is  $\mu$ -strongly convex w.r.t.  $x$ , with  $\mu > 0$ .