SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peiyao Xiao $^{1}$ , Hao Ban $^{2}$ , Kaiyi Ji $^{1*}$
$^{1}$ Department of CSE, University at Buffalo
$^{2}$  Zuoyi Technology
{peiyaoxi,kaiyiji}@buffalo.edu,bhsimon0810@gmail.com
Multi-objective optimization (MOO) has become an influential framework in many machine learning problems with multiple objectives such as learning with multiple criteria and multi-task learning (MTL). In this paper, we propose a new direction-oriented multi-objective formulation by regularizing the common descent direction within a neighborhood of a direction that optimizes a linear combination of objectives such as the average loss in MTL or a weighted loss that places higher emphasis on some tasks than the others. This formulation includes GD and MGDA as special cases, enjoys the direction-oriented benefit as in CAGrad, and facilitates the design of stochastic algorithms. To solve this problem, we propose Stochastic Direction-oriented Multi-objective Gradient descent (SDMGrad) with simple SGD type of updates, and its variant SDMGrad-OS with an efficient objective sampling. We develop a comprehensive convergence analysis for the proposed methods with different loop sizes and regularization coefficients. We show that both SDMGrad and SDMGrad-OS achieve improved sample complexities to find an  $\epsilon$ -accurate Pareto stationary point while achieving a small  $\epsilon$ -level distance toward a conflict-avoidant (CA) direction. For a constant-level CA distance, their sample complexities match the best known  $\mathcal{O}(\epsilon^{-2})$  without bounded function value assumption. Extensive experiments show that our methods achieve competitive or improved performance compared to existing gradient manipulation approaches in a series of tasks on multi-task supervised learning and reinforcement learning. Code is available at https://github.com/ml-opt-lab/sdmgrad.
In recent years, multi-objective optimization (MOO) has drawn intensive attention in a wide range of applications such as online advertising [1], hydrocarbon production [2], autonomous driving [3], safe reinforcement learning [4], etc. In this paper, we focus on the stochastic MOO problem, which takes the formulation of
$$
\min  _ {\theta \in \mathbb {R} ^ {m}} \boldsymbol {L} (\theta) := \left(\mathbb {E} _ {\xi} \left[ L _ {1} (\theta ; \xi) \right], \mathbb {E} _ {\xi} \left[ L _ {2} (\theta ; \xi) \right], \dots , \mathbb {E} _ {\xi} \left[ L _ {K} (\theta ; \xi) \right]\right), \tag {1}
$$
where  $m$  is the parameter dimension,  $K \geq 2$  is the number of objectives and  $L_{i}(\theta) \coloneqq \mathbb{E}_{\xi} L_{i}(\theta; \xi)$ . One important example of MOO in eq. (1) is the multi-task learning (MTL) [5, 6], whose objective
Algorithms	Batch size	Nonconvex	Bounded function value	Sample complexity	CA distance
SMG [11]	O(ε-2)	X	X	O(ε-4)	N/A
CR-MOGM [12]	O(1)	✓	✓	O(ε-2)	N/A
MoCo [13]	O(1)	✓	✓	O(ε-2)	N/A
MoDo [14]	O(1)	✓	X	O(ε-2)	O(1)
SDMGrad (Theorem 3)	O(1)	✓	X	O(ε-2)	O(1)
SDMGrad-OS (Theorem 4)	O(1)	✓	X	O(ε-2)	O(1)
MoCo [13]	O(1)	✓	X	O(ε-10)	O(ε)
MoDo [14]	O(1)	✓	X	O(ε-8)	O(ε)
SDMGrad (Corollary 1)	O(1)	✓	X	O(ε-6)	O(ε)
SDMGrad-OS (Theorem 2)	O(1)	✓	X	O(ε-6)	O(ε)
Table 1: Comparison of different algorithms for stochastic MOO problem to achieve an  $\epsilon$ -accurate Pareto stationary point. MoDo [14] is a concurrent work. Definition of CA distance can be found in Definition 2.  $\mathcal{O}(\cdot)$  omits logarithm factors.
function is often regarded as the average loss over  $K$  objectives.
$$
\theta^ {*} = \arg \min  _ {\theta \in \mathbb {R} ^ {m}} \left\{\bar {L} (\theta) \triangleq \frac {1}{K} \sum_ {i = 1} ^ {K} L _ {i} (\theta) \right\}, \tag {2}
$$
where  $L_{i}(\theta)$  is the loss function for task  $i$ . However, solving the MOO problem is challenging because it can rarely find a common parameter  $\theta$  that minimizes all individual objective functions simultaneously. As a result, a widely-adopted target is to find the Pareto stationary point at which there is no common descent direction for all objective functions. In this context, a variety of gradient manipulation methods have been proposed, including the multiple gradient descent algorithm (MGDA) [7], PCGrad [8], CAGrad [9] and Nash-MTL [10]. Among them, MGDA updates the model parameter  $\theta$  using a time-varying multi-gradient, which is a convex combination of gradients for all objectives. CAGrad further improves MGDA by imposing a new constraint on the difference between the common descent direction and the average gradient to ensure convergence to the minimum of the average loss of MTL. However, these approaches mainly focus on the deterministic case, and their stochastic versions still remain under-explored.
Stochastic MOO has not been well understood except for several attempts recently. Inspired by MGDA, [11] proposed stochastic multi-gradient (SMG), and established its convergence with convex objectives. However, their analysis requires the batch size to increase linearly. In the more practical nonconvex setting, [12] proposed a correlation-reduced stochastic multi-objective gradient manipulation (CR-MOGM) to address the non-convergence issue of MGDA, CAGrad and PCGrad in the stochastic setting. However, their analysis requires a restrictive assumption on the bounded function value. Toward this end, [13] recently proposed a method named MoCo as a stochastic counterpart of MGDA by introducing a tracking variable to approximate the stochastic gradient. However, their analysis requires that the number  $T$  of iterations is big at an order of  $K^{10}$ , where  $K$  is the number of objectives, and hence may be unsuitable for the scenario with many objectives. Thus, it is highly demanding but still challenging to develop efficient stochastic MOO methods with guaranteed convergence in the nonconvex setting under mild assumptions.
Motivated by the limitations of existing methods, we propose Stochastic Direction-oriented Multi-objective Gradient descent (SDMGrad), which is easy to implement, flexible with a new direction-oriented objective formulation, and achieves a better convergence performance under milder assumptions. Our specific contributions are summarized as follows.
New direction-oriented formulation. We propose a new MOO objective (see eq. (6)) by regularizing the common descent direction  $d$  within a neighborhood of a direction  $d_0$  that optimizes a
linear combination  $L_{0}(\theta)$  of objectives such as the average loss in MTL or a weighted loss that places higher emphasis on some tasks than the others. This formulation is general to include gradient descent (GD) and MGDA as special cases and takes a similar direction-oriented spirit as in CAGrad [9]. However, different from the objective in CAGrad [9] that enforces the gap between  $d$  and  $d_{0}$  to be small via constraint, our regularized objective is easier to design near-unbiased multi-gradient, which is crucial in developing provably convergent stochastic MOO algorithms.
New stochastic MOO algorithms. The proposed SDMGrad is simple with efficient stochastic gradient descent (SGD) type of updates on the weights of individual objectives in the multi-gradient and on the model parameters with (near-)unbiased (multi-)gradient approximations at each iteration. SDMGrad does not require either the linearly-growing batch sizes as in SMG or the extra tracking variable for gradient estimation as in MoCo. Moreover, we also propose SDMGrad-OS (which refers to SDMGrad with objective sampling) as an efficient and scalable counterpart of SDMGrad in the setting with a large number of objectives. SDMGrad-OS samples a subset of data points and objectives simultaneously in all updates and is particularly suitable in large-scale MTL scenarios.
Convergence analysis and improved complexity. We provide a comprehensive convergence analysis of SDMGrad for stochastic MOO with smooth nonconvex objective functions. For a constant-level regularization parameter  $\lambda$  (which covers the MGDA case), when an  $\epsilon$ -level conflict-avoidant (CA) distance (see Definition 2) is required, the sample complexities (i.e., the number of samples to achieve an  $\epsilon$ -accurate Pareto stationary point) of the proposed SDMGrad and SDMGrad-OS improve those of MoCo [13] and MoDo [14] by an order of  $\epsilon^{-4}$  and  $\epsilon^{-2}$  (see Table 1), respectively. In addition, when there is no requirement on CA distance, the sample complexities of SDMGrad and SDMGrad-OS are improved to  $\mathcal{O}(\epsilon^{-2})$ , matching the existing best result in the concurrent work [14]. For an increasing  $\lambda$ , we also show this convergent point reduces to a stationary point of the linear combination  $L_0(\theta)$  of objectives.
**Promising empirical performance.** We conduct extensive experiments in multi-task supervised learning and reinforcement learning on multiple datasets and show that SDMGrad can achieve competitive or improved performance compared to existing state-of-the-art gradient manipulation methods such as MGDA, PCGrad, GradDrop, CAGrad, IMTL-G, MoCo, MoDo, Nash-MTL and FAMO, and can strike a better performance balance on different tasks. SDMGrad-OS also exhibits a much better efficiency than SDMGrad in the setting with a large number of tasks due to the efficient objective sampling.
Multi-task learning. One important application of MOO is MTL, whose target is to learn multiple tasks with possible correlation simultaneously. Due to this capability, MTL has received significant attention in various applications in computer vision, natural language process, robotics, and reinforcement learning [5, 15, 16, 17]. A group of studies have focused on how to design better MTL model architectures. For example, [18, 19, 20] enhance the MTL models by introducing task-specific modules, an attention mechanism, and different activation functions for different tasks, respectively. Another line of research aims to learn a bunch of smaller models of local tasks split from the original problem, which are then aggregated into a single model via knowledge distillation [21]. Recent several works [22, 23] have explored the connection between MTL and gradient-based meta-learning. [24] and [25] highlight the importance of task grouping and unrelated tasks in MTL, respectively. This paper focuses on a single model by learning multiple tasks simultaneously with novel model-agnostic SDMGrad and SDMGrad-OS methods.
Gradient-based MOO. Various gradient manipulation methods have been developed to learn multiple tasks simultaneously. One popular class of approaches re-weight different objectives based on uncertainty [26], gradient norm [27], and training difficulty [28]. MOO-based approaches have received more attention due to their principled designs and training stability. For example, [6] viewed MTL as a MOO problem and proposed an MGDA-type method for optimization. A class of approaches has been proposed to address the gradient conflict problem. Among them, [8] proposed PCGrad by projecting the gradient direction of each task on the norm plane of other tasks. GradDrop randomly dropped out highly conflicted gradients [29], and RotoGrad rotated task gradients to alleviate the conflict [30]. [9] proposed CAGrad by constraining the common direction direction within a local region around the average gradient. These approaches mainly focus on the determin
istic setting. In the stochastic case, [13] proposed MoCo as a stochastic counterpart of MGDA, and provided a comprehensive convergence and complexity analysis. More recently, a concurrent work [14] analyzed a three-way trade-off among optimization, generalization, and conflict avoidance, providing an impact on designing the MOO algorithm. In addition, [31] found that scalarization SGD could be incapable of fully exploring the Pareto front compared with MGDA-variant methods. In this paper, we propose a new stochastic MOO method named SDMGrad, which benefits from a direction-oriented regularization and an improved convergence and complexity performance.
Concurrent work. A concurrent work [14] proposed a multi-objective approach named MoDo, which uses a similar double sampling strategy (see Section 4.2). However, there are still several differences between this work and [14]. First, our method benefits from a direction-oriented mechanism and is general to include MGDA and CAGrad as special cases, whereas MoDo [14] does not have such features. Second, MoDo takes a single-loop structure, whereas our SDMGrad features a double-loop scheme with an improved sample complexity. Third, [14] focuses more on the trade-off among optimization, generalization and conflict-avoidance, whereas our work focuses more on the optimization efficiency and performance balance in theory and in experiments.
3.1 Pareto Stationarity in MOO
Differently from single-objective optimization, MOO aims to find points at which all objectives cannot be further optimized. Consider two points  $\theta$  and  $\theta'$ . It is claimed that  $\theta$  dominates  $\theta'$  if  $L_i(\theta) \leq L_i(\theta')$  for all  $i \in [K]$  and  $L(\theta) \neq L(\theta')$ . In the general nonconvex setting, MOO aims to find a Pareto stationary point  $\theta$ , at which there is no common descent direction for all objectives. In other words, we say  $\theta$  is a Pareto stationary point if  $\text{range}(\nabla L(\theta)) \cap (-\mathbb{R}_{++}^K) = \emptyset$  where  $\mathbb{R}_{++}^K$  is the positive orthant cone.
3.2 MGDA and Its Stochastic Variants
Deterministic MGDA. The deterministic MGDA algorithm was first studied by [7], which updates the model parameter  $\theta$  along a multi-gradient  $d = \sum_{i=1}^{K} w_i^* g_i(\theta) = G(\theta)w^*$ , where  $G(\theta) = (g_1(\theta), g_2(\theta), \dots, g_K(\theta))$  are the gradients of different objectives, and  $w^* := (w_1^*, w_2^*, \dots, w_K^*)^T$  are the weights of different objectives obtained via solving the following problem.
$$
w ^ {*} \in \arg \min  _ {w} \| G (\theta) w \| ^ {2} \text {s . t .} w \in \mathcal {W} := \{w \in \mathbb {R} ^ {K} | \mathbf {1} ^ {T} w = 1, w \geq 0 \}, \tag {3}
$$
where  $\mathcal{W}$  is the probability simplex. The deterministic MGDA and its variants such as PCGrad and CAGrad have been well studied, but their stochastic counterparts have not been understood well.
Stochastic MOO algorithms. SMG [11] is the first stochastic variant of MGDA by replacing the full gradient  $G(\theta)$  in eq. (3) by its stochastic gradient  $G(\theta; \xi) \coloneqq (g_1(\theta; \xi), \dots, g_K(\theta; \xi))$ , and updates the model parameters  $\theta$  along the direction given by
$$
d _ {\xi} = G (\theta ; \xi) w _ {\xi} ^ {*} s. t. w _ {\xi} ^ {*} \in \underset {w} {\arg \min} \| G (\theta ; \xi) w \| ^ {2}.
$$
However, this direct replacement can introduce biased multi-gradient estimation, and hence SMG required to increase the batch sizes linearly with the iteration number. To address this limitation, [13] proposed MoCo by introducing an additional tracking variable  $y_{t,i}$  as the stochastic estimate of the gradient  $\nabla L_i(\theta)$ , which is iteratively updated via
$$
y _ {t + 1, i} = \Pi_ {\mathcal {Y} _ {i}} \left(y _ {t, i} - \beta_ {t} \left(y _ {t, i} - h _ {t, i}\right)\right), i = 1, 2, \dots , K, \tag {4}
$$
where  $\beta_{t}$  is the step size,  $\Pi_{\mathcal{Y}_i}$  denotes the projection on a bounded set  $\mathcal{Y}_i = \{y\in \mathbb{R}^m ||y||\leq C_{y,i}\}$  for some constant  $C_{y,i}$ , and  $h_{t,i}$  denotes stochastic estimator of  $\nabla L_i(\theta)$  at t-th iteration. Then, MoCo was shown to achieve an asymptotically unbiased multi-gradient, but with a relatively strong assumption that the number  $T$  of iterations is much larger than the number  $K$  of objectives. Thus, it is important but still challenging to develop provable and easy-to-implement stochastic MOO algorithms with mild assumptions.
We first provide a new direction-oriented MOO problem, and then introduce a new stochastic MOO algorithm named SDMGrad and its variant SDMGrad-OS with objective sampling.
4.1 Direction-oriented Multi-objective Optimization
MOO generally targets at finding a direction  $d$  to maximize the minimum decrease across all objectives via solving the following problem.
$$
\max  _ {d \in \mathbb {R} ^ {m}} \min  _ {i \in [ K ]} \left\{\frac {1}{\alpha} \left(L _ {i} (\theta) - L _ {i} (\theta - \alpha d)\right) \right\} \approx \max  _ {d \in \mathbb {R} ^ {m}} \min  _ {i \in [ K ]} \langle g _ {i}, d \rangle , \tag {5}
$$
where the first-order Taylor approximation of  $L_{i}$  is applied at  $\theta$  with a small stepsize  $\alpha$ .
In some scenarios, the target is to not only find the above common descent direction but also optimize a specific objective that is often a linear combination  $L_0(\theta) = \sum_i \widetilde{w}_i L_i(\theta)$  for some  $\widetilde{w} \in \mathcal{W}$  of objectives. For instance, MTL often takes the averaged loss over tasks as the objective function, and every element in  $\widetilde{w}$  will be set as  $\frac{1}{K}$ . In addition, it is quite possible that there is a preference for tasks. In this case, motivated by the framework in [32], we can regard  $\widetilde{w}$  as a preference vector and tend to approach a preferred stationary point along this direction. To address this problem, we propose the following multi-objective problem formulation by adding an inner-product regularization  $\lambda \langle g_0, d \rangle$  in eq. (5) such that the common descent direction  $d$  stays far away from a target direction  $g_0 = \sum_i \widetilde{w}_i g_i$ .
$$
\max  _ {d \in \mathbb {R} ^ {m}} \min  _ {i \in [ K ]} \langle g _ {i}, d \rangle - \frac {1}{2} \| d \| ^ {2} + \lambda \langle g _ {0}, d \rangle . \tag {6}
$$
In the above eq. (6), the term  $-\frac{1}{2}\|d\|^2$  is used to regularize the magnitude of the direction  $d$  and the constant  $\lambda$  controls the distance between the update vector  $d$  and the target direction  $g_0$ .
Compared to CAGrad. We note that CAGrad also takes a direction-oriented objective but uses a different constraint-enforced formulation as follows.
$$
\max  _ {d \in \mathbb {R} ^ {m}} \min  _ {i \in [ K ]} \left\langle g _ {i}, d \right\rangle \text {s . t .} \| d - h _ {0} \| \leq c \| h _ {0} \| \tag {7}
$$
where  $h_0$  is the average gradient and  $c \in [0,1)$  is a constant. To optimize eq. (7), CAGrad involves the evaluations of the product  $\| h_0\| \| g_w\|$  and the relation  $\frac{\|h_0\|}{\|g_w\|}$  with  $g_{w} \coloneqq \sum_{i} w_{i} g_{i}$ , both of which complicate the designs of unbiased stochastic gradient/multi-gradient in the  $w$  and  $\theta$  updates. As a comparison, our formulation in eq. (6), as shown later, admits very simple and provable stochastic algorithmic designs, while still enjoying the direction-oriented benefit as in CAGrad.
To efficiently solve the problem in eq. (6), we then substitute the relation that  $\min_{i\in [K]}\langle g_i,d\rangle = \min_{w\in \mathcal{W}}\langle \sum_{i = 1}^{K}g_i w_i,d\rangle = \min_{w\in \mathcal{W}}\langle g_w,d\rangle$  into eq. (6), and obtain an equivalent problem as
$$
\max  _ {d \in \mathbb {R} ^ {m}} \min  _ {w \in \mathcal {W}} \langle g _ {w} + \lambda g _ {0}, d \rangle - \frac {1}{2} \| d \| ^ {2}.
$$
By switching min and max in the above problem, which does not change the solution due to the concavity in  $d$  and the convexity in  $w$ , we finally aim to solve  $\min_{w\in \mathcal{W}}\max_{d\in \mathbb{R}^m}\langle g_w + \lambda g_0,d\rangle - \frac{1}{2}\| d\|^2$ , where the solution to the min problem on  $w$  is
$$
w _ {\lambda} ^ {*} \in \arg \min  _ {w \in \mathcal {W}} \frac {1}{2} \| g _ {w} + \lambda g _ {0} \| ^ {2}, \tag {8}
$$
and the solution to the max problem is  $d^{*} = g_{w_{\lambda}^{*}} + \lambda g_{0}$ .
Connection with MGDA and  $GD$ . It can be seen from eq. (8) that the updating direction  $d^{*}$  reduces to that of MGDA when we set  $\lambda = 0$ , and is consistent with that of GD for  $\lambda$  large enough. This consistency is also validated empirically in Appendix A.2 by varying  $\lambda$ .
4.2 Proposed SDMGrad Algorithm
The natural idea to solve the problem in eq. (8) is to use a simple SGD-type method. The detailed steps are provided in algorithm 1. At each iteration  $t$ , we run  $S$  steps of projected SGD with warm-start initialization and with a double-sampling-based stochastic gradient estimator, which is unbiased by noting that
$$
\mathbb {E} _ {\xi , \xi^ {\prime}} \left[ G (\theta_ {t}; \xi) ^ {T} \left(G (\theta_ {t}; \xi^ {\prime}) w _ {t, s} + \lambda g _ {0} (\theta_ {t}; \xi^ {\prime})\right) \right] = G (\theta_ {t}) ^ {T} \left(G (\theta_ {t}) w _ {t, s} + \lambda g _ {0} (\theta_ {t})\right),
$$
where  $\xi, \xi'$  are sampled data and  $g_0(\theta_t) = G(\theta_t)\widetilde{w}$  denotes the orientated direction. After obtaining the estimate  $w_{t,S}$ , SDMGrad updates the model parameters  $\theta_t$  based on the stochastic counterpart  $G(\theta_t;\zeta)w_{t,S} + \lambda g_0(\theta_t;\zeta)$  of the direction  $d^* = g_{w_\lambda^*} + \lambda g_0$  with a stepsize of  $\alpha_t$ . It can be seen that SDMGrad is simple to implement with efficient SGD type of updates on both  $w$  and  $\theta$  without introducing other auxiliary variables.
Algorithm 1 Stochastic Direction-oriented Multi-objective Gradient descent (SDMGrad)
1: Initialize: model parameters  $\theta_0$  and weights  $w_0$   
2: for  $t = 0,1,\dots,T - 1$  do  
3: for  $s = 0,1,\dots,S - 1$  do  
4: Set  $w_{t,0} = w_{t - 1,S}$  (warm start) and sample data  $\xi, \xi'$   
5:  $w_{t,s + 1} = \Pi_{\mathcal{W}}(w_{t,s} - \beta_{t,s}[G(\theta_t;\xi)^T(G(\theta_t;\xi')w_{t,s} + \lambda g_0(\theta_t;\xi'))]$ )  
6: end for  
7: Sample data  $\zeta$  and  $\theta_{t + 1} = \theta_t - \alpha_t(G(\theta_t;\zeta)w_{t,S} + \lambda g_0(\theta_t;\zeta))$   
8: end for
4.3 SDMGrad with Objective Sampling (SDMGrad-OS)
Another advantage of SDMGrad is its simple extension via objective sampling to the more practical setting with a large of objectives, e.g., in large-scale MTL. In this setting, we propose SDMGrad-OS by replacing the gradient matrix  $G(\theta_t; \xi)$  in Algorithm 1 by a matrix  $H(\theta_t; \xi)$  with randomly sampled columns, which takes the form of
$$
H \left(\theta_ {t}; \xi , \mathcal {S}\right) = \left(h _ {1} \left(\theta_ {t}; \xi\right), h _ {2} \left(\theta_ {t}; \xi\right), \dots , h _ {K} \left(\theta_ {t}; \xi\right)\right), \tag {9}
$$
where  $h_i(\theta_t; \xi) = g_i(\theta_t; \xi)$  with a probability of  $n / K$  or 0 otherwise,  $S$  corresponds to the randomness by objective sampling, and  $n$  is the expected number of sampled objectives. Then, the stochastic gradient in the  $w$  update is adapted to
$$
\text {(G r a d i e n t o n} w) \quad \frac {K ^ {2}}{n ^ {2}} H (\theta_ {t}; \xi , \mathcal {S}) ^ {T} \big (H (\theta_ {t}; \xi^ {\prime}, \mathcal {S} ^ {\prime}) w _ {t, s} + \lambda h _ {0} (\theta_ {t}; \xi^ {\prime}, \mathcal {S} ^ {\prime}) \big),
$$
where  $h_0(\cdot) = H(\cdot)\widetilde{w}$ . Similarly, the updating direction  $d = \frac{K}{n} H(\theta_t;\zeta ,\widetilde{S})w_{t,S} + \frac{K}{n}\lambda h_0(\theta_t;\zeta ,\widetilde{S})$ . In the practical implementation, we first use np.random.binomial to sample a 0-1 sequence  $s$  following a Bernoulli distribution with length  $K$  and probability  $n / K$ , and then compute the gradient  $g_{i}(\theta ;\xi)$  of  $G(\theta ;\xi)$  only if the  $i^{th}$  entry of  $s$  equals to 1. This sampling strategy greatly speeds up the training with a large number of objectives, as shown by the reinforcement learning experiments in Section 6.3.
5.1 Definitions and Assumptions
We first make some standard definitions and assumptions, as also adopted by existing MOO studies in [11, 12, 13]. Since we focus on the practical setting where the objectives are nonconvex, algorithms are often expected to find an  $\epsilon$ -accurate Pareto stationary point, as defined below.
Definition 1. We say  $\theta \in \mathbb{R}^m$  is an  $\epsilon$ -accurate Pareto stationary point if  $\mathbb{E}\left[\min_{w \in \mathcal{W}} \|g_w(\theta)\|^2\right] \leq \epsilon$  where  $\mathcal{W}$  is the probability simplex.