SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xidong Wu†
xidong_wu@outlook.com
Jianhui Sun†
Charlottesville, VA 22903
js9gu@virginia.edu
Zhengmian Hu
huzhengmian@gmail.com
Aidong Zhang
aidong@virginia.edu
Heng Huang*
Conditional stochastic optimization has found applications in a wide range of machine learning tasks, such as invariant learning, AUPRC maximization, and meta-learning. As the demand for training models with large-scale distributed data grows in these applications, there is an increasing need for communication-efficient distributed optimization algorithms, such as federated learning algorithms. This paper considers the nonconvex conditional stochastic optimization in federated learning and proposes the first federated conditional stochastic optimization algorithm (FCSG) with a conditional stochastic gradient estimator and a momentum-based algorithm (i.e., FCSG-M). To match the lower bound complexity in the single-machine setting, we design an accelerated algorithm (Acc-FCSG-M) via the variance reduction to achieve the best sample and communication complexity. Compared with the existing optimization analysis for Meta-Learning in FL, federated conditional stochastic optimization considers the sample of tasks. Extensive experimental results on various tasks validate the efficiency of these algorithms.
The conditional stochastic optimization arises throughout a wide range of machine learning tasks, such as the policy evaluation in reinforcement learning [5], invariant learning [16], instrumental variable regression in causal inference [30], Model-Agnostic Meta-Learning (MAML) [10], AUPRC maximization [28] and so on. Recently many efficient conditional stochastic optimization algorithms have been developed [16-18, 28, 34, 32] to solve the corresponding machine learning problems and applications. However, all existing conditional stochastic optimization algorithms were only designed for centralized learning (i.e., model and data both deployed at a single machine) or finite-sum optimization, without considering the large-scale online distributed scenario. Many federated learning algorithms [26, 22, 29, 23, 44, 39, 25, 45] were proposed since FL is a communication-efficient training paradigm for large-scale machine learning training preserving data privacy. In federated learning, clients update the model locally, and the global server aggregates the model
$^\S$ This work was partially supported by NSF CNS 2213700 and CCF 2217071 at UVA.
*This work was partially supported by NSF IIS 1838627, 1837956, 1956002, 2211492, CNS 2213701, CCF 2217003, DBI 2225775 at Pitt and UMD.
parameters periodically. Although federated learning has been actively applied to numerous real-world applications in the past years, the federated conditional stochastic optimization problem is still underexplored. To bridge this gap, in this paper we study the following federated conditional stochastic optimization problem:
$$
\min  _ {x \in \mathcal {X}} F (x) := \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\xi^ {n}} f _ {\xi^ {n}} ^ {n} \left(\mathbb {E} _ {\eta^ {n} | \xi^ {n}} g _ {\eta^ {n}} ^ {n} \left(x, \xi^ {n}\right)\right), \tag {1}
$$
where  $\mathcal{X} \subseteq \mathbb{R}^d$  is a closed convex set,  $\mathbb{E}_{\xi^n} f_{\xi^n}^n(\cdot): \mathbb{R}^{d'} \to \mathbb{R}$  is the outer-layer function on the  $n$ -th device with the randomness  $\xi^n$ , and  $\mathbb{E}_{\eta^n | \xi^n} g_{\eta^n}^n(\cdot, \xi^n): \mathbb{R}^d \to \mathbb{R}^{d'}$  is the inner-layer function on the  $n$ -th device with respect to the conditional distribution of  $\eta^n \mid \xi^n$ . We assume  $f_{\xi}^n(\cdot)$  and  $g_{\eta}^n(\cdot, \xi)$  are continuously differentiable. The objective subsumes two stochastic functions in (1), where the inner functions rely on the randomnesses of both inner and outer layers, and  $\xi$  and  $\eta$  are not independent, which makes the federated conditional stochastic optimization more challenging compared with the standard federated learning optimization problems.
Federated conditional stochastic optimization contains the standard federated learning optimization as a special situation when the inner-layer function  $g_{\eta^n}^n (x,\xi^n) = x$ . In addition, federated stochastic compositional optimization is similar to federated conditional stochastic optimization given that both problems contain two-layer nested expectations. However, they are fundamentally different. In federated stochastic compositional optimization, the inner randomness  $\eta$  and the outer randomness  $\xi$  are independent and data samples of the inner layer are available directly from  $\eta$  (instead of a conditional distribution as in Problem (1)). Therefore, when randomnesses  $\eta$  and  $\xi$  are independent and  $g_{\eta^n}^n (x,\cdot) = g_{\eta^n}^n (x)$ , (1) is converted into federated stochastic compositional optimization [11].
Recently, to solve the conditional stochastic optimization problem efficiently, [16] studied the sample complexity of the sample average approximation for conditional stochastic optimization. Afterward, [17] proposed the algorithm called biased stochastic gradient descent (BSGD) and an accelerated algorithm called biased SpiderBoost (BSpiderBoost). The convergence guarantees of BSGD and BSpiderBoost under different assumptions are established. More recently, [28, 34, 37, 32, 14] reformulated the AUC maximization into a finite-sum version of conditional stochastic optimization and introduced algorithms to solve it. In an increasing amount of distributed computing settings, efficient federated learning algorithms are absolutely necessary but still lacking. An important example of conditional stochastic optimization is MAML. In meta-learning, we attempt to train models that can efficiently adapt to unseen tasks via learning with metadata from similar tasks [10]. When the tasks are distributed at different clients, a federated version of MAML would be beneficial to leverage information from all workers [3]. A lot of existing efforts [19, 11] have been made to convert FL MAML into federated compositional optimization. Nonetheless, they ignore the sample of tasks in MAML, and federated conditional stochastic optimization problems have never been studied. Thus, there exists a natural question: Can we design federated algorithms for conditional stochastic optimization while maintaining the fast convergence rate to solve Problem (1)?
In this paper, we give an affirmative answer to the above question. We propose a suite of approaches to solve Problem (1) and establish their corresponding convergence guarantee. To our best knowledge, this is the first work that thoroughly studies federated conditional stochastic optimization problems and provides completed theoretical analysis. Our proposed algorithm matches the lower-bound sample complexity in a single-machine setting and obtains convincing results in empirical experiments. Our main contributions are four-fold:
1) we propose the federated conditional stochastic gradient (FCSG) algorithm to solve Problem (1). We establish the theoretical convergence analysis for FCSG. In the general nonconvex setting, we prove that FCSG has a sample complexity of  $O(\epsilon^{-6})$  and communication complexity of  $O(\epsilon^{-3})$  to reach an  $\epsilon$ -stationary point, and achieves an appealing linear speedup w.r.t the number of clients.
2) To further improve the empirical performance of our algorithm, we introduce a momentum-based FCSG algorithm, called FCSG-M since the momentum-based estimator could reduce noise from samples with history information. FCSG-M algorithm obtains the same theoretical guarantees as FCSG.
3) To reach the lower bound of sample complexity of the single-machine counterpart [17], we propose an accelerated version of FCSG-M (Acc-FCSG-M) based on the momentum-based variance reduction technique. We prove that Acc-FCSG-M has a sample complexity
of  $O(\epsilon^{-5})$ , and communication complexity of  $O(\epsilon^{-2})$ , which matches the best sample complexity attained by single-machine algorithm BSpiderBoost with variance reduction.
4) Experimental results on the robust logistic regression, MAML and AUPRC maximization tasks validate the effectiveness of our proposed algorithms.
2.1 Conditional Stochastic Optimization
[16] studied the generalization error bound and sample complexity of the sample average approximation (SAA) for conditional stochastic optimization. Subsequently, [17] proposed a class of efficient stochastic gradient-based methods for general conditional stochastic optimization to reach either a global optimal point in the convex setting or a stationary point in the nonconvex setting, respectively. In the nonconvex setting, BSGD has the sample complexity of  $O(\epsilon^{-6})$  and a variance reduction algorithm (BSpiderBoost) has the sample complexity of  $O(\epsilon^{-5})$ . [18] utilized the Monte Carlo method to achieve better results compared with the vanilla stochastic gradient method. Recently, [28] converted AUPRC maximization optimization into the finite-sum version of the conditional stochastic optimization and propose adaptive and non-adaptive stochastic algorithms to solve it. Similarly, recent work [34] used moving average techniques to improve the convergence rate of AUPRC maximization optimization and provide theoretical analysis for the adaptive algorithm. Furthermore, [32] focused on finite-sum coupled compositional stochastic optimization, which limits the outer-layer function to the finite-sum structure. The algorithms proposed in [32] improved oracle complexity with the parallel speed-up. More recently, [14] use federated learning to solve AUC maximization. However, algorithms proposed in [28, 34, 32, 14] for AUC maximization have a significant limitation because they maintain an inner state for each data point. As a result, its convergence rate depends on the number of data points and cannot be extended to other tasks and large-scale model training. It is also not applicable to online learning due to the dependence on each local data point. [38] consider the decentralised online AUPRC maximization but the theoretical analysis cannot be applied into the federated learning.
2.2 Stochastic Compositional Optimization
Recently, a related optimization problem, stochastic compositional optimization, has attracted widely attention [36, 41, 11] and solve the following objective:
$$
\min  _ {x \in \mathcal {X}} F (x) := \mathbb {E} _ {\xi} f _ {\xi} \left(\mathbb {E} _ {\eta} g _ {\eta} (x)\right). \tag {2}
$$
To address this problem, [36] developed SCGD, which utilizes the moving average technique to estimate the inner-layer function value. [35] further developed an accelerated SCGD method with the extrapolation-smoothing scheme. Subsequently, a series of algorithms [20, 43, 15, 41] were presented to improve the complexities using the acceleration or variance reduction techniques.
More recently, [19] and [11] studied the stochastic compositional problem in federated learning. [19] transformed the distributionally robust federated learning problem (i.e., a minimax optimization problem) into a simple compositional optimization problem by using KL divergence regularization and proposed the first federated learning compositional algorithm and analysis. [8] formulated the model personalization problem in federated learning as a model-agnostic meta-learning problem. In personalized federated learning, each client's task assignment is fixed and there is no task sampling on each client in the training procedure. The sampling of the inner layer and outer layer are independent. Therefore, personalized federated learning is formulated as the stochastic compositional optimization [19]. [33] solves personalized federated learning utilizing SCGD, in contrast to SGD in [19], to reduce the convergence complexities. However, the algorithm in [33] has a drawback in that keeping an inner state for each task is necessary, which is prohibitively expensive in large-scale settings. More recently, [11] proposed a momentum-like method for nonconvex problems with better complexities to solve the stochastic compositional problem in the federated learning setting. Although [11] claims their algorithm can be used in the MAML problem, it does not consider the two fundamental characteristics in MAML, i.e., task sampling and the dependency of inner data distribution on the sampled task.
Overall, problems (1) and (2) differ in two aspects: i) in Problem (2), the inner randomness  $\eta$  and the outer randomness  $\xi$  are independent, while in Problem (1),  $\eta$  is conditionally dependent on the  $\xi$ ; and ii) in Problem (1), the inner function depends on both  $\xi$  and  $\eta$ . Therefore, Problem (2) can be regarded as a special case of (1). Thus, the conditional stochastic optimization (1) is more general.
Table 1: Complexity summary of proposed federated conditional stochastic optimization algorithms to reach an  $\varepsilon$ -stationary point. Sample complexity is defined as the number of calls to the First-order Oracle (IFO) by clients to reach an  $\varepsilon$ -stationary point. Communication complexity denotes the total number of back-and-forth communication rounds between each client and the central server required to reach an  $\varepsilon$ -stationary point.
Algorithm	Sample	Communication
FCSG	O(ε-6)	O(ε-3)
FCSG-M	O(ε-6)	O(ε-3)
Lower Bound [16]	O(ε-5)	-
Acc-FCSG-M	O(ε-5)	O(ε-2)
For solving the problem (1), we first consider the local objective  $F^n(x)$  and its gradient. We have
$$
F ^ {n} (x) = \mathbb {E} _ {\xi^ {n}} f _ {\xi^ {n}} ^ {n} \left(\mathbb {E} _ {\eta^ {n} | \xi^ {n}} g _ {\eta^ {n}} ^ {n} (x, \xi^ {n})\right)
$$
$$
\nabla F ^ {n} (x) = \mathbb {E} _ {\xi^ {n}} \left[ \left(\mathbb {E} _ {\eta^ {n} | \xi^ {n}} \nabla g _ {\eta^ {n}} ^ {n} (x, \xi^ {n})\right) ^ {\top} \nabla f _ {\xi^ {n}} ^ {n} \left(\mathbb {E} _ {\eta^ {n} | \xi^ {n}} g _ {\eta^ {n}} ^ {n} (x, \xi^ {n})\right) \right]
$$
Since there are two layers of stochastic functions, the standard stochastic gradient estimator is not an unbiased estimation for the full gradient. Instead of constructing an unbiased stochastic gradient estimator, [17] considered a biased estimator of  $\nabla F(x)$  using one sample of  $\xi$  and  $m$  samples of  $\eta$ :
$$
\nabla \hat {F} ^ {n} (x; \xi^ {n}, \mathcal {B} _ {n}) = (\frac {1}{m} \sum_ {\eta_ {j} ^ {n} \in \mathcal {B} _ {n}} \nabla g _ {\eta_ {j} ^ {n}} ^ {n} (x, \xi^ {n})) ^ {\top} \nabla f _ {\xi^ {n}} ^ {n} (\frac {1}{m} \sum_ {\eta_ {j} ^ {n} \in \mathcal {B} _ {n}} g _ {\eta_ {j} ^ {n}} ^ {n} (x, \xi^ {n}))
$$
where  $\mathcal{B}_n = \left\{\eta_j^n\right\}_{j = 1}^m$ . And  $\nabla \hat{F}^n (x;\xi^n,\mathcal{B}_n)$  is the gradient of an empirical objective such that
$$
\hat {F} ^ {n} \left(x; \xi^ {n}, \mathcal {B} _ {n}\right) := f _ {\xi^ {n}} ^ {n} \left(\frac {1}{m} \sum_ {\eta_ {j} ^ {n} \in \mathcal {B} _ {n}} g _ {\eta_ {j} ^ {n}} \left(x, \xi^ {n}\right)\right) \tag {3}
$$
Assumption 3.1. (Smoothness)  $\forall n\in [N]$ , the function  $f_{\xi^n}^n (\cdot)$  is  $S_{f}$ -Lipschitz smooth, and the function  $g_{\eta^n}^n (\cdot ,\xi^n)$  is  $S_{g}$ -Lipschitz smooth, i.e., for a sample  $\xi^n$  and  $m$  samples  $\{\eta_j^n\}_{j = 1}^m$  from the conditional distribution  $P(\eta^n\mid \xi^n),\forall x_1,x_2\in \operatorname {dom}f^n (\cdot)$ , and  $\forall y_{1},y_{2}\in \operatorname {dom}g^{n}(\cdot)$ , there exist  $S_{f} > 0$  and  $S_{g} > 0$  such that
$$
\mathbb {E} \| \nabla f _ {\xi^ {n}} ^ {n} (x _ {1}) - \nabla f _ {\xi^ {n}} ^ {n} (x _ {2}) \| \leq S _ {f} \| x _ {1} - x _ {2} \| \quad \mathbb {E} \| \nabla g _ {\eta^ {n}} ^ {n} (y _ {1}, \xi^ {n}) - \nabla g _ {\eta^ {n}} ^ {n} (y _ {2}, \xi^ {n}) \| \leq S _ {g} \| y _ {1} - y _ {2} \|
$$
Assumption 3.1 is a widely used assumption in optimization analysis. Many single-machine stochastic algorithms use this assumption, such as BSGD [17], SPIDER [9], STORM [4], ADSGD [1], and D2SG [12]. In distributed learning, the convergence analysis of distributed learning algorithms, such as DSAL [2], and many federated learning algorithms such as MIME [21], Fed-GLOMO [6], STEM [23] and FAFED [39] also depend on it.
Assumption 3.2. (Bounded gradient)  $\forall n\in [N]$ , the function  $f^n (\cdot)$  is  $L_{f}$ -Lipschitz continuous, and the function  $g^{n}(\cdot)$  is  $L_{g}$ -Lipschitz continuous, i.e.,  $\forall x\in \mathrm{dom}f^{n}(\cdot)$ , and  $\forall y\in \mathrm{dom}g^{n}(\cdot)$ , the second moments of functions are bounded as below:
$$
\mathbb {E} \| \nabla f _ {\xi^ {n}} ^ {n} (x) \| ^ {2} \leq L _ {f} ^ {2} \quad \mathbb {E} \| \nabla g _ {\eta^ {n}} ^ {n} \left(y _ {1}, \xi^ {n}\right) \| ^ {2} \leq L _ {g} ^ {2}
$$
Assumption 3.2 is a typical assumption in the multi-layer problem optimization to constrain the upper bound of the gradient of each layer, as in [36, 28, 11, 14].
Assumption 3.3. (Bounded variance) [17]  $\forall n\in [N]$ , and  $x\in \mathcal{X}$ :
$$
\sup  _ {\xi^ {n}, x \in \mathcal {X}} \mathbb {E} _ {\eta^ {n} | \xi^ {n}} \| g _ {\eta^ {n}} (x, \xi^ {n}) - \mathbb {E} _ {\eta^ {n} | \xi^ {n}} g _ {\eta^ {n}} (x, \xi^ {n}) \| ^ {2} \leq \sigma_ {g} ^ {2}
$$
where  $\sigma_g^2 < +\infty$ . Assumption 3.3 indicates that the random vector  $g_{\eta^n}$  has bounded variance.
Algorithm 1 FCSG and FCSG-M Algorithm
1: Input: Parameters:  $T$ , momentum weight  $\beta$ , learning rate  $\alpha$ , the number of local updates  $q$ , inner batch size  $m$  and outer batch size  $b$ , as well as the initial outer batch size B;  
2: Initialize:  $x_0^n = \bar{x}_0 = \frac{1}{N}\sum_{k=1}^{N}x_0^n$ . Draw  $B$  samples of  $\{\xi_{t,1}^n,\dots,\xi_{t,B}^n\}$  and draw  $m$  samples  $\mathcal{B}_{0,i}^n = \{\eta_{ij}^n\}_{j=1}^m$  from  $P(\eta^n \mid \xi_{0,i}^n)$  for each  $\xi_{0,i}^n \in \{\xi_{t,1}^n,\dots,\xi_{t,B}^n\}$ ;  $u_1^n = \frac{1}{B}\sum_{i=1}^{B}\nabla \hat{F}^n(x_0^n;\xi_0^n,\mathcal{B}_{0,i}^n)$ .  
3: for  $t=1,2,\ldots,T$  do  
4: for  $n=1,2,\ldots,N$  do  
5: if mod  $(t,q)=0$  then  
6: Server Update:  
7:  $u_t^n = \bar{u}_t = \frac{1}{N}\sum_{i=1}^{N}u_t^n$   
8:  $x_t^n = \bar{x}_t = \frac{1}{N}\sum_{n=1}^{N}(x_{t-1}^n - \alpha u_t^n)$   
9: else  
10:  $x_t^n = x_{t-1}^n - \alpha u_t^n$   
11: end if  
12: Draw  $b$  samples of  $\{\xi_{t,1}^n,\dots,\xi_{t,b}^n\}$   
13: Draw  $m$  samples  $\mathcal{B}_{t,n}^n = \{\eta_{ij}^n\}_{j=1}^m$  from  $P(\eta^n \mid \xi_{t,i}^n)$  for each  $\xi_{t,i}^n \in \{\xi_{t,1}^n,\dots,\xi_{t,b}^n\}$ ,  
14:  $u_{t+1}^n = \frac{1}{b}\sum_{i=1}^{b}\nabla \hat{F}^n(x_t^n;\xi_t^n,\mathcal{B}_{t,i}^n)$   
15:  $u_{t+1}^n = (1-\beta)u_t^n + \frac{\beta}{b}\sum_{i=1}^{b}\nabla \hat{F}^n(x_t^n;\xi_{t,i}^n,\mathcal{B}_{t,i}^n)$   
16: end for  
17: end for  
18: Output:  $x$  chosen uniformly random from  $\{\bar{x}_t\}_{t=1}^T$ .
4 Proposed Algorithms
In the section, we propose a class of federated first-order methods to solve the Problem (1). We first design a federated conditional stochastic gradient (FCSG) algorithm with a biased gradient estimator and the momentum-based algorithm FCSG-M. To further accelerate our algorithm and achieve the lower bound of sample complexity of the single-machine algorithm, we design the Acc-FCSG-M with a variance reduction technique. Table 1 summarizes the complex details of each algorithm.
4.1 Federated Conditional Stochastic Gradient (FCSG)
First, we design a federated conditional stochastic gradient (FCSG) algorithm with the biased gradient estimator. We leverage a mini-batch of conditional samples to construct the gradient estimator  $u_{t}$  with controllable bias as (6). At each iteration, clients update their local models  $x_{t}$  with local data, which can be found in Line 9-14 of Algorithm 1. Once every  $q$  local iterations, the server collects local models and returns the averaged models to each client, as Line 5-8 of Algorithm 1. Here, the number of local update steps  $q$  is greater than 1 such that the number of communication rounds is reduced to  $T / q$ . The details of the method are summarized in Algorithm 1. Then we study the convergence properties of our new algorithm FCSG. Detailed proofs are provided in the supplementary materials.
Theorem 4.1. Suppose Assumptions 3.1, 3.2 and 3.3 hold, if  $\alpha \leq \frac{1}{6qS_F}$ , FCSG has the following convergence result
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F (\bar {x} _ {t}) \| ^ {2} \leq \frac {2 [ F (\bar {x} _ {0}) - F (\bar {x} _ {T}) ]}{\alpha T} + \frac {2 L _ {g} ^ {2} S _ {f} ^ {2} \sigma_ {g} ^ {2}}{m} + \frac {2 \alpha S _ {F} L _ {f} ^ {2} L _ {g} ^ {2}}{N} + 4 2 (q - 1) q \alpha^ {2} L _ {f} ^ {2} L _ {g} ^ {2} S _ {F} ^ {2}
$$
Corollary 4.2. We choose  $\alpha = \frac{1}{6S_F}\sqrt{\frac{N}{T}}$  and  $q = (T / N^3)^{1 / 4}$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F (\bar {x} _ {t}) \| ^ {2} \leq \frac {1 2 S _ {F} [ F (\bar {x} _ {0}) - F (\bar {x} _ {*}) ]}{(N T) ^ {1 / 2}} + \frac {2 L _ {g} ^ {2} S _ {f} ^ {2} \sigma_ {g} ^ {2}}{m} + \frac {L _ {f} ^ {2} L _ {g} ^ {2}}{6 (N T) ^ {1 / 2}} + \frac {1 9 L _ {f} ^ {2} L _ {g} ^ {2}}{9 (N T) ^ {1 / 2}}
$$
Remark 4.3. We choose  $B = b = O(1) \geq 1$ , and  $m = O(\varepsilon^{-2})$ , according to Corollary 4.2 to let  $\frac{1}{T} \sum_{t=0}^{T-1} \| \nabla F(\bar{x}_t) \|^2 \leq \varepsilon^2$ , we get  $T = O(N^{-1} \varepsilon^{-4})$ .  $O(N^{-1} \varepsilon^{-4})$  indicates the linear speedup of our algorithm. Given  $q = (T / N^3)^{1/4}$ , the communication complexity is  $\frac{T}{q} = (NT)^{3/4} = O(\varepsilon^{-3})$ . Then the sample complexity is  $mT = O(N^{-1} \varepsilon^{-6})$ .
4.2 Federated Conditional Stochastic Gradient with Momentum (FCSG-M)
Next, we propose a momentum-based local updates algorithm (FCSG-M) for federated conditional stochastic optimization problems. Momentum is a popular technique widely used in practice for training deep neural networks. The motivation behind it in local updates is to use the historic information (i.e., averaging of stochastic gradients) to reduce the effect of stochastic gradient noise. The details of our method are shown in Algorithm 1.
Initially, each device utilizes the standard stochastic gradient descent method to update the model parameter, as seen in Line 2 of Algorithm 1. Afterward, compared with FCSG, at each step, each client uses momentum-based gradient estimators  $u_{t}$  to update the local model, which can be found in Lines 9-15 of Algorithm 1. The coefficient  $\beta$  for the update of  $u_{t}$  should satisfy  $0 < \beta < 1$ . In every  $q$  iterations, the clients communicate  $\{x_{t},u_{t}\}$  to the server, which computes the  $\{\bar{x}_t,\bar{u}_t\}$ , and returns the averaged model and gradient estimator to each client, as Lines 5-8 of Algorithm 1. Then we study the convergence properties of our new algorithm FCSG-M. The details of the proof are provided in the supplementary materials.
Theorem 4.4. Suppose Assumptions 3.1, 3.2 and 3.3 hold,  $\alpha \leq \frac{1}{6qS_F}$  and  $\beta = 5S_F\eta$ . FCSG-M has the following convergence result
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla F (\bar {x} _ {t}) \| ^ {2} \leq 2 \frac {F (\bar {x} _ {0}) - F (\bar {x} _ {T})}{\alpha T} \\ + \frac {9 6 S _ {F} ^ {2}}{5} q ^ {2} \alpha^ {2} \left[ L _ {f} ^ {2} L _ {g} ^ {2} \left(1 + \frac {1}{N}\right) + 3 L _ {f} ^ {2} L _ {g} ^ {2} \right] + \frac {4 L _ {g} ^ {2} S _ {f} ^ {2} \sigma_ {g} ^ {2}}{m} + \frac {8 L _ {f} ^ {2} L _ {g} ^ {2}}{\beta B T} + \frac {8 \beta L _ {f} ^ {2} L _ {g} ^ {2}}{N} \\ \end{array}
$$
Corollary 4.5. We choose  $\alpha = \frac{1}{6S_F}\sqrt{\frac{N}{T}}$ ,  $q = (T / N^3)^{1/4}$ , we have
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F (\bar {x} _ {t}) \| ^ {2} \leq \frac {1 2 S _ {F} [ F (\bar {x} _ {0}) - F (\bar {x} _ {*}) ]}{(N T) ^ {1 / 2}} \\ + \frac {1 1 2 L _ {f} ^ {2} L _ {g} ^ {2}}{3 (N T) ^ {1 / 2}} + \frac {4 L _ {g} ^ {2} S _ {f} ^ {2} \sigma_ {g} ^ {2}}{m} + \frac {4 8 L _ {f} ^ {2} L _ {g} ^ {2}}{5 (N T) ^ {1 / 2}} + \frac {2 0 L _ {f} ^ {2} L _ {g} ^ {2}}{3 (N T) ^ {1 / 2}} \\ \end{array}
$$