SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jie MA $^{1}$ , Tianyi Zhou $^{2}$ , Guodong Long $^{1}$ , Jing Jiang $^{1}$ , Chengqi Zhang $^{1}$ $^{1}$ Australian Artificial Intelligence Institute, FEIT, University of Technology Sydney  
 $^{2}$ University of Maryland  
jie.ma-5@student.uts.edu.au, Tianyi@umd.au  
{guodong.long, jing.jiang, chengqi.zhang}@uts.edu.au
Heterogeneous federated learning without assuming any structure is challenging due to the conflicts among non-identical data distributions of clients. In practice, clients often comprise near-homogeneous clusters so training a server-side model per cluster mitigates the conflicts. However, FL with client clustering often suffers from "clustering collapse", i.e., one cluster's model excels on increasing clients, and reduces to single-model FL. Moreover, cluster-wise models hinder knowledge sharing between clusters and each model depends on fewer clients. Furthermore, the static clustering assumption on data may not hold for dynamically changing models, which are sensitive to cluster imbalance/initialization or outliers. To address these challenges, we propose "Clustered Additive Modeling (CAM)", which applies a globally shared model  $\Theta_{g}$  on top of the cluster-wise models  $\Theta_{1:K}$ , i.e.,  $y = h(x; \Theta_{g}) + f(x; \Theta_{k})$  for clients of cluster- $k$ . The global model captures the features shared by all clusters so  $\Theta_{1:K}$  are enforced to focus on the difference among clusters. To train CAM, we develop a novel Fed-CAM algorithm that alternates between client clustering and training global/cluster models to predict the residual of each other. We can easily modify any existing clustered FL methods by CAM and significantly improve their performance without "clustering collapse" in different non-IID settings. We also provide a convergence analysis of Fed-CAM algorithm.
Federated learning (FL) trains a global model over distributed clients and enforces data localization, i.e., data only stay local for model training at the client side while the server periodically averages client models' weights to update a global model and broadcast it to all clients. When the local data distributions are identical across clients, one global model suffices to serve all clients [1] However, non-identical distributions across clients (i.e., statistical heterogeneity) [2] are more common in practical FL scenarios, which leads to conflicts between the global objective and local ones. Instead of applying one model to all the  $m$  clients, an ideal case for non-IID settings would be training a local model per client without any interference from others. However, the local data are usually insufficient so a global model trained on heterogeneous clients exploiting all their data can still be helpful to local training. Hence, non-IID FL methods [3-5] usually struggle to find a sweet spot between the global consensus and local personalization. Without any assumptions on the structure among clients, all clients' distributions can be equally different from each other so a global model is impacted by the conflicts of all clients and may provide limited guidance to their local training.
Clustered Federated Learning. That being said, non-IID clients in practice usually have rich structures that have not been explored by most existing FL methods. A common structure is clusters, i.e., heterogeneous clients can be grouped into several near-homogeneous clusters each composed of
Figure 1: Cluster sizes during IFCA vs.  $\mathrm{IFCA + CAM}$  in client/cluster-wise non-IID settings on CIFAR-10. Legend: cluster ID (cluster size) in the last round. CAM effectively mitigates clustering collapse/imbalance.
Figure 2: Test accuracy and macro-F1 (mean±std) of IFCA/FeSEM (w/o CAM) and IFCA/FeSEM (CAM) in cluster/client-wise non-IID settings on CIFAR-10 dataset. "IFCA(5)" represents IFCA with  $K = 5$  clusters. CAM consistently brings substantial improvement to IFCA/FeSEM on both metrics and in both settings.
clients with similar distributions. In practice, clusters might be associated with geographic/age/income groups, affiliations, etc. Hence, we can train a server-side model for each cluster, hence mitigating the conflicts caused by heterogeneity. Unfortunately, clients' cluster memberships are usually undefined or inaccessible due to sensitive/private information and have to be jointly optimized with cluster-wise models, as recent clustered FL [6-9] approaches do. They maintain  $K$  models  $\Theta_{1:K}$  for  $K$  clusters and assigns one  $\Theta_k$  to each client-  $i$  (with local data  $X_i$  and local model  $\theta_i$ ), e.g., by min-loss  $(\Theta_k$  with the minimum loss on  $X_i$ ) or K-means (the nearest  $\Theta_k$  to  $\theta_i$ ) criterion. Hence,  $1 \leq K \leq m$  models can accommodate more heterogeneity than single-model FL but also allows knowledge sharing among similar clients, which is lacking if training  $m$  client models independently. Hence, it may reach a better trade-off between global consensus and local personalization in non-IID settings.
However, compared to the general non-IID assumption, clustered FL's assumption might be too restrictive since it prohibits inter-cluster knowledge sharing and enforces every cluster-wise model's training to only depend on a few clients. This is contradictory to the widely studied strategy that different tasks or domains can benefit from sharing low-level or partial representations. It is due to the gap between the assumption of "clustered data distributions" and the algorithms of "clustering models (represented by loss vectors or model weights)": they are not equal and the latter is more restrictive. In other words, clients of different clusters can still benefit from feature/parameter sharing.
Moreover, clustered FL usually suffers from optimization instability because dynamically changing models can violate the static clustering assumption and lead to imbalanced cluster assignment, which affects  $\Theta_{1:K}$  and local training in the future. In particular: (1) Clustering collapse, i.e., the clients assigned to one cluster keeps growing so "the rich becomes richer (i.e., the cluster-wise model becomes even stronger)" until reducing to single-model FL. This happens because most clients tend to first learn shared features before focusing on client-specific ones; (2) Fragile to outliers such as malicious clients that may dominate some clusters and push all other benign ones to one or a few clusters; (3) Sensitive to initialization. The process highly depends on initial and earlier cluster assignments since they determine which clients' local training starts from the same model.
Main Contributions. To overcome the above problems of clustered FL, we propose a novel structured FL model termed "Clustered Additive Modeling (CAM)". Compared to clustered FL, CAM trains a global model  $\Theta_{g}$  on top of the  $K$  clusters' models  $\Theta_{1:K}$ . Its prediction for client- $i$  combines the outputs of  $\Theta_{g}$  and the associated cluster  $c(i)$ 's model, i.e.,  $y = h(x; \Theta_{g}) + f(x; \Theta_{c(i)})$ . This simple additive model removes the restriction of clustered FL by letting all clients share a base model  $\Theta_{g}$ . It enforces  $\Theta_{1:K}$  to focus on learning the different features between clusters, hence mitigating "clustering collapse". Moreover, CAM tends to learn balanced clusters and determine the number of clusters automatically (by starting from more clusters and then zeroing out some of them), as demonstrated in Fig 1. Furthermore, CAM is less vulnerable to outliers, which can be mainly captured by  $\Theta_{1:K}$  and have less impact on the global model  $\Theta_{g}$ . In addition, interactions between  $\Theta_{1:K}$  and  $\Theta_{g}$  make CAM less sensitive to initial cluster assignments since updating  $\Theta_{g}$  also changes the clustering.
CAM is a general model-agnostic method that can modify any existing clustered FL methods. As examples, we apply CAM to two representative methods, i.e., IFCA [6] and FeSEM [8]. In the optimization of CAM,  $\Theta_{1:K}$  and  $\Theta_g$  aim to fit the residual of each other's prediction. To this end, we propose an efficiently structured FL algorithm "Fed-CAM", which alternates between cluster assignment (server), local training (clients), and update of  $\Theta_{1:K}$  and  $\Theta_g$  (server). In experiments on several benchmarks in different non-IID settings, CAM significantly improves SOTA clustered FL methods, as shown in Fig 2. Moreover, we provide a convergence analysis of Fed-CAM algorithm.
Non-IID FL aims to tackle statistical heterogeneity across clients. FedAvg [1] is designed for the IID setting, so it suffers from client drift and slow convergence with non-IID clients [2]. To address this challenge, FedDANE [10] proposed a federated Newton-type optimization method by adapting a method for classical distributed optimization, i.e., DANE, to the FL setting. Instead of synchronizing all clients' models to be the same global model periodically, FedProx [3] only adds a proximal term to the local training objective that discourages the local model from drifting away from the global model and thus preserves the heterogeneity. [11] applies adaptive learning rates to clients and [12] conducts attention-based adaptive weighting to aggregate clients' models. [13] studies the convergence of the FedAvg in non-IID scenarios. Recent work also studies client-wise personalized FL [14-20], which aim to address the non-IID challenge by training a personalized model per client with the help of the shared global model. Their objectives focus on training local models rather than the server-side model.
Clustered FL assumes that non-IID clients can be partitioned into several groups and clients in each group share a cluster-wise model. It jointly optimizes the cluster assignments and the clusters' models. K-means-based methods [8] assign clusters to clients according to their model parameters' distance. CFL [21] divides clients into two partitions based on the cosine similarity between client gradients and then checks whether a partition is congruent according to the gradient norm. IFCA [6] and HypCluster [7] assign to each client the cluster whose model achieves the minimum loss on the client's data. Few-shot clustering has been introduced to clustered FL by [22, 23]. FedP2P [24] allows communication between clients in the same cluster. [25] uses cluster-based contexts to enhance the fine-tuning of personalized FL models. [9] proposed the first cluster-wise non-IID setting and a bi-level optimization framework unifying most clustered FL methods.
Additive modeling in FL trains multiple models and adds their outputs together as its prediction. It was introduced to FL very recently. Federated residual learning [26] proposed an FL algorithm to train an additive model for regression tasks. [27] applies additive modeling to combining the outputs of a shared model and a local model in a partial model personalization framework. However, additive modeling has not been studied for clustered FL.
3 Clustered Additive Modeling (CAM)
In this section, we introduce clustered additive modeling (CAM), which combines a global model and cluster-wise model prediction in FL. CAM conducts a joint optimization of the global and cluster-wise models defined by a cluster assignment criterion. In particular, we provide two examples of CAM using different cluster assignment criteria, i.e., min-loss and K-means, which have been adopted respectively by two SOTA clustered-FL methods, i.e., IFCA and FeSEM. For each of them, we derive alternating optimization procedures (i.e., IFCA-CAM and FeSEM-CAM) that can be implemented in FL setting using two parallel threads of local model training. At the end of this section, we unify both algorithms in a structured FL algorithm Fed-CAM.
Notations. We assume that there are  $m$  clients and  $K$  clusters, where client-  $i$  has  $n_i$  examples and all clients have  $n = \sum_{i=1}^{m} n_i$  examples. On the server side, we have a global model  $\Theta_g$  and  $K$  cluster-wise models  $\Theta_{1:K}$ . On the client side, we train  $m$  cluster models  $\theta_{1:m}^0$  used to update the global model  $\Theta_g$  in FL and  $\theta_{1:m}$  used to update the cluster-wise model  $\Theta_{c(i)}$  assigned to each client-  $i$ , where  $c(i)$  is its cluster label determined by the cluster assignment criterion  $c(\cdot)$ . We further define  $C_k \triangleq \{i \in [m] : c(i) = k\}$  as the set of clients in cluster-  $k$ . For simplicity, we will use  $X_i$  and  $Y_i$  to respectively represent the local training data on client-  $i$  and their ground truths, and  $\ell(Y_i, F(X_i))$  denotes the batch loss of model  $F(\cdot)$  on  $(X_i, Y_i)$ . A CAM model for client-  $i$  can be
$$
F _ {i} (\cdot) = h \left(\cdot ; \Theta_ {g}\right) + f \left(\cdot ; \Theta_ {c (i)}\right)). \tag {1}
$$
For classification,  $F_{i}(\cdot)$  produces logits and we can apply softmax to get the class probabilities.
3.1 IFCA-CAM: model performance-driven clustering
We extend the min-loss criterion used in IFCA [6] to CAM for cluster assignment, i.e., each client- $i$  is assigned to the cluster- $k$  whose model  $\Theta_{k}$  leads to the minimal loss of CAM on client- $i$ 's data, i.e.,
$$
c (i) = \arg \min  _ {k \in [ K ]} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \Theta_ {k}\right)\right). \tag {2}
$$
IFCA-CAM optimizes  $\Theta_g$  and  $\Theta_{1:K}$  for minimizing the above minimal loss over all the  $m$  clients, i.e.,
$$
\text {I F C A - C A M :} \min  _ {\Theta_ {g}, \Theta_ {1: K}} \sum_ {i = 1} ^ {m} \frac {n _ {i}}{n} \min  _ {k \in [ K ]} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \Theta_ {k}\right)\right), \tag {3}
$$
where the inner minimization performs the min-loss assignment in Eq. (2). We solve Eq. (3) by the following alternating minimization of cluster membership, cluster-wise models, and the global model.
(i) Cluster assignment by applying Eq. (2) to the latest  $\Theta_g$  and  $\Theta_{1:K}$ . This yields  $c(\cdot)$  and  $C_{1:K}$ .
(ii) Fixing  $\Theta_g$ , we can optimize the cluster-wise models  $\Theta_{1:K}$  by gradient descent:
$$
\Theta_ {k} \leftarrow \Theta_ {k} - \eta \sum_ {i \in C _ {k}} \frac {n _ {i}}{n} \nabla_ {\Theta_ {k}} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \Theta_ {k}\right)\right), \forall k \in [ K ]. \tag {4}
$$
In FL, the gradient can be approximated by aggregating the model updates of local models  $\theta_{i}$  from clients, whose training on the client side is: (1) initializing  $\theta_{i} \gets \Theta_{c(i)}$ ; (2) starting from the initialization, running  $E$  local epochs updating  $\theta_{i}$  by
$$
\theta_ {i} \leftarrow \theta_ {i} - \eta \nabla_ {\theta_ {i}} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \theta_ {i}\right)\right), \forall i \in [ m ]; \tag {5}
$$
and (3) aggregating the local model update  $\theta_{i} - \Theta_{k}$  from client  $i\in C_k$  to update  $\Theta_{k}$ , i.e.,
$$
\Theta_ {k} \leftarrow \left(1 - \sum_ {i \in C _ {k}} \frac {n _ {i}}{n}\right) \Theta_ {k} + \sum_ {i \in C _ {k}} \frac {n _ {i}}{n} \theta_ {i}. \tag {6}
$$
(iii) Fixing  $\Theta_{1:K}$ , we can optimize the global model  $\Theta_g$  by gradient descent:
$$
\Theta_ {g} \leftarrow \Theta_ {g} - \eta \sum_ {i \in [ m ]} \frac {n _ {i}}{n} \nabla_ {\Theta_ {g}} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \Theta_ {c (i)}\right)\right). \tag {7}
$$
In FL, this gradient step can be approximated by aggregating the local models  $\theta_i^0$  (similar to FedAvg): (1) initializing  $\theta_i^0\gets \Theta_g$ ; (2) running  $E$  local epochs training  $\theta_i^0$  by
$$
\theta_ {i} ^ {0} \leftarrow \theta_ {i} ^ {0} - \eta \nabla_ {\theta_ {i} ^ {0}} \ell \left(Y _ {i}, h \left(X _ {i}; \theta_ {i} ^ {0}\right) + f \left(X _ {i}; \Theta_ {c (i)}\right)\right), \forall i \in [ m ]; \tag {8}
$$
and (3) aggregating the updated local models  $\theta_i^0$  of all the  $m$  clients to update  $\Theta_g$ , i.e.,
$$
\Theta_ {g} \leftarrow \sum_ {i \in [ m ]} \frac {n _ {i}}{n} \theta_ {i} ^ {0}. \tag {9}
$$
We can run two parallel threads of local training for  $\theta_i^0$  and  $\theta_{i}$  for each client-  $i$  because their training in Eq. (8) and Eq. (5) does not depend on each other (but they both depend on the cluster assignments in (i)). This is analogous to the simultaneous update algorithm (FedSim) in [27]. One may also consider an alternative update algorithm (which may enjoy a slightly faster convergence) that iterates  $(\mathrm{i})\rightarrow (\mathrm{ii})\rightarrow (\mathrm{i})\rightarrow (\mathrm{iii})$  in each round. However, it doubles the communication rounds ((i) requires one communication round) and does not allow parallel local training. Since the alternative update does not show a significant empirical improvement over FedSim in [27], we mainly focus on the parallel one in the remainder of this paper.
3.2 FeSEM-CAM: parameter similarity-based clustering
We follow a similar procedure of IFCA-CAM to derive FeSEM-CAM, which applies a K-means style clustering to the client models  $\theta_{1:m}$ , whose objective is minimizing the sum of squares of client-cluster distance, i.e.,
$$
\min  _ {\Theta_ {1: K}} \sum_ {i = 1} ^ {m} \frac {n _ {i}}{n} \min  _ {j \in [ K ]} \| \theta_ {i} - \Theta_ {j} \| _ {2} ^ {2}. \tag {10}
$$
Hence, similar to FeSEM [8], FeSEM-CAM assigns the nearest cluster-wise model to each client and updates the cluster-wise models as the cluster centroids (i.e., K-means algorithm), i.e.,
$$
c (i) = \arg \min  _ {k \in [ K ]} \| \theta_ {i} - \Theta_ {k} \| _ {2} ^ {2}, \quad \Theta_ {k} \leftarrow \sum_ {i \in C _ {k}} \frac {n _ {i}}{\sum_ {i \in C _ {k}} n _ {i}} \theta_ {i}. \tag {11}
$$
We iterate the above K-means steps for a few times until convergence in practice. FeSEM-CAM applies the K-means objective in Eq. (10) as a regularization to the loss of CAM model  $\ell(Y_i, h(X_i; \Theta_g) + f(X_i; \theta_i))$ , i.e.,
$$
\text {F e S E M - C A M :} \quad \min  _ {\Theta_ {g}, \Theta_ {1: K}, \theta_ {1: m}} \sum_ {i = 1} ^ {m} \frac {n _ {i}}{n} \left[ \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \theta_ {i}\right)\right) + \frac {\lambda}{2} \min  _ {j \in [ K ]} \| \theta_ {i} - \Theta_ {j} \| _ {2} ^ {2} \right], \tag {12}
$$
where the minimization w.r.t.  $\Theta_{1:K}$  (with  $\theta_{1:m}$  fixed) recovers the (weighted) K-means objective in Eq. (10). Unlike IFCA-CAM, where client model  $\theta_{i}$  is an auxiliary/latent variable for FL not showing in the objective of Eq. (3), it is explicitly optimized in Eq. (12). Similar to IFCA-CAM, we solve Eq. (3) by iterating the following alternating minimization steps (i)-(iii).
(i) K-means clustering that iterates Eq. (11) for a few steps until convergence, which yields  $c(\cdot)$ ,  $C_{1:K}$ , and  $\Theta_{1:K}$ . The update of  $\Theta_{1:K}$  is analogous to Eq. (6).
(ii) Fixing  $\Theta_g$ , we optimize  $\theta_{1:m}$  by client-side local gradient descent:
$$
\theta_ {i} \leftarrow (1 - \eta \lambda) \theta_ {i} + \eta \lambda \Theta_ {c (i)} - \eta \frac {n _ {i}}{n} \nabla_ {\theta_ {i}} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \theta_ {i}\right)\right), \forall i \in [ m ]. \tag {13}
$$
The first two terms in Eq. (13) compute a linear interpolation between  $\theta_{i}$  and its assigned cluster's model  $\Theta_{c(i)}$ . This is a result of the K-means regularization term in Eq. (12) and keeps  $\theta_{i}$  close to  $\Theta_{c(i)}$ .
(iii) Fixing  $\theta_{1:m}$ , we can optimize  $\Theta_g$  by gradient descent:
$$
\Theta_ {g} \leftarrow \Theta_ {g} - \eta \sum_ {i \in [ m ]} \frac {n _ {i}}{n} \nabla_ {\Theta_ {g}} \ell \left(Y _ {i}, h \left(X _ {i}; \Theta_ {g}\right) + f \left(X _ {i}; \theta_ {i}\right)\right). \tag {14}
$$
$$
\theta_ {i} ^ {0} \leftarrow \theta_ {i} ^ {0} - \eta \nabla_ {\theta^ {0}} \ell \left(Y _ {i}, h \left(X _ {i}; \theta_ {i} ^ {0}\right) + f \left(X _ {i}; \theta_ {i}\right)\right), \forall i \in [ m ]; \tag {15}
$$
and (3) aggregating the updated local models  $\theta_i^0$  of all the  $m$  clients to update  $\Theta_g$  by Eq. (9).
In Algorithm 1, we propose a structured FL algorithm for CAM, i.e., Fed-CAM, which can unify the derived optimization procedures for IFCA-CAM and FeSEM-CAM and can be easily extended to other clustered FL and clustering criteria.
Warmup. As an alternating optimization framework, it would be unstable if both  $\Theta_g$  and  $\Theta_{1:K}$  are randomly initialized and jointly optimized in parallel since they may capture overlapping information and result in an inefficient competitive game. To encourage them to learn complementary knowledge, warmup training for one of them before the joint optimization is helpful. For example, a few rounds of FedAvg can produce a "warm"  $\Theta_g$ , whose predictions' residuals are more informative to train  $\Theta_{1:K}$ . Another warmup strategy could be to run a few local training epochs and extract warm  $\Theta_{1:K}$  by clustering the lightly-trained local models  $\theta_{1:m}$ . In Fed-CAM, we can apply the former warmup to IFCA-CAM and the latter to FeSEM-CAM.
Algorithm 1: Fed-CAM
initialize :Randomly initialize  $\Theta_{1:K}$  and  $\Theta_g$    
warmup :(1)  $\beta$  rounds of FedAvg to get an initial  $\Theta_g$  (IFCA-CAM) or (2)  $\beta$  epochs of local training only to get a initial  $\theta_{1:m}$  (FedSEM-CAM). Broadcast  $\Theta_g$  and  $\Theta_{1:K}$  to all clients.   
while not converge do /\* Client (in parallel) \*/ for every selected client i do Model performance-driven clustering (e.g., IFCA-CAM): cluster assignment by Eq. (2); Initialize  $\theta_i\gets \Theta_{c(i)}$  and  $\theta_i^0\gets \Theta_g$  Local training of  $\theta_{i}$  for  $E$  epochs: e.g., Eq. (5) (IFCA-CAM) or Eq. (13) (FeSEM-CAM); Local training of  $\theta_i^0$  for  $E$  epochs: e.g., Eq. (8) (IFCA-CAM) or Eq. (15) (FeSEM-CAM); Upload  $\theta_{i}$  and  $\theta_i^0$  to the server; /\* Server Update cluster-wise models  $\Theta_{1:K}$  : e.g., Eq. (6) (IFCA-CAM) or Eq. (11) (FeSEM-CAM); Update global model  $\Theta_g$  by Eq. (9); Broadcast  $\Theta_g$  and  $\Theta_{1:K}$  to all clients; output :Global model  $\Theta_g$  , cluster-wise models  $\Theta_{1:K}$  and  $c(i)\forall i\in [m]$
Based on the convergence analysis presented in [27], which aims to minimize the following objective:
$$
\min  _ {u, V} F (u, V) := \frac {1}{n} \sum_ {i = 1} ^ {m} F _ {i} (u, v _ {i}), \tag {16}
$$
where  $u$  represents the shared parameters and  $V = v_{1}, v_{2}, \dots, v_{m}$  denotes the personalized parameters. If we map  $\Theta_{g}$  to  $u$ , and  $\Theta_{1:K}$  to  $V$  respectively, this appears strikingly similar to our methods as illustrated in Equations 3 and 12. Provided that the clustering remains stable, we can employ the theoretical framework of [27]. And firstly, we make some standard assumptions for the convergence analysis as below.
Assumption 1. (Smoothness). For  $i = 1, \dots, m$ , the loss function  $l$  is continuously differentiable, and there exist constants  $L$  that  $\nabla_{\Theta_g} \ell(\Theta_g, \Theta_k)$  is L-Lipschitz with respect to  $\Theta_g$  and  $\Theta_k$ , and  $\nabla_{\Theta_k} \ell(\Theta_g, \Theta_k)$  is L-Lipschitz with respect to  $\Theta_g$  and  $\Theta_k$ .
Assumption 2. (Unbiased gradients and bounded variance). The stochastic gradients are unbiased and have bounded variance. For all  $\Theta_g$  and  $\Theta_k$ ,
$$
\mathbb {E} [ \widetilde {\nabla} _ {\Theta_ {g}} \ell (\Theta_ {g}, \Theta_ {k}) ] = \nabla_ {\Theta_ {g}} \ell (\Theta_ {g}, \Theta_ {k}), \mathbb {E} [ \widetilde {\nabla} _ {\Theta_ {k}} \ell (\Theta_ {g}, \Theta_ {k}) ] = \nabla_ {\Theta_ {k}} \ell (\Theta_ {g}, \Theta_ {k}),
$$