SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinwen Zhang*
Philadelphia, PA, USA
ellenz@temple.edu
Yihan Zhang *
yihan.zhang0002@temple.edu
Tianbao Yang
College Station, TX, USA
ianbao-yang@tamu.edu
Richard Souvenir
souvenir@temple.edu
Hongchang Gao
hongchang.gao@temple.edu
Federated learning has attracted increasing attention due to the promise of balancing privacy and large-scale learning; numerous approaches have been proposed. However, most existing approaches focus on problems with balanced data, and prediction performance is far from satisfactory for many real-world applications where the number of samples in different classes is highly imbalanced. To address this challenging problem, we developed a novel federated learning method for imbalanced data by directly optimizing the area under curve (AUC) score. In particular, we formulate the AUC maximization problem as a federated compositional minimax optimization problem, develop a local stochastic compositional gradient descent ascent with momentum algorithm, and provide bounds on the computational and communication complexities of our algorithm. To the best of our knowledge, this is the first work to achieve such favorable theoretical results. Finally, extensive experimental results confirm the efficacy of our method.
Federated learning [19, 34] is a paradigm for training a machine learning model across multiple devices without sharing the raw data from each device. Practically, models are trained on each device, and, periodically, model parameters are exchanged between these devices. By not sharing the data itself, federated learning allows private information in the raw data to be preserved to some extent. This property has allowed federated learning to be proposed for numerous real-world computer vision and machine learning tasks.
Currently, one main drawback of existing federated learning methodologies is the assumption of balanced data, where the number of samples across classes is essentially the same. Most real-world data is imbalanced, even highly imbalanced. For example, in the healthcare domain, it is common to encounter problems where the amount of data from one class (e.g., patients with a rare disease) is significantly lower than the other class(es), leading to a distribution that is highly imbalanced. Traditional federated learning methods do not handle such imbalanced data scenarios very well. Specifically, training the classifier typically requires minimizing a classification-error induced loss function (e.g., cross-entropy). As a result, the resulting classifier may excel at classifying the majority, while failing to classify the minority.
To handle imbalanced data classification, the most common approach is to train the classifier by optimizing metrics designed for imbalanced data distributions. For instance, under the single-machine
setting, Ying et al. [35] proposed to train the classifier by maximizing the Area under the ROC curve (AUC) score. Since the AUC score can be affected by performance on both the majority and minority classes, the classifier is less prone to favoring one class above the rest. Later, [8, 40] extended this approach to federated learning. However, optimizing the AUC score introduces some new challenges, since maximizing the AUC score requires solving a minimax optimization problem, which is more challenging to optimize than conventional minimization problems. More specifically, when the classifier is a deep neural network, recent work [39] has demonstrated empirically that training a deep classifier from scratch with the AUC objective function cannot learn discriminative features; the resulting classifier sometimes fails to achieve satisfactory performance. To address this issue, [39] developed a compositional deep AUC maximization model under the single-machine setting, which combines the AUC loss function and the traditional cross-entropy loss function, leading to a stochastic compositional minimax optimization problem. This compositional deep AUC maximization model can learn discriminative features, achieving superior performance over traditional models consistently.
Considering its remarkable performance under the single-machine setting, a natural question is: How can a compositional deep AUC maximization model be applied to federated learning? The challenge is that the loss function of the compositional model involves two levels of distributed functions. Moreover, the stochastic compositional gradient is a biased estimation of the full gradient. Therefore, on the algorithmic design side, it is unclear what variables should be communicated when estimating the stochastic compositional gradient. On the theoretical analysis side, it is unclear if the convergence rate can achieve the linear speedup with respect to the number of devices in the presence of a biased stochastic compositional gradient, two levels of distributed functions, and the minimax structure of the loss function.
To address the aforementioned challenges, in this paper, we developed a novel local stochastic compositional gradient descent ascent with momentum (LocalSCGDAM) algorithm for federated compositional deep AUC maximization. In particular, we demonstrated which variables should be communicated to address the issue of two levels of distributed functions. Moreover, for this nonconvex-strongly-concave problem, we established the convergence rate of our algorithm, disclosing how the communication period and the number of devices affect the computation and communication complexities. Specifically, with theoretical guarantees, the communication period can be as large as  $O(T^{1/4}/K^{3/4})$  so that our algorithm can achieve  $O(1/\sqrt{KT})$  convergence rate and  $O(T^{3/4}/K^{3/4})$  communication complexity, where  $K$  is the number of devices and  $T$  is the number of iterations. To the best of our knowledge, this is the first work to achieve such favorable theoretical results for the federated compositional minimax problem. Finally, we conduct extensive experiments on multiple image classification benchmark datasets, and the experimental results confirm the efficacy of our algorithm.
In summary, we made the following important contributions in our work.
- We developed a novel federated optimization algorithm, which enables compositional deep AUC maximization for federated learning.
- We established the theoretical convergence rate of our algorithm, demonstrating how it is affected by the communication period and the number of devices.
- We conducted extensive experiments on multiple imbalanced benchmark datasets, confirming the efficacy of our algorithm.
Imbalanced Data Classification. In the field of machine learning, there has been a fair amount of work addressing imbalanced data classification. Instead of using conventional cross-entropy loss functions, which are not suitable for imbalanced datasets, optimizing the AUC score has been proposed. For instance, Ying et al. [35] have proposed the minimax loss function to optimize the AUC score for learning linear classifiers. Liu et al. [18] extended this minimax method to deep neural networks and developed the nonconvex-strongly-concave loss function. Yuan et al. [39] have proposed a compositional training framework for end-to-end deep AUC maximization, which minimizes a compositional loss function, where the outer-level function is an AUC loss, and the inner-level function substitutes a gradient descent step for minimizing a traditional loss. Based on empirical results, this approach improved the classification performance by a large degree.
To address stochastic minimax optimization problems, there have been a number of diverse efforts launched in recent years. In particular, numerous stochastic gradient descent ascent (SGDA) algo
rithms [42, 17, 21, 33] have been proposed. However, most of them focus on non-compositional optimization problems. On the other hand, to solve compositional optimization problems, existing work [31, 41, 38, 7] tends to only focus on the minimization problem. Only two recent works [5, 39] studied how to optimize the compositional minimax optimization problem, but they focused on the single-machine setting.
Federated Learning. In recent years, federated learning has shown promise with several empirical studies in the field of large-scale deep learning [19, 20, 26]. The FedAvg [19] algorithm has spawned a number of variants [25, 37, 36] designed to address the minimization problem. For instance, by maintaining a local momentum, Yu et al. [36] have provided rigorous theoretical studies for the convergence of the local stochastic gradient descent with momentum (LocalSGDM) algorithm. These algorithms are often applied to balanced datasets, and their performance in the imbalanced imbalanced regime is lacking.
To address minimax optimization for federated learning, Deng et al.[2] proposed local stochastic gradient descent ascent (LocalSGDA) algorithms to provably optimize federated minimax problems. However, their theoretical convergence rate was suboptimal and later improved by [27]. However, neither method could achieve a linear speedup with respect to the number of devices. Recently, Sharma et al. [23] developed the local stochastic gradient descent ascent with momentum (LocalSGDAM) algorithm, whose convergence rate is able to achieve a linear speedup for nonconvex-strongly-concave optimization problems. Guo et al. [8] proposed and analyzed a communication-efficient distributed optimization algorithm (CoDA) for the minimax AUC loss function under the assumption of PL-condition, which can also achieve a linear speedup, in theory. Yuan et al. [40] extended CoDA to heterogeneous data distributions and established its convergence rate. Shen et al. [24] proposed to handle the imbalance issue via introducing a constrained optimization problem and then formulated it as an unconstrained minimax problem. While these algorithms are designed for federated minimax problems, none can deal with the federated compositional minimax problems.
To handle the compositional optimization problem under the distributed setting, Gao et al. [3] developed the first parallel stochastic compositional gradient descent algorithm and established its convergence rate for nonconvex problems, inspiring many federated learning methods [12, 28, 4, 27, 7] in the past few years. For instance, [12] directly used a biased stochastic gradient to do local updates, suffering from large sample and communication complexities. [4] employed the stochastic compositional gradient and the momentum technique, which can achieve much better sample and communication complexities than [12]. [7] considered the setting where the inner-level function is distributed on different devices, which shares similar sample and communication complexities as [4]. However, all these works restrict their focus on the compositional minimization problem.
It is worth noting that our method is significantly different from the heterogeneous federated learning approaches [14, 16, 29]. Specifically, most existing heterogeneous federated learning approaches consider a setting where the local distribution is imbalanced but the global distribution is balanced. For example, Scaffold [14] method uses the global gradient to correct the local gradient because it assumes the global gradient is computed on a balanced distribution. On the contrary, our work considers a setting where both the local and global distributions are imbalanced, which is much more challenging than existing heterogeneous federated learning methods.
In this section, we first introduce the compositional deep AUC maximization model under the single-machine setting and then provide the problem setup in federated learning.
3.1 Compositional Deep AUC Maximization
Training classifiers by optimizing AUC ([9, 11]) is an effective way to handle highly imbalanced datasets. However, traditional AUC maximization models typically depend on pairwise sample input, limiting the application to large-scale data. Recently, Ying et al. [35] formulated AUC maximization model as a minimax optimization problem, defined as follows:
$$
\begin{array}{l} \min  _ {\mathbf {w}, \tilde {w} _ {1}, \tilde {w} _ {2}} \max  _ {\tilde {w} _ {3}} \mathcal {L} _ {A U C} (\mathbf {w}, \tilde {w} _ {1}, \tilde {w} _ {2}, \tilde {w} _ {3}; a, b) \triangleq (1 - p) \left(h (\mathbf {w}; a) - \tilde {w} _ {1}\right) ^ {2} \mathbb {I} _ {[ b = 1 ]} - p (1 - p) \tilde {w} _ {3} ^ {2} \tag {1} \\ + p \left(h (\mathbf {w}; a) - \tilde {w} _ {2}\right) ^ {2} \mathbb {I} _ {[ b = - 1 ]} + 2 \left(1 + \tilde {w} _ {3}\right) \left(p h (\mathbf {w}; a) \mathbb {I} _ {[ b = - 1 ]} - (1 - p) h (\mathbf {w}; a) \mathbb {I} _ {[ b = 1 ]}\right), \\ \end{array}
$$
where  $h$  denotes the classifier parameterized by  $\mathbf{w} \in \mathbb{R}^d$ ,  $\tilde{w}_1 \in \mathbb{R}$ ,  $\tilde{w}_2 \in \mathbb{R}$ ,  $\tilde{w}_3 \in \mathbb{R}$  are the parameters for measuring AUC score,  $(a, b)$  substitutes the sample's feature and label,  $p$  is the prior probability of the positive class, and  $\mathbb{I}$  is an indicator function that takes value 1 if the argument is true and 0 otherwise. Such a minimax objective function decouples the dependence of pairwise samples so that it can be applied to large-scale data.
Since training a deep classifier from scratch with  $\mathcal{L}_{AUC}$  loss function did not yield satisfactory performance, Yuan et al. [39] developed the compositional deep AUC maximization model, which is defined as follows:
$$
\min  _ {\tilde {\mathbf {w}}, \tilde {w} _ {1}, \tilde {w} _ {2}} \max  _ {\tilde {w} _ {3}} \mathcal {L} _ {A U C} (\tilde {\mathbf {w}}, \tilde {w} _ {1}, \tilde {w} _ {2}, \tilde {w} _ {3}; a, b) \quad s. t. \quad \tilde {\mathbf {w}} = \mathbf {w} - \rho \nabla_ {\mathbf {w}} \mathcal {L} _ {C E} (\mathbf {w}; a, b). \tag {2}
$$
Here,  $\mathcal{L}_{CE}$  denotes the cross-entropy loss function,  $\mathbf{w} - \rho \nabla_{\mathbf{w}}\mathcal{L}_{CE}$  indicates using the gradient descent method to minimize the cross-entropy loss function, where  $\rho > 0$  is the learning rate. Then, for the obtained model parameter  $\tilde{\mathbf{w}}$ , one can optimize it through optimizing the AUC loss function.
By denoting  $g(\mathbf{x}) = \mathbf{x} - \rho \Delta(\mathbf{x})$  and  $\mathbf{y} = \tilde{w}_3$ , where  $\mathbf{x} = [\mathbf{w}^T, \tilde{w}_1, \tilde{w}_2]^T$ ,  $\Delta(\mathbf{x}) = [\nabla_{\mathbf{w}} \mathcal{L}_{CE}(\mathbf{w}; a, b)^T, 0, 0]^T$ , and  $f = \mathcal{L}_{AUC}$ , Eq. (2) can be represented as a generic compositional minimax optimization problem as follows:
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d _ {1}}} \max  _ {\mathbf {y} \in \mathbb {R} ^ {d _ {2}}} f (g (\mathbf {x}), \mathbf {y}) ， \tag {3}
$$
where  $g$  is the inner-level function and  $f$  is the outer-level function. It is worth noting that when  $f$  is a nonlinear function, the stochastic gradient regarding  $\mathbf{x}$  is a biased estimation of the full gradient. As such, the stochastic compositional gradient [31] is typically used to optimize this kind of problem. We will demonstrate how to adapt this compositional minimax optimization problem to federated learning and address the unique challenges.
3.2 Problem Setup
In this paper, to optimize the deep compositional AUC maximization problem under the cross-silo federated learning setting, we will concentrate on developing an efficient optimization algorithm to solve the following generic federated stochastic compositional minimax optimization problem:
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d _ {1}}} \max  _ {\mathbf {y} \in \mathbb {R} ^ {d _ {2}}} \frac {1}{K} \sum_ {k = 1} ^ {K} f ^ {(k)} \left(\frac {1}{K} \sum_ {k ^ {\prime} = 1} ^ {K} g ^ {\left(k ^ {\prime}\right)} (\mathbf {x}), \mathbf {y}\right), \tag {4}
$$
where  $K$  is the number of devices,  $g^{(k)}(\cdot) = \mathbb{E}_{\xi \sim \mathcal{D}_g^{(k)}}[g^{(k)}(\cdot ;\xi)]\in \mathbb{R}^{d_g}$  denotes the inner-level function for the data distribution  $\mathcal{D}_g^{(k)}$  of the  $k$ -th device,  $f^{(k)}(\cdot ,\cdot) = \mathbb{E}_{\zeta \sim \mathcal{D}_f^{(k)}}[f^{(k)}(\cdot ,\cdot ;\zeta)]$  represents the outer-level function for the data distribution  $\mathcal{D}_f^{(k)}$  of the  $k$ -th device. It is worth noting that both the inner-level function and the outer-level function are distributed on different devices, which is significantly different from traditional federated learning models. Therefore, we need to design a new federated optimization algorithm to address this unique challenge.
Here, we introduce the commonly-used assumptions from existing work [6, 41, 39, 5] for investigating the convergence rate of our algorithm.
Assumption 1. The gradient of the outer-level function  $f^{(k)}(\cdot, \cdot)$  is  $L_{f}$ -Lipschitz continuous where  $L_{f} > 0$ , i.e.,
$$
\left\| \nabla_ {g} f ^ {(k)} \left(\mathbf {z} _ {1}, \mathbf {y} _ {1}\right) - \nabla_ {g} f ^ {(k)} \left(\mathbf {z} _ {2}, \mathbf {y} _ {2}\right) \right\| ^ {2} \leq L _ {f} ^ {2} \| \left(\mathbf {z} _ {1}, \mathbf {y} _ {1}\right) - \left(\mathbf {z} _ {2}, \mathbf {y} _ {2}\right) \| ^ {2}, \tag {5}
$$
$$
\left\| \nabla_ {\mathbf {y}} f ^ {(k)} \left(\mathbf {z} _ {1}, \mathbf {y} _ {1}\right) - \nabla_ {\mathbf {y}} f ^ {(k)} \left(\mathbf {z} _ {2}, \mathbf {y} _ {2}\right) \right\| ^ {2} \leq L _ {f} ^ {2} \| \left(\mathbf {z} _ {1}, \mathbf {y} _ {1}\right) - \left(\mathbf {z} _ {2}, \mathbf {y} _ {2}\right) \| ^ {2},
$$
hold for  $\forall (\mathbf{z}_1,\mathbf{y}_1),(\mathbf{z}_2,\mathbf{y}_2)\in \mathbb{R}^{d_g}\times \mathbb{R}^{d_2}$ . The gradient of the inner-level function  $g^{(k)}(\cdot)$  is  $L_{g}$ -Lipschitz continuous where  $L_{g} > 0$ , i.e.,
$$
\left\| \nabla g ^ {(k)} \left(\mathbf {x} _ {1}\right) - \nabla g ^ {(k)} \left(\mathbf {x} _ {2}\right) \right\| ^ {2} \leq L _ {g} ^ {2} \left\| \mathbf {x} _ {1} - \mathbf {x} _ {2} \right\| ^ {2}, \tag {6}
$$
holds for  $\forall \mathbf{x}_1, \mathbf{x}_2 \in \mathbb{R}^{d_1}$ .
Assumption 2. The second moment of  $\nabla_g f^{(k)}(\mathbf{z},\mathbf{y};\zeta)$  and  $\nabla g^{(k)}(\mathbf{x};\xi)$  satisfies:
$$
\mathbb {E} \left[ \| \nabla_ {g} f ^ {(k)} (\mathbf {z}, \mathbf {y}; \zeta) \| ^ {2} \right] \leq C _ {f} ^ {2}, \mathbb {E} \left[ \| \nabla g ^ {(k)} (\mathbf {x}; \xi) \| ^ {2} \right] \leq C _ {g} ^ {2}, \tag {7}
$$
for  $\forall (\mathbf{z},\mathbf{y})\in \mathbb{R}^{d_g}\times \mathbb{R}^{d_2}$  and  $\forall \mathbf{x}\in \mathbb{R}^{d_1}$ , where  $C_f > 0$  and  $C_g > 0$ . Meanwhile, the second moment of the full gradient is assumed to have the same upper bound.
Assumption 3. The variance of the stochastic gradient of the outer-level function  $f^{(k)}(\cdot, \cdot)$  satisfies:
$$
\mathbb {E} \left[ \| \nabla_ {g} f ^ {(k)} (\mathbf {z}, \mathbf {y}; \zeta) - \nabla_ {g} f ^ {(k)} (\mathbf {z}, \mathbf {y}) \| ^ {2} \right] \leq \sigma_ {f} ^ {2}, \mathbb {E} \left[ \| \nabla_ {\mathbf {y}} f ^ {(k)} (\mathbf {z}, \mathbf {y}; \zeta) - \nabla_ {\mathbf {y}} f ^ {(k)} (\mathbf {z}, \mathbf {y}) \| ^ {2} \right] \leq \sigma_ {f} ^ {2}, \tag {8}
$$
for  $\forall (\mathbf{z},\mathbf{y})\in \mathbb{R}^{d_g}\times \mathbb{R}^{d_2}$ , where  $\sigma_f > 0$ . Additionally, the variance of the stochastic gradient and the stochastic function value of  $g^{(k)}(\cdot)$  satisfies:
$$
\mathbb {E} \left[ \| \nabla g ^ {(k)} (\mathbf {x}; \xi) - \nabla g ^ {(k)} (\mathbf {x}) \| ^ {2} \right] \leq \sigma_ {g ^ {\prime}} ^ {2}, \mathbb {E} \left[ \| g ^ {(k)} (\mathbf {x}; \xi) - g ^ {(k)} (\mathbf {x}) \| ^ {2} \right] \leq \sigma_ {g} ^ {2}, \tag {9}
$$
for  $\forall \mathbf{x} \in \mathbb{R}^{d_1}$ , where  $\sigma_g > 0$  and  $\sigma_{g'} > 0$ .
Assumption 4. The outer-level function  $f^{(k)}(\mathbf{z},\mathbf{y})$  is  $\mu$ -strongly-concave with respect to  $\mathbf{y}$  for any fixed  $\mathbf{z}\in \mathbb{R}^{d_g}$ , where  $\mu >0$ , i.e.,
$$
f ^ {(k)} (\mathbf {z}, \mathbf {y} _ {1}) \leq f ^ {(k)} (\mathbf {z}, \mathbf {y} _ {2}) + \left\langle \nabla_ {y} f ^ {(k)} (\mathbf {z}, \mathbf {y} _ {2}), \mathbf {y} _ {1} - \mathbf {y} _ {2} \right\rangle - \frac {\mu}{2} \| \mathbf {y} _ {1} - \mathbf {y} _ {2} \| ^ {2}. \tag {10}
$$
Notation: Throughout this paper,  $\mathbf{a}_t^{(k)}$  denotes the variable of the  $k$ -th device in the  $t$ -th iteration and  $\bar{\mathbf{a}}_t = \frac{1}{K}\sum_{k=1}^K \mathbf{a}_t^{(k)}$  denotes the averaged variable across all devices, where  $a$  denotes any variables used in this paper.  $\mathbf{x}_*$  denotes the optimal solution.
In this section, we present the details of our algorithm for the federated compositional deep AUC maximization problem defined in Eq. (4).
Algorithm 1 LocalSCGDAM
Input:  $\mathbf{x}_0, \mathbf{y}_0, \eta \in (0,1), \gamma_x > 0, \gamma_y > 0, \beta_x > 0, \beta_y > 0, \alpha > 0, \alpha \eta \in (0,1), \beta_x \eta \in (0,1), \beta_y \eta \in (0,1)$ .
All workers conduct the steps below to update  $\mathbf{x},\mathbf{y}.\mathbf{x}_0^{(k)} = \mathbf{x}_0,\mathbf{y}_0^{(k)} = \mathbf{y}_0$
$$
\mathbf {h} _ {0} ^ {(k)} = g ^ {(k)} \left(\mathbf {x} _ {0} ^ {(k)}; \xi_ {0} ^ {(k)}\right),
$$
$$
\mathbf {u} _ {0} ^ {(k)} = \nabla g ^ {(k)} (\mathbf {x} _ {0} ^ {(k)}; \xi_ {0} ^ {(k)}) ^ {T} \nabla_ {g} f ^ {(k)} (\mathbf {h} _ {0} ^ {(k)}, \mathbf {y} _ {0} ^ {(k)}; \zeta_ {0} ^ {(k)}), \quad \mathbf {v} _ {0} ^ {(k)} = \nabla_ {y} f ^ {(k)} (\mathbf {h} _ {0} ^ {(k)}, \mathbf {y} _ {0} ^ {(k)}; \zeta_ {0} ^ {(k)}),
$$
1: for  $t = 0,\dots ,T - 1$  do
2: Update  $\mathbf{x}$  and  $\mathbf{y}$ :
$$
\mathbf {x} _ {t + 1} ^ {(k)} = \mathbf {x} _ {t} ^ {(k)} - \gamma_ {x} \eta \mathbf {u} _ {t} ^ {(k)}, \quad \mathbf {y} _ {t + 1} ^ {(k)} = \mathbf {y} _ {t} ^ {(k)} + \gamma_ {y} \eta \mathbf {v} _ {t} ^ {(k)},
$$
3: Estimate the inner-level function:
$$
\mathbf {h} _ {t + 1} ^ {(k)} = (1 - \alpha \eta) \mathbf {h} _ {t} ^ {(k)} + \alpha \eta g ^ {(k)} (\mathbf {x} _ {t + 1} ^ {(k)}; \xi_ {t + 1} ^ {(k)}),
$$
4: Update momentum:
$$
\mathbf {\tilde {u}} _ {t + 1} ^ {(k)} = (1 - \beta_ {x} \eta) \mathbf {u} _ {t} ^ {(k)} + \beta_ {x} \eta \nabla g ^ {(k)} (\mathbf {x} _ {t + 1} ^ {(k)}; \xi_ {t + 1} ^ {(k)}) ^ {T} \nabla_ {g} f ^ {(k)} (\mathbf {h} _ {t + 1} ^ {(k)}, \mathbf {y} _ {t + 1} ^ {(k)}; \zeta_ {t + 1} ^ {(k)}),
$$
$$
\mathbf {v} _ {t + 1} ^ {(k)} = (1 - \beta_ {y} \eta) \mathbf {v} _ {t} ^ {(k)} + \beta_ {y} \eta \nabla_ {y} f ^ {(k)} (\mathbf {h} _ {t + 1} ^ {(k)}, \mathbf {y} _ {t + 1} ^ {(k)}; \zeta_ {t + 1} ^ {(k)}),
$$
5: if  $\mathrm{mod}(t + 1,p) == 0$  then
6:  $\mathbf{h}_{t + 1}^{(k)} = \bar{\mathbf{h}}_{t + 1}\triangleq \frac{1}{K}\sum_{k' = 1}^{K}\mathbf{h}_{t + 1}^{(k')},$
$$
\mathbf {u} _ {t + 1} ^ {(k)} = \bar {\mathbf {u}} _ {t + 1} \triangleq \frac {1}{K} \sum_ {k ^ {\prime} = 1} ^ {K} \mathbf {u} _ {t + 1} ^ {(k ^ {\prime})}, \quad \mathbf {v} _ {t + 1} ^ {(k)} = \bar {\mathbf {v}} _ {t + 1} \triangleq \frac {1}{K} \sum_ {k ^ {\prime} = 1} ^ {K} \mathbf {v} _ {t + 1} ^ {(k ^ {\prime})},
$$
$$
\mathbf {x} _ {t + 1} ^ {(k)} = \bar {\mathbf {x}} _ {t + 1} \triangleq \frac {1}{K} \sum_ {k ^ {\prime} = 1} ^ {K} \mathbf {x} _ {t + 1} ^ {(k ^ {\prime})}, \quad \mathbf {y} _ {t + 1} ^ {(k)} = \bar {\mathbf {y}} _ {t + 1} \triangleq \frac {1}{K} \sum_ {k ^ {\prime} = 1} ^ {K} \mathbf {y} _ {t + 1} ^ {(k ^ {\prime})},
$$
7: end if
To optimize Eq. (4), we developed a novel local stochastic compositional gradient descent ascent with momentum algorithm, shown in Algorithm 1. Generally speaking, in the  $t$ -th iteration, we employ the local stochastic (compositional) gradient with momentum to update the local model parameters  $\mathbf{x}_t^{(k)}$  and  $\mathbf{y}_t^{(k)}$  on the  $k$ -th device. There exists an unique challenge when computing the local stochastic compositional gradient compared to traditional federated learning models. Specifically, as shown in Eq. (4), the objective function depends on the global inner-level function. However, it is not feasible to communicate the inner-level function in every iteration. To address this challenge, we propose to employ the local inner-level function to compute the stochastic compositional gradient at each iteration and then communicate the estimation of this function periodically to obtain the global inner-level function.