SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Input:  $T$ , step sizes  $\hat{c}, c, \eta$ ; momentum coefficient  $\alpha, \beta$ , the number of local updates  $Q$ , and mini-batch size  $b$  and initial mini-batch size  $B$ ;
2: Initialize:  $x_0^i = \bar{x}_0 = \frac{1}{N}\sum_{i=1}^{N}x_0^i, y_0^i = \bar{y}_0 = \frac{1}{N}\sum_{i=1}^{N}y_0^i$ ,  $u_1^i = \nabla_x f(x_0^i, y_0^i; \mathcal{B}_0^i)$  and  $v_1^i = \nabla_y f(x_0^i, y_0^i; \mathcal{B}_0^i)$  where  $|\mathcal{B}_0^i| = B$  are drawn from  $D_i$  for  $i \in [N]$ .
3: for  $t = 1,2,\ldots ,T$  do
4: for  $i = 1,2,\dots ,N$  do
5: if mod  $(t,Q) = 0$  then  
6: Sever Update:
7:  $u_{t}^{i} = \bar{u}_{t} = \frac{1}{N}\sum_{j = 1}^{N}u_{t}^{j}$
8:  $v_{t}^{i} = \bar{v}_{t} = \frac{1}{N}\sum_{j = 1}^{N}v_{t}^{j}$
9:  $x_{t}^{i} = \bar{x}_{t} = \frac{1}{N}\sum_{j = 1}^{N}(x_{t - 1}^{j} - \hat{c}\eta u_{t}^{j})$
0:  $y_{t}^{i} = \bar{y}_{t} = \frac{1}{N}\sum_{j = 1}^{N}(y_{t - 1}^{j} + c\eta v_{t}^{j})$
1: else
2:  $x_{t}^{i} = x_{t - 1}^{i} - \hat{c}\eta u_{t}^{i}$
3:  $y_{t}^{i} = y_{t - 1}^{i} + c\eta v_{t}^{i}$
4: end if
5: Draw mini-batch samples  $\mathcal{B}_t^i = \{\xi_i^j\}_{j=1}^b$  with  $|\mathcal{B}_t^i| = b$  from  $D_i$  locally
6:  $u_{t + 1}^{i} = \nabla_{x}f_{i}(x_{t}^{i},y_{t}^{i};\mathcal{B}_{t}^{i}) + (1 - \alpha)(u_{t}^{i} - \nabla_{x}f_{i}(x_{t - 1}^{i},y_{t - 1}^{i};\mathcal{B}_{t}^{i}))$
7:  $v_{t + 1}^{i} = \nabla_{y}f_{i}(x_{t}^{i},y_{t}^{i};\mathcal{B}_{t}^{i}) + (1 - \beta)(v_{t}^{i} - \nabla_{y}f_{i}(x_{t - 1}^{i},y_{t - 1}^{i};\mathcal{B}_{t}^{i}))$
8: end for
9: end for
20: Output:  $x$  and  $y$  chosen uniformly random from  $\{\bar{x}_t,\bar{y}_t\}_{t = 1}^T$
In FedSGDA-M, each client initializes the gradient estimators  $\{u_1^i,v_1^i\}$  with stochastic gradient as seen in line 2 of Algorithm 2. Following that, each client updates the model variables  $\{x_t^i,y_t^i\}$  locally as standard stochastic gradient descent ascent method (lines 12-13 of Algorithm 2). Compared with local momentum SGDA [50], the key difference is that clients utilize variance reduction gradient estimators  $\{u_t^i,v_t^i\}$ , which are constructed in lines 15-17 of Algorithm 2. For the update step of  $\{u_{t},v_{t}\}$ , the coefficients should satisfy  $0 < \alpha < 1$  and  $0 < \beta < 1$ . In every  $Q$  iteration, clients transmit model parameters and gradient estimators to the server, which computes  $\{\bar{x}_t,\bar{y}_t,\bar{u}_t,\bar{v}_t\}$ . Then the server sends averaged model variables and gradient estimators to each client to update the local variables, as shown in lines 5-10 of Algorithm 2.
Definition 3.10. According to Assumption 3.9, there exists at least one solution to the problem  $\max_y F(x,y)$  for any  $x$ . Here we define  $\Phi (x) = F(x,y^{*}(x)) = \max_y F(x,y)$ . We use  $\varepsilon$ -stationary point of  $\Phi (x)$ , i.e.  $\| \nabla \Phi (x)\| \leq \varepsilon$  as the convergence metric.
We know  $\Phi (x)$  is differentiable and  $(L + \kappa L)$ -smooth and  $y^{*}(\cdot)$  is  $\kappa$ -Lipschitz from [50]. Given that  $\nabla_yF(\bar{x}_t,y^* (x_t)) = 0$ , we have  $\nabla \Phi (\bar{x}_t) = \nabla_xF(\bar{x}_t,y^* (x_t)) + \nabla_yF(\bar{x}_t,y^* (x_t))\cdot \partial y^* (\bar{x}_t) = \nabla_xF(\bar{x}_t,y^* (x_t))$  which is widely used in the analysis of nonconvex-PL [10] and nonconvex-strongly-concave minimax optimization [62, 38]. Then we discuss the convergence analysis of FedSGDA-M. The proofs are provided in the supplementary materials.
Assumption 3.11. (Lipschitz Smoothness) Each component function  $f_{i}(x,y;\xi)$  has a  $L_{f}$ -Lipschitz gradient, i.e.,  $\forall x_{1},x_{2}$  and  $y_{1},y_{2}$ , we have
$$
\mathbb {E} \| \nabla f _ {i} \left(x _ {1}, y _ {1}; \xi\right) - \nabla f _ {i} \left(x _ {2}, y _ {2}; \xi\right) \| \leq L _ {f} \| \left(x _ {1}, y _ {1}\right) - \left(x _ {2}, y _ {2}\right) \| \tag {5}
$$
$F(x,y)$  has an  $L_{f}$ -Lipschitz gradient based on the convexity of norm and Assumption 3.11. We have
$$
\begin{array}{l} \| \nabla_ {x} F (x _ {1}, y _ {1}) - \nabla_ {x} F (x _ {2}, y _ {2}) \| = \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} \left[ \nabla_ {x} f _ {i} (x _ {1}, y _ {1}; \xi) - \nabla_ {x} f _ {i} (x _ {2}, y _ {2}; \xi) \right] \right\| \\ \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} \| \nabla_ {x} f _ {i} (x _ {1}, y _ {1}; \xi) - \nabla_ {x} f _ {i} (x _ {2}, y _ {2}; \xi) \| \\ \leq L _ {f} \| \left(x _ {1}, y _ {1}\right) - \left(x _ {2}, y _ {2}\right) \| \\ \end{array}
$$
In optimization analysis, it is standard to use the Assumption 3.11. Several widely used single-machine stochastic algorithms, such as SPIDER [13] and STORM [7], make use of this assumption. It is also used by numerous FL algorithms, including MIME [32] Fed-GLOMO [8], STEM [34] and FAFED [58].
Theorem 3.12. Suppose that sequence  $\{\bar{x}_t,\bar{y}_t\}_{t = 0}^T$  is generated from Algorithm 2. Under the above Assumptions (3.1,3.9,3.11), given  $\eta = \frac{1}{20QL}$ $\alpha = c_{1}\eta^{2},\beta = c_{2}\eta^{2},c_{1} = \frac{30L^{2}}{bN\kappa^{1 - \nu}},c_{2} = \frac{30L^{2}}{bN\kappa^{2 - 2\nu}},c = \frac{1}{6\kappa^{1 - \nu}},\hat{c} = \frac{1}{54\kappa^{3 - \nu}}$  where  $\nu \in [0,1]$  we have
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla \Phi (\bar {x} _ {t}) \| ^ {2} \leq \frac {2 [ \Phi (\bar {x} _ {0}) - \Phi (\bar {x} _ {T}) ]}{\hat {c} \eta T} + \frac {3 \sigma^ {2}}{\alpha T B N} + \frac {3 6 L _ {f} ^ {2} \sigma^ {2}}{\mu^ {2} \beta T B N} + \frac {1 2 L _ {f} ^ {2}}{c \eta \mu^ {2} T} [ \Phi (\bar {x} _ {0}) - F (\bar {x} _ {0}, \bar {y} _ {t}) ] \\ + \frac {6 \alpha \sigma^ {2}}{N b} + \frac {7 2 \beta \sigma^ {2} L _ {f} ^ {2}}{N b \mu^ {2}} + \left[ \frac {\sigma^ {2} \left(c _ {1} ^ {2} + c _ {2} ^ {2}\right)}{3 0 b L ^ {2}} + \frac {\zeta^ {2} \left(c _ {1} ^ {2} + c _ {2} ^ {2}\right)}{1 2 L ^ {2}} \right] \kappa^ {2} \eta^ {2} \\ \end{array}
$$
Corollary 3.13. By setting  $b = O(\kappa^{\nu})$  for  $\nu \in [0,1]$ ,  $c_{1} = \frac{30L^{2}}{bN\kappa^{1 - \nu}}$ ,  $c_{2} = \frac{30L^{2}}{bN\kappa^{2 - 2\nu}}$ ,  $c = \frac{1}{6\kappa^{1 - \nu}}$ ,  $\hat{c} = \frac{1}{54\kappa^{3 - \nu}}$ ,  $T = \kappa^{3 - \nu}T_{0}$ ,  $Q = \frac{T_{0}^{1 / 3}}{N^{2 / 3}}$ ,  $\eta = \frac{1}{20QL} = \frac{N^{2 / 3}}{20LT_{0}^{1 / 3}}$ ,  $B = \frac{T_0^{1 / 3}b_\kappa^{1 - \nu}}{N^{2 / 3}}$ , we have  $\alpha = c_{1}\eta^{2} = \frac{3N^{1 / 3}}{40T_{0}^{2 / 3}b\kappa^{1 - \nu}}$ ,  $\beta = c_{2}\eta^{2} = \frac{3N^{1 / 3}}{40T_{0}^{2 / 3}b\kappa^{2 - 2\nu}}$ .
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla \Phi (\bar {x} _ {t}) \| ^ {2} \leq \frac {2 1 6 0 L [ \Phi (\bar {x} _ {0}) - \Phi^ {*} ]}{(N T _ {0}) ^ {2 / 3}} + \frac {4 0 \sigma^ {2}}{\kappa^ {3 - \nu} (N T _ {0}) ^ {2 / 3}} + \frac {4 8 0 \sigma^ {2}}{\kappa^ {2} (N T _ {0}) ^ {2 / 3}} \\ + \frac {2 4 0 L _ {f}}{\left(N T _ {0}\right) ^ {2 / 3}} \left[ \Phi (\bar {x} _ {0}) - F (\bar {x} _ {0}, \bar {y} _ {0}) \right] + \frac {9 \sigma^ {2}}{2 0 b \kappa (N T _ {0}) ^ {2 / 3}} + \frac {2 7 \sigma^ {2}}{5 (N T _ {0}) ^ {2 / 3}} + \left[ \frac {3 \sigma^ {2}}{2 0 b} + \frac {1 5 \zeta^ {2}}{4 0} \right] \frac {1}{\left(N T _ {0}\right) ^ {2 / 3}} \\ \end{array}
$$
where  $\Phi^{*}$  is the optimal.
Remark 3.14. (Complexity) To make the  $\frac{1}{T} \sum_{t=0}^{T-1} \mathbb{E} \| \nabla \Phi(\bar{x}_t) \|^2 \leq \varepsilon^2$ , we get  $T_0 = O(N^{-1} \varepsilon^{-3})$  and  $T = O(\kappa^{3-\nu} N^{-1} \varepsilon^{-3})$ . Considering the  $b = \kappa^\nu$ , we have communication complexity  $\frac{T}{Q} = \kappa^{3-\nu} (NT_0)^{2/3} = \kappa^{3-\nu} \varepsilon^{-2}$  and sample complexity  $bT = O(\kappa^3 N^{-1} \varepsilon^{-3})$ . When  $\nu = 1, b = \kappa$ , communication Complexity  $\frac{T}{Q} = \kappa^2 \varepsilon^{-2}$  for finding an  $\varepsilon$ -stationary point. Sample complexity  $bT = O(\kappa^3 N^{-1} \varepsilon^{-3})$  matches the complexity result achieved by the single-machine algorithms, such as SREDA and Acc-MDA in [42, 29] but we do not require a large batch size b compared with these algorithms. And  $O(\kappa^3 N^{-1} \varepsilon^{-3})$  also exhibits a linear speed-up compared with the aforementioned single-machine algorithms.
3.3 Nonconvex-Strongly-Concave (NC-SC) Problems
Assumption 3.15. Each local function function  $f_{i}(x,y)$  is  $\mu$ -strongly concave in  $y\in \mathcal{V}$ , i.e.,  $\forall x\in \mathcal{X}$  and  $y_{1},y_{2}\in \mathcal{V}$ , we have
$$
\left\| \nabla_ {y} f _ {i} (x, y _ {1}) - \nabla_ {y} f _ {i} (x, y _ {2}) \right\| \geq \mu \| y _ {1} - y _ {2} \|
$$
When the function  $F(x,y)$  is strongly concave in  $y \in \mathcal{V}$ , there exists a unique solution to the problem  $\max_{y \in \mathcal{V}} f(x,y)$  for any  $x$ . Since PL condition is weaker than strong concavity, the convergence result of FedSGDA-M in Algorithm 2 under NC-PL also apply to NC-SC problem and FedSGDA-M has the sample complexity of  $O(\kappa^3 n^{-1}\varepsilon^{-3})$  and the communication complexity of  $O(\kappa^2\varepsilon^{-2})$  under nonconvex-strongly-concave setting.
We conduct experiments on AUROC maximization and fair classification tasks to verify the efficiency of our algorithms under nonconvex-strongly-concave and nonconvex-concave settings. Experiments are completed on the computer cluster with AMD EPYC 7513 Processors and NVIDIA RTX A6000. The code is available *
*https://github.com/xidongwu/Federated-Minimax-and-Conditional-Stochastic-Optimization
Datasets and Models: We test the performance of algorithms on three typical datasets: Fashion-MNIST dataset, CIFAR-10 dataset and Tiny-/ImageNet. Fashion-MNIST dataset has  $70,00028 \times 28$  gray images (10 categories, 60,000 training images and 10,000 testing images). CIFAR-10 dataset consists of 50,000 training images and 10,000 testing images. Each image is the  $3 \times 32 \times 32$  arrays of color image. Tiny-/ImageNet dataset has 200 classes of  $(64 \times 64)$  colored images and each class has 500 training images, 50 validation images, and 50 test images. For Fashion MNIST and Cifar10 datasets, we choose convolutional neural network from [28] (The details are shown in the supplementary materials). For Tiny-/ImageNet, we choose ResNet-18 [24] as the model.
4.1 Fair Classification
First, we follow [50, 28] and train fair classification networks by minimizing the maximum loss over different categories.
$$
\min  _ {x} \max  _ {y \in \mathcal {Y}} \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {c = 1} ^ {C} y _ {c} \mathcal {L} _ {c} ^ {i} (x) \quad \text {s . t .} \mathcal {Y} = \left\{y \mid y _ {i} \geq 0, \sum_ {i = 1} ^ {C} y _ {i} = 1 \right\}
$$
where  $\mathcal{L}_c^i$  denotes the cross-entropy loss functions corresponding to the class  $c$  in C different classes and  $x$  denotes the CNN model parameters. Clearly, the problem in (6) is nonConvex in x (deep model parameters), and Concave in y. We compare our algorithm (FedSGDA+) and local SGDA+ with varying model, datasets, local update numbers, step size. Although the constraint is not considered in the theoretical analysis of local SGDA+ and FedSGDA+, FedSGDA+ still shows better performance compared with local SGDA+.
The network has 20 clients. The datasets are partitioned into disjoint sets across all clients and each client holds part of the data from all the classes [34]. We initialize the reseed18 with pretrained weights in PyTorch. In experiments, we run grid search for step size, and choose the step size for primal variable in the set  $\{0.01, 0.03, 0.05, 0.1, 0.3\}$  and that for dual variable in the set  $\{0.001, 0.01, 0.1\}$ . We choose the global step size in the set  $\{0.1, 0.5, 1, 1.5, 2\}$ . The batch-size  $b$  is in 50 and the inner loop number  $Q$  is selected from  $\{20, 50, 100\}$ , The outer loop number  $S$  is selected from  $\{1, 5, 10\}$  for FedSGDA+ and  $\{1, 5, 10, Q\}$  for local SGDA+.
Figure 1 shows that FedSGDA+ has a better convergence rate than local SGDA+. This confirms that our algorithm can effectively accelerate SGDA by using the structure of federated learning. Due to the page limitation, the ablation analysis of step size is presented in the supplementary materials.
(a) Fashion-MNIST
(c) Tiny ImageNet
Figure 1: Test Accuracy vs the number of communication rounds during the training phase.
4.2 AUROC Maximization
[39] showed the AUROC maximization problem could be reformulated as the non-convex-strongly-concave minimax optimization, as below:
$$
\min  _ {\substack {\mathbf {m} \in \mathbb {R} ^ {d} \\ (a, b) \in \mathbb {R} ^ {2}}} \max  _ {w \in \mathbb {R}} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\xi_ {i} \sim \mathcal {D} _ {i}} \left[ f _ {i} (\mathbf {m}, a, b, w; \xi) \right] \tag{6}
$$
$$
\begin{array}{l} f (\mathbf {m}, a, b, w; \xi) = (1 - p) (h (\mathbf {m}; \mathbf {x}) - a) ^ {2} \mathbb {I} _ {[ y = 1 ]} + p (h (\mathbf {m}; \mathbf {x}) - b) ^ {2} \mathbb {I} _ {[ y = - 1 ]} \\ + 2 (1 + w) \left[ p h (\mathbf {m}; \mathbf {x}) \mathbb {I} _ {[ y = - 1 ]} - (1 - p) h (\mathbf {m}, \mathbf {x}) \mathbb {I} _ {[ y = 1 ]} \right] - p (1 - p) w ^ {2} \\ \end{array}
$$
\(\xi_{i} = (\mathbf{x},y)\sim \mathcal{D}_{i}\) denotes a random data point and \(\mathbf{x}\) represents the data features and \(y\in \mathcal{V} = \{-1, + 1\}\) is the label. \(h(\mathbf{m};\mathbf{x})\) denotes the prediction score of the data point \(\mathbf{x}\) calculated by a model with parameter \(\mathbf{m}\). \(p = Pr(y = 1) = \mathbb{E}_y[\mathbb{I}_{[y = 1]}\] denotes the prior probability of the positive data.
(b) CIFAR-10
Figure 2: AUROC scores on the test datasets vs the number of communication rounds during the training phase.
(c) Tiny ImageNet
Following [19, 67], we constructed the imbalanced binary-class versions of datasets as follow: firstly, the first half of the classes (0 - 4) in the original Fashion-MNIST, CIFAR10 and classes (0 - 99) in Tiny-ImageNet datasets are converted into the negative class, and the rest half of classes are considered to be a positive class.  $80\%$  of the negative data points are randomly dropped in each dataset. Then the datasets are evenly divided into disjoint sets across 16 clients. In this case, each client share completely different imbalanced datasets. In the experiment, we use xavier normal initialization to deep models.
We compare our algorithm (i.e., FedSGDA-M) with local SGDA [10, 50], CODA+ [19, 67], Momentum SGDA [50], CODASCA [67] and SAGDA [63] as baselines in AUROC maximization. In experiments, we carefully tune hyperparameters for all methods. We run a grid search for step size, and choose the step size for the primal variable in the set  $\{0.001, 0.005, 0.01\}$  and that for dual variable in the set  $\{0.0001, 0.001, 0.01\}$ . We choose the global step size from  $\{0.9, 1, 1.5, 2\}$  for CODASCA and SAGDA. We choose the momentum parameter in Local Momentum SGDA in the set  $\{0.1, 0.5, 0.9\}$ . The  $\alpha$  and  $\beta$  in FedSGDA-M are chosen from  $\{0.1, 0.5, 0.9\}$ . The batch-size  $b$  is 50 and the inner loop number  $Q \in \{10, 20, 50\}$ .
As shown in Figure 2, we compare the performance of FedSGDA-M and other baseline methods against the number of communication rounds. Figure 2 shows that our algorithms consistently outperform the other baseline algorithms on testing datasets, which validates the efficacy of our algorithms. Due to space limitation, other test results are provided in the supplementary materials.
**Limitation.** Minimax optimization has many applications and a more comprehensive discussion of our proposed algorithms on these tasks will be a future study because the theoretical analysis is the main contribution of this paper.
In this paper, we study a class of federated nonconvex minimax optimization problems (1). We consider the three most common settings (NC-SC, NC-PL, NC-C). Under the NC-C setting, we propose FedSGDA+ and prove it has the best communication complexity of  $O(\varepsilon^{-6})$ . It also achieves a linear speedup to the number of clients. Under NC-PL and NC-PL settings, we propose FedSGDA-M with variance reduction technique and we prove that our algorithm (FedSGDA-M) has the best sample complexity ( $O(\kappa^3 n^{-1}\varepsilon^{-3})$ ) and the best sample communication complexity ( $O(\kappa^2\varepsilon^{-2})$ ). We prove that FedSGDA also enjoys linear speedup with respect to the number of clients. Therefore, we reduce the existing complexity results for the most common nonconvex minimax optimization problems under the federated learning setting.
[61] Yihan Wu, Aleksandar Bojchevski, and Heng Huang. Adversarial weight perturbation improves generalization in graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10417-10425, 2023.
[62] Wenhan Xian, Feihu Huang, Yanfu Zhang, and Heng Huang. A faster decentralized algorithm for nonconvex minimax problems. Advances in Neural Information Processing Systems, 34:25865-25877, 2021.
[63] Haibo Yang, Zhuqing Liu, Xin Zhang, and Jia Liu. Sagda: Achieving  $\mathcal{O}(\epsilon^{-2})$  communication complexity in federated min-max learning. arXiv preprint arXiv:2210.00611, 2022.
[64] Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. Advances in Neural Information Processing Systems, 33: 1153-1165, 2020.
[65] Junchi Yang, Xiang Li, and Niao He. Nest your adaptive algorithm for parameter-agnostic nonconvex minimax optimization. arXiv preprint arXiv:2206.00743, 2022.
[66] Junchi Yang, Antonio Orvieto, Aurelien Lucchi, and Niao He. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics, pages 5485-5517. PMLR, 2022.
[67] Zhuoning Yuan, Zhishuai Guo, Yi Xu, Yiming Ying, and Tianbao Yang. Federated deep auc maximization for heterogeneous data with a constant communication complexity. In International Conference on Machine Learning, pages 12219-12229. PMLR, 2021.
[68] Xin Zhang, Zhuqing Liu, Jia Liu, Zhengyuan Zhu, and Songtao Lu. Taming communication and sample complexities in decentralized policy evaluation for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:18825-18838, 2021.
[69] Xuan Zhang, Necdet Serhat Aybat, and Mert Gurbuzbalaban. Sapd+: An accelerated stochastic method for nonconvex-concave minimax problems. arXiv preprint arXiv:2205.15084, 2022.
[70] Hu Zhengmian, Wu Xidong, and Huang Heng. Beyond lipschitz smoothness: A tighter analysis for nonconvex optimization. In International Conference on Machine Learning (ICML), 2023.
[71] Hanhan Zhou, Tian Lan, Guru Prasadh Venkataramani, and Wenbo Ding. Federated learning with online adaptive heterogeneous local models. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022), 2022.
[72] Hanhan Zhou, Tian Lan, Guru Prasadh Venkataramani, and Wenbo Ding. Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction. In Advances in Neural Information Processing Systems, volume 36, 2023.