DeepAFL: Deep Analytic Federated Learning
Authors: Jianheng Tang
Yajiang Huang
Kejia Fan
Feijiang Han
Jiaxu Li
Jinfeng Xu
Run He
Anfeng Liu
Houbing Herbert Song
Huiping Zhuang
Yunhuai Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Data Heterogeneity
Representation Learning
Analytic Learning
Continual Learning
Lifelong Learning
Incremental Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: In this paper, we propose our DeepAFL to achieve representation learning while preserving heterogeneity invariance in FL via analytical (i.e., closed-form) solutions.

Abstract: Federated Learning (FL) is a popular distributed learning paradigm to break down data silo. Traditional FL approaches largely rely on gradient-based updates, facing significant issues about heterogeneity, scalability, convergence, and overhead, etc. Recently, some analytic-learning-based work has attempted to handle these issues by eliminating gradient-based updates via analytical (i.e., closed-form) solutions. Despite achieving superior invariance to data heterogeneity, these approaches are fundamentally limited by their single-layer linear model with a frozen pre-trained backbone. As a result, they can only achieve suboptimal performance due to their lack of representation learning capabilities. In this paper, to enable representable analytic models while preserving the ideal invariance to data heterogeneity for FL, we propose our Deep Analytic Federated Learning approach, named DeepAFL. Drawing inspiration from the great success of ResNet in gradient-based learning, we design gradient-free residual blocks in our DeepAFL with analytical solutions. We introduce an efficient layer-wise protocol for training our deep analytic models layer by layer in FL through least squares. Both theoretical analyses and empirical evaluations validate our DeepAFL's superior performance with its dual advantages in heterogeneity invariance and representation learning, outperforming state-of-the-art baselines by up to 5.68%-8.42% across three benchmark datasets. Related code is available at https://github.com/tangent-heng/DeepAFL.

Published as a conference paper at ICLR 2026
DEEPAFL: DEEP ANALYTIC FEDERATED LEARNING
Jianheng Tang!., Yajiang Huang?? 2, Kejia Fan', Feijiang Han', Jiaxu Jiaxu Li,, Jinfeng Xu*, Run He Anfeng Liu?, Houbing Herbert Song", Huiping Zhuang" $, Yunhuai Liu !Peking University, PRC: "Central South University, PRC; University Pennsylvania, USA: *The University of Hong Kong, PRC; South China University of Technology, PRC; University of Maryland, Baltimore County (UMBC), USA;
"Key Lab of High Confidence Software Technologies (PKU), Ministry Education, PRC
ABSTRACT
Federated Learning (FL.) is a popular distributed learning paradigm to break down data silo. Traditional Fl. approaches largely rely gradient-based based updates, facing significant issues about heterogeneity, scalability, convergence, and overhead, etc. Recently, some analytic-learning- senc-llearnnn-basee work has attempted handle these issues by eliminating gradient-basee updates via analytical (i.e., closed-form) solutions. Despite achieving superior invariance data heterogeneity, these approaches are fundamentally limited by their single-layer linear model with a frozen pre-tran pre-trained backbone. As a result, they can only achieve suboptimal performance due to their lack of representation learning capabilities. In this paper, to enable representable analytic models while preserving the ideal invariance data heterogeneity for FL, we propose Deep Analytic Federated Learning approach, named DeepAFL. Drawing inspiration from the great success of ResNet gradient- based learning, we design gradient-fire free residual blocks in DeepAFL with analytical solutions. We introduce an efficient layer-wist wise protocol for training our deep analytic models layer by layer in FL through least squares. Both theoretical analyses and empirical evaluations validate our Deep AFL's L'ss superior performance with its dual advantages in heterogeneity invariance and representation learning, outperforming state-of- the-art baselines by up to 5.68% 8 4.2%% across three benchmark datasets. Related code is available http::t:////mm
INTRODUCTION
Federated Learning (FL) has emerged prominent paradigm that enables distributed learning to
break down data silos (Fan et al. 2025c Yang et al. [023] Guo et al., 2025 Li et al. (025c). The objective FL to allow a group of chents collaboratively train powerful and robust global 2024b) The field of FL. has model, while preserving their data privacy (Ren et al., 2025 Liu et al.
seen substantial growth across a wide range of applications (Zhou et 2024 Wu al., 2025).
Traditional FL methods rely on gradient- based optimization paradigm, exemplified by the classic FedAvg (McMahan et al. al. 2017) and its following variants (Li et al., /202lb/Yang et al. [2024). These methods typically necessitate iterative optimization processes to achieve convergence (Wang et al.)
2024 Tan et al. 2022a). Yet, these gradient-basee techniques are widely acknowledged to suffer from several major challenges (Ye al. 2023a Chai et al. 2024 He et al. (025), follows.
(1) Heterogeneity Issues:
The data across clients are often Not Independently and Identically Distributed (Non-IID), which can severely impact model performance and convergence. Scalability Issues: As the number clients increases, especially to a large scale (e.g., thousands of clients), the FL systems can experience substantial performance degradation. Convergence Issues: The FL methods may struggle to converge within limited aggregation rounds, particularly in challenging scenarios of non-IID data or large-scale clients.
(4) Overhead Issues: The overall FL process incurs significant overhead from multi-epoch training on each client and multi-round model aggregation across clients for convergence.
Many researchers have come to realize that the aforementioned challenges in FL are fundamentally rooted in the long- standing reliance on gradient- based updates, which are inherently sensitive and
"orresponniig Authors.

---

Published as a conference paper at ICLR 2026
Computation
Random Projection
Activation Function
Add Function
(a)
(c)
(d)
[e] Rendrem Projection
Random Projectinn& Activtiio Functions
We+1
Wes1
DeepAFL (Oun)
++1
11+1
Y(+1
+++
4
F,01+1
W
Ve
Accuray (%)
6.3%
P
6.1%
(b)
We+1
+++1
Y,+1
Fe
2.1%
2.1%
O
1.3%
W
4 W
Y:
Number of Layers
Figure 1: Comparing our proposed DeepAFL with the naive approaches for representation learning- (a) Illustration of multi-analytic layers with random projections. (b) Illustration of analy-analytic layers with random projections & activation functions. (c) Illustration of our proposed AFe (d) Among these, our Ael exhibits the best performance improvements with increasing layers. For all these approaches, the activation function is GELU, the random projection dimension is 1024, the pre-trained model is the ResNet-18 used in AFL, and the evaluation dataset is the CIFAR- 100.
costly in the distributed FL scenario Ye al. perspective, existing gradient-base methods can only superficially alleviate these issues rather than fundamentally address them. Therefore, natural and promising avenue to fundamentally address these gradient-related issues is to eliminate gradient-based updates entirely (He et al. 2025).
2023a Fani et al. 2024 He al. 2025) From this
As prominent gradient-free technique, analytic learning has exhibited great promise by achieving analytical (closed-form) solutions through the least squares (Lai et al.] 2025) Fan et al.] (025a). To introduce this technique into the FL, Analytic Federated Learning (AFL) has been proposed with the help of pre-trained models (He et al., 2025). The core idea of AFL is to leverage frozen pre- trained models (e.g.. foundation models) for feature extraction on input data. Based on the extracted features, AFL can then build a Ainge-llaer linear model, which has an analytical solution. Through its specific protocols on local training and global aggregation, AFL achieves ideal invariance to data heterogeneity, as its final aggregated global model is equivalent to the centralized analytical solution.
Despite achieving Srate-Of-The-Ae Art (SOTA) performance with its unique invariance to data hetero- geneity, AFI. is significantly limited by its single-layer linear model. On the one hand, this analytic model is foundational to AFL, as its simplicity with a convex optimization objective is a prerequisite for deriving an analytical solution. On the other hand, it also severely limits AFL's application, it can only learn a linear mapping from the frozen backbone's ' features to the final output, thus funda- mentally failing to perform representation learning within the FL systems. Consequently, due to the limited learning capacity of the linear analytic model, AFL is prone to underfitting, especially when the backbone itself is lightweight. Additionally, even if the backbone itself possesses sufficient feature extraction capabilities, the linear separability of its output features may still be insufficient.
Thus, an interesting and challenging problem arises: can we deepen AFL's analytic model enable its representation learning capabilities while simultaneously preserving its analytical solutions for invariance data heterogeneity? A quite naive approach using a random projection after each analytic model to construct the input features for the next layer, as displayed in Figure Yet, this approach brings almost no performance improvement, as all its layers are simply linear mappings. Then, we attempt to introduce an activation function after each feature to provide non-linearity, as illustrated in Figure bb). Based on the evaluation results presented in Figure [d), the deep activated random projections can, to some extent, enrich the feature representations. However, as the number of layers increases, the deepening approach (b) struggles to improve the performance through representation learning. Therefore, these naive approaches fall far short of our requirements.
Drawing inspiration from the great success of ResNet in gradient- based learning (He et al. 2016), in this paper, adopt the similar skip connections to boost the representation of the analytic layers. Specifically, we model this representation learning as o, = Φ,- g ( (Φ,-1), where (-()) represents nonlinear feature transformation. In gradient-based learning, the residual blocks go (p,1) can
2

---

Published as a conference paper at ICLR 2026
be easily learned, as Stochastic Gradient Descent (SGD) with backpropagation automatically adjust the network weights to learn appropriate representations. Given that analytic models preclude gradient-based based updates via backpropagation, key technical challenge lies in how to effectively learn the residual blocks gt. (+,-) for meaningful boosting within the framework of analytic learning.
To enable representable analytic models while preserving their ideal invariance to data heterogeneity for FL., in this paper, we propose our Deep Analytic Federated Learning approach, named DeepAFL. Inherited from AFL, our DeepAFL also employs pre-trained backbone for initial feature extraction. Then, we propose to randomly project and activate these features to form the zero-layer features Po. which can yield an immediate performance gain of about 2.1 %2 as shown Figure [d). After this setup. our DeepAFL continuously refines the features o. layer by layer. Specifically, to obtain the residual block ge (+: for layer I, we first build nonlinear representation F, F from 4;,1 using a random projection layer with an activation function, as illustrated in Figure Subsequently, we can obtain Ir ( = F 1-1 by introducing learnable transformation S2, to adjust and scale the representation F,- -- We derive the optimal analytical solution for S2 via sandwiched least squares. As shown in Figure our DeepAFL exhibits a desired capability of deep representation learning, as its performance continuously and markedly improves with increasing number of layers.
Our primary contributions are summarized as follows:
Conceptually, we propose our DecpAFL, novel approach that can achieve gradient-free representation learning while preserving ideal invariance to data heterogeneity in FL..
Technically, develop efficient layer-wise protocol for learning deep analytic models via least squares. In our DeepAFL. the clients only need to conduct lightweight forward- propagation computations, so that the server can aggregate the global models layer by layer.
Theoretically, demonstrate two ideal properties of our DeepAFL: its invariance to data heterogeneity and its capability of representation learning. To the best of our knowledge, our DeepAFL represents the first achieve both of these ideal properties simultaneously.
Experimentally, we provide extensive evaluations on three benchmark datasets to show the superiority of our DeepAFL, which outperforms SOTA baselines by up to 5.68%-8.44 -.44%, thanks to its dual advantages in heterogeneity invariance and representation learning.
2
RELATED WORK
2.1 FEDERATED LEARNING
As prominent distributed learning paradigm, FL allows multiple clients to collaboratively train global model to break down data silos (Li et al. 2025bb Yang et al. 2023). Existing FL techniques largely derived from FedAvg (McMahan al. 2017 and rely on gradient-based based optimization. Despite advancements in FL, their reliance on gradients also causes inherent issues about overhead,
convergence, heterogeneity, and scalability (Ye et al., 2023a Chai al. 2024 He
2025).
Recently, AFL has introduced a new and promising wave that fundamentally handles these issues by avoiding gradient- based updates via analytic learning (He al. 2025). Based on it, some subsequent research has extended this concept to personalized FL and federated continual learning (Fan et al.) 2025b Tang et al. 20055. achieving excellent performance. Nevertheless, a key limitation of existing analytic-learning learning-based based FI. approaches is their lack of representation learning capability.
2.2 ANALYTIC LEARNING
2024), also known as pseudoinverse learning (Cline 1964 Guo et al. 2001), ). has emerged as a popular gradient- free technique to address gradient-relater
Analytic learning Zhuang et al., 2022 2023
issues (Toh 2018; Lanthaler & Nelsen 2023 Prabhu et al.] 2024 Zozoulenko et al. 2025 Fan
et al. 2025a Bolager et al. 2023). core idea is to directly derive analytical (i.e., closed-for form)
solutions using least squares, to eliminate gradient- based updates (Li et al.) 2025a Peng et al., 2025)
Despite the extensive superiority shown by analytic learning, current approaches are largely limited by their single-layer linear models, which sacrifice the capabilities of deep representation learning. On the other hand, achieving representable deep analytic models while preserving their closed-fore form solutions stands great challenge, especially in the distributed scenario of FL. Drawing inspiration from the success of ResNet (He et al.] (0166) in gradient- bassed learning. we fundamentally solve this problem by carefully designing gradient-free freee residual blocks with closed-form solutions.
3

---

Published as a conference paper at ICLR 2026
3 OUR PROPOSED DEEPAFL
In this section, we claborate in detail on our proposed DeepAFL. For clarity, we consider the typical FL setting involving one server and K clients. Moreover, to align with existing research, we use the most prevalent task in FL. image recognition, as the prime example to describe the workflow of our
DeepAFL (Yu et al.) 2024 2025 Miao al.) [202 He et al. 2025). Each client's local dataset is denoted as D [X+
where X E R xdc and Y E R-NexC represent the N₂ local samples and their corresponding labels, respectively. Here, du x dw x de denotes the 3 dimensions (height, width, and channels) of the input images, and C denotes the number of output classes.
The objective of our proposed DeepA FL. is construct deep residual analytic network comprising T' layers for deriving the features F and the corresponding global classifier Wr, in a distributed and gradient-free manner. In Section 3.T we detail the motivation and insight of the proposed deep analytic learning in our DeepAFL. which is described from a centralized perspective for clarity. In Section 3.2 we elaborate on the specific workflow and implementation of our DeepAFL within the data-distributed FL scenario. In Section 3.3 we provide theoretical analyses of our DeepAFL for its validity, privacy, and efficiency, particularly showing its dual ideal properties.
3.1 DEEP RESIDUAL ANALYTIC LEARNING OF OUR DEEPAFL
Here, we aim to introduce the key motivation and insights behind the proposed deep residual analytic model in our AeL. For clarity and brevity, we first adopt a centralized perspective and consider a full dataset D {X, Y} to construct the T-layer deep residual analytic network. Specifically, progressively learn better representations and refine the features { 17 layer by layer. Based on the features 4 for each layer I, we construct a corresponding analytic classifier W..
2025), we employ a pre-trained backbone for our initial First of all, inherited from AFL (He et al. feature extraction from X to obtain X E IR Nxxx via (1. This has been widely adopted as common
practice many recent studies of FL (Nguyen et al. 2023] Piao et al. 2024 Yu et al.) 2024 2025).
(I) where Backboone-)) represents the pre-trained backbone with frozen parameters O Based on this initial feature extraction of AFL, we further incorporate the activated random projection to boost the features' representation, thereby forming the zero-layer features Po E R.Nxd as follows.
X = BBcckboneeX X, +).
D (XA); where a-)) represents the activation function, and A E Rd is the random projection matrix. This activated random projection can increase the feature dimension to a suitable size, boosting its linear separability,
widely validated in existing studies (Cover) 2006 Zhuang et al. 2022 2024 Zhang et al.) 2025). To progressively learn richer representations, we attempt to deepen the network. Yet, naive approaches deepening the network fail to achieve effectively, as shown in Figure [
Drawing inspiration from the great success of ResNet in gradient-based learning (He et al. 2016),
we adopt the similar skip connections to boost the representations of the analytic layers. Specifically, we model this representation learning process as the feature updating formula in (3):
(3) Here, gc(-) represents the residual block as a nonlinear feature transformation. We will specify the detailed gradient-fred free design of the residual block within our DeepAFL in (6) later. Built upon the obtained feature matrix 4; for each layer I, we can construct a corresponding analytic classifier W.. Specifically, consistent with other existing analytic-learing--ase optimization objective for the analytic classifier W, can be formulated as follows.
+ Φ.-1 + Jt 1)). Vt E [1,7]-
approaches (He al. 2025). the
W. arg min
|| L w + I|| Vt E [O,T].
(4)
W
where \ denotes the regularization parameter and I|: represents the Frobenius norm. Since cross- entropy loss does not admit a closed-form solution, we use the Mean Squared Error (MSE) loss here.
In fact, the MSE loss is widely adopted in analytic learning (Zhuang et al. 2022 2023
2024 and
can achieve performance comparable to that of the cross-entropy loss (Hui & Belkin 2021). Using the least squares method, we can derive the optimal analytical solution to the objective (4) as given by (5) The detailed proof of the analytical solution (5) is provided in Lemma of Section 3.3
W, (0) t. AI)
$ Y A1 E [0, 7].
4

---

Published as a conference paper at ICLR 2026
After obtaining the analytic classifier Wt, E. then proceed with deep residual representation learn- ing to update the (t + -ttthleye features. Based on (3), the key to this problem is the gradient- free design of the residual blocks, which possess analytical solutions with the properties of stochasticity, nonlinearity, and learnability. Thus, in our DeepAFI., the residual block instantiated as follows:
g++1 (+,) = ( F, VI E (0,
(6)
Here, B, E Rdd represents the random projection matrix for stochasticity, akin that provided by SGD in gradient-based learning. Meanwhile, (-) means the activation function for nonlinearity, a component that widely demonstrated be essential in deep representation learning. Moreover, S E de represents the trainable transformation matrix for providing learnability. In (6). we further define F, = a $ B,) as the hidden random feature, thereby isolating the trainable 1 In we will provide extensive ablation studies to show the effectiveness of these components.
Section 4
Then, we focus on learning the optimal 24+1 for effective representation boosting within the residual block. Here, our objective can be written as, given fixed classifier W, from the previous layer, to minimize the empirical risk by optimizing the new features Pi+1 = 4 F, as follows:
su+1 = arg min 52
Y
( + F,
F + >|
arg min RR FOSWW 12 > 9
(),
Here, we define the residual of the current layer's classification as R, = Y ( W,) for notational convenience. The optimization objective (7) can be seen special case of generalized Sylvester matrix equations (Wu et al., 2008; Ding et al., 2008 Duan 2015 Zozoulenko et al. 2025). with the unknown variable 52 being sandwiched between two known ones, F, and W.. This structure enables the derivation of an analytical solution in (8). We term this as the sandwiched least squares problem, and provide detailed proof of the analytical solution (8) in Lemma of Section 3.3
12 VV[(V, R,F.R.WWW... W, U,) @ (71 diag(A.)) @ (iag(A)))
Vt E (0,T).
where F, F, V V,A,V and W, W. U,A,U, spectral decompositions, while @ and & represent element-wist division and outer product, respectively. Once we obtain the transformation matrix 12++1, we can compute the next-layer features via the following recursive formula.
4 1 4b F, Vi E [0, T).
In summary, the construction of our deep residual analytic network proceeds in wise manner, involving the alternating derivation of We and S2,-1 analytical solutions (5) and (8). respectively. Specifically, the solution procedure follows the sequence Wo + Ωn + W beginning with the zero-layer classifier Wo o and concluding with the final-layer classifier Wr. For further clarity, we provide detailed illustrations of our DeepAFL's formulations in Figures I6 16 Appendix F. Based on the above centralized process, we will then elaborate on the specific workflow and implementation of our DeepAFL for the distribbut FL scenario in Section 3.2
22 r Wr,
3.2 FEDERATED IMPLEMENTATION OF OUR DEEPAFL.
In this subsection, we display the federated implementation of our DeepAFL for the data-distributed FL scenario, where each client owns its local dataset D {X* Y*). The overall implementation workflow of our DeepAFL still follows the layer-wise procedure that is described in Section Yet, in the FL setting, the analytical computation of the global weights W, and 244111 requires additional aggregation of all clients' local knowledge to update the global knowledge.
First of all, each client ls performs feature extraction on its local data X to obtain its local zero-layer feature matrix & E R xde similar to (1 and 2 as follows.
k 0 a (Backbone(X* O)A)
(10)
Then, for each layert e [J, each client utilizes its features & and labels Y compute its local Matrix G and Label Cross- Correlation Matrix H/. as follows.
Feature
G ) T/ H ( Y*
(11)
Then, the server needs to aggregate ! and H from the clients through 12 This process can
be implemented via existing Secure Aggregation Protocols (Bonawitz et al. 2016
2017
So et al.)
2023). We provide related privacy analyses in Section 3.3
G c1:K = > k=1
K
G
H
H
H
(12)
k=1
5

---

Published as a conference paper at ICLR 2026
Once obtaining G and H the server can derive the global classifier W, via 13 and distribute it to all clients then. As shown in Section 3.3 regardless of how the data are distributed across clients, the global analytic classifier W, obtained through (13) is exactly identical to the optimal solution of centralized analytic learning on the full dataset, thus achieving invariance to data heterogeneity.
W. [
1:K AI
(
( Y
(13)
= (G) I. + AI) H K
Then, each client is required to compute its hidden random features F., and local residual matrix R based on the previously obtained features T and classifiers W,, as follows.
F = o B.), R Y (Φ.
(14)
Subsequently, each client utilizes the hidden random features F and local residual matrix R, get its local Hidden Auto- uuto-Correlation Matrix IIŞ and Residual Wvoss-Correlation Matrix YX via 15
I = (F)) TF. r = (FF) RR.
(15)
Then, akin to G and H. the server aggregates IIA and r. from the clients via (16).
II m- I
r!
(16)
Once obtaining I!..K and riKK K the server can derive S4+1 via 17 and distribute it to all clients.
Su+ Vi[(VV (FI-K) 'R IK W. @ (y1 + diag (A,) @ diag(A) (ang(A)))) JU?
(17)
=V.((V r w U.) @ (<1 + (Ai) @ diag(A)))
where V,, AP, U. and A, are obtained by spectral decompositions, follows.
IT::K (F!:K) F. V.A.V,, W,W, U,A,UT
(18)
After receiving 84 from the server, each client thus update the next-layer features t
#++ = x 04
(19)
The preceding process continues layer-by-laye by-layer until the final-layer classifier WT T is completed. For clarity, we summarize the detailed training and inference procedures of our DeepAFL in Algorithm and Algorithm 2 respectively, within Appendix A In the subsection, we will comprehensively provide theoretical analyses for our DeepAFL, including its validity, privacy, and efficiency.
3.3 THEORETICAL ANALYSES
Validity Analyses: Here, we provide detailed analyses of the validity of our DeepAFL. Specifically, first derive the analytical solutions of our DeepAFL in Lemmas 1-2, and then demonstrate DeepAFL's dual properties of heterogeneity invariance and representation learning in Theorems 1-2.
Lemma 1: For any least squares problem with the following form:
W = arg min || - w + +
(20)
W
yields distinct analytical (i.e., closed-form) solution, which can be formulated as:
w (
+
- TY
(21)
Proof. See Appendix for
for details.
Lemma 2: For any sandwiched least squares problem with the following form:
5 = arg min
R FnW] + 2
(22)
ft
it yields a distinct analytical (i.e., forme)-form) solution, which can be formulated as:
22 = V((VFF RW (WTT) (71+ + diag(AF) @ diag(A U
(23)
where FTF VA and WWT UAWUT spectral decompositions, while @ and x represent element-wise division and outer product.
Proof. See Appendix B.I for details.

---

Published as a conference paper at ICLR 2026
Theorem (Invariance to Data Heterogeneity)): In the FL scenario, let D be the full dataset, and P = (D', De .......]]... be any heterogeneous partition of D among K clients, where D; DD, = = forit) and -
D. Given fixed seeds for the random projection matrices A and [B.)== the global model weights (W and {Ω, ) derived from DeepAFL. are invariant to any data partition P with different hetcrogeneity, being identical to the centralized analytical solutions on D.
Proof. See Appendix B.2 for details.
Theorem 2 (Capability of Representation Learning): Let the empirical risk H( W), i.c. the loss function, for a given feature representation & and a classifier W be defined as:
H(4 W) = Y bWIF
(24)
where Y denotes the ground truth labels. Our DeepAFL yields sequence of feature-classifie pairs (D,, We) at each layer & E [0, Ti. When setting regularization parameters and À to O, the sequence of the empirical risks {H( W Wi) in our DeepAFL keeps monotonically non- inteeeceessin i.e.,
H( W:) 2 H+ W++1), Vt E T),
(25)
Furthermore, the number of layers 7 within our DeepAFI. increases (i.e., T - oo), the sequence of empirical risks is guaranteed to converge to a limit H < H(Po. Wo).
Proof. See Appendix B.3 for details.
Notably, Theorem 2 demonstrates our DeepAFL's representation learning capability under the basic condition of no regularization (y = À = 0). In practice, we often employ regularization to enhance generalization performance and numerical stability. Thus, we introduce Theorem 3 in Appendix [B.3] to extend our analysis to cover the regularized settings. In addition, we will further provide extensive empirical evidence in Section to support these dual advantageous properties of our DeepAFI.
For the privacy of our DeepAFL., several previous studies have provided similar analyses for Auto-Correlation and Cross-Correlation Matrices (Tan et al.) 2022b He et al.) 2025 Fan et al., 2025b). Particularly, inferring each client's raw data is challenging as the size of its local dataset N₂ remains private. Furthermore, this aggregation process is form of Secure Multi-Parre Purti-Party Computation, and many existing protocols can be readily integrated into our DeepAFL to further enhance privacy (Bonawitz al. 2016 2017 So et al., 2023). See Appendix C for more details.
Privacy Analyses:
Efficiency Analyses: In our DeepAFL, the overall complexities of computation and communication for each client are (TTN, N₂ (dz, + dæ + df)) and (T + d)). while those for the central server are (d + de + dz d)) and d(TKd)))). respectively. See Appendix D for the detailed derivations and analyses. Notably, in Section we will show the superior efficiency of our DeepAFL in comparison to existing gradient-based baselines. In our later experiments, we default to selecting projection dimensions dp = dF- 1024 to balance the effectiveness and efficiency.
4 EXPERIMENTAL EVALUATIONS
4.1 EXPERIMENTAL SETUP
Datasets & Settings. We conduct our experiments three prominent benchmark datasets in FL.: the CIFAR-10 10 (Krizhevsky & Hinton 2009). CIFAR-100 (Krizhevsky & Hinton 2009). and Tiny- ImageNet (Le & Yang 2015). To simulate diverse data heterogeneity scenarios in FL, we use two common non-IID partitioning settings: Latent Dirichlet Allocation (Lin et al. 2020) (as Non-IID-1) and Sharding (Lin et al. 2020) (as Non-IID-2). We use the parameters a and to control the level of heterogeneity in these two non-lID settings, respectively. In both cases, smaller parameter values indicate more heterogeneous data distributions. Specifically. we set a E {0.1, 0.05). s E {2, 4) for CIFAR-10, while setting a E (0.1, 0.01). 8 E (5, 10} for CIFAR- 100 and Imny-mm
Baselines & Metrics. We compare our AFl against traditional gradient- based baselines, including FedAvg (McMahan et al. 2017), FedProx (Li et al. 2020) and MOON (Li et al. 2021a),
FedGen (hhu et al., 2021), FedDyn (Acar et al., 2021). FedNTD (Lee et al. (002), and FedDisco Ye et al. 2023b). Moreover, we also include the analytic learning- based method AFL (He et al. 2025) as a baseline to further highlight our advantages. For fair comparisons, we align the experimental benchmark with those of AFL (He et al. 2025) and use the same pre- pre-trained ResNet-18 backbone for all methods. We employ the accuracy (%), computational cost (s), and communication cost (MB)
as the main metrics. See Appendix H.2 for more details of our experimental implementation.
7

---

Published as a conference paper at ICLR 2026
Table Performance comparisons of the top-1 accuracy (%) among our DeepAFL and the baselines,
on the CIFAR-100 100 and Iiny-ImageNet. The best result is highlighted in bold, and the second-best result underlined. All the experiments were conducted three times, and the results are shown as Error All the improvements of our DeepAFL were validated by Chi-squared tests.
The results of all baselines are directly obtained from the given benchmark in AFL (He al. 2025
CIFAR-100
Tiny-ImageNet
Baseline
Non-1ID-1
Noa-1ID-2
Non-1ID-1
Non-IID-2
= 0.1
10
0.1
10
FedAvg
2017
26.6220.12
32.09
55.7620.13
48.33ga.15
26.0420.2
32.63
39.06go.2
29.6640.19
FedProx
2020
56.45.0.22
33.37
55.80
48.29
46.47
32. 26
35 35.9740.29
29. 29.17
MOON
202
56.5820.02
33.34
60.70
48.34
46.2 20.14
32 a2.38
35. 7940.4
29. 44.300
FedGen
2021
56. 26.18
33.09
60.93
48. 28.12
16.27 26.17
32.33 32.33
38 8240..
29.37 20:25
FodDyn
2021
57.5 37.55
36.1 36.12
61.09 L0.00
39.31
17. 17.72.0.22
36.19.c.mm
11 11.36 36.0o
35. 35.18: jo.rs
FedNTD
2022
56 16.00
32.591...
54.69
19.00
16.17 10.11
31.8610.44
37 37.53 37.551ooce
29.01 1o.14
FedDisco (223g
16.7910.01
25.72
54.66_...09
15.86go.15
47. 18.488...0
27.1516.10
388.86110.12
27. 2T..72.. 18
AFL 22025
58.566
58.56
88.56.0.00
98.56g0.00
54.67
54.677
64.67
64.67
DecpAFL (T 5)
61.7210.07
61.72
64.722..07
61.722..0>
60.31
60.31
00.311004
60.31 1o00
DeepAFL (T 10;
65,96
65.961
65.96_o.0s
65.96_a.ms
61:37
61.37
61.37
61. 01.37 1o.07
DeepAFL (T 20)
66. 06.988..
66. 98
66.98
66.96 10.00
62, 32.55
02.35 1ovo
62 355 g.000
62. 02.35
Improvement
8. 8.42 pc0.00
8.4 p-00005
5.89 p-c 05
7.64 p-c0.05
7.68 p-co00s
7.68 p-c0v0s
7.68 p.c0 0s
7.68 p-c0.00
10"-
DccpAEL
DeepAEL 1P' 5)
-33395
Communication wwtunccciioonn Cost
200
DecsiRL
DeepARL (f'=
293
10"
Gradien based Boselines
270
Grattent bosed
-50.1%
Computstonal
.9.7%%
105-
146,188
74.49
10"
88.00
4B
91 74
39.53
Different Approechee
Different Approaches
(a) Computational Cost
(b) Communication Cost
Figure 2: Efficiency evaluations on the CIFAR-100 dataset.
Computatoral Cos
10P
OcopAFL (2' 10)
DeepAFiL (2 20=-8 5)
47510
Communcation Coss (MB)
6000
DespARR (T 10)
DeepAA (T
$87.00
Deep4F 12' 20)
DoupprFL 20)
Grad basec Banelines
Gireteett bsedd Pres
-70.1%
-89.6%
200
60p
170.38
150.75
72
100.01
100
28
86 08
10
Different Approeches
Dffierent Approaches
(a) Computational Cost
(b) Communication Cost
Figure Efficiency evaluations the Tiny-ImageNet dataset.
4.2 EXPERIMENTAL RESULTS
Main Comparisons. Here, we compare our DeepAFL extensively against baselines across various datasets and settings, shown in Tables and 2 For a fair comparison, we included the benchmark of the AFL (e et al. 2025) and show our DeepAFL's performance under (5, 10, 20). Notably, with small value of T 5, our DeepAFL already surpasses all SOTA baselines, thanks to its dual advantages in heterogeneity invariance and representation learning. As increases (5 - 20), the testing accuracy of our DeepAFL consistently enhances and eventually reaches 86.43% 66.98%, and 62.35% for the CIFAR-10, CIFAR- 100, and improvements of up 5. 11w.88%88... 42% over the SOTA baselines. See Appendix E.3. for more details.
respectively. achieving impressive
Invariance Analyses. In Tables I and 2 our DeepAFL shows invariant performance across various Non-IID settings on the same dataset, akin the AFL. In contrast, other gradient-based baselines commonly suffer performance degradation as the degree of heterogeneity increases. This invariance property of our DeepAFL can further extended the number of clients, As shown in Figure 7 the superiority of our DeepAFL over the gradient- based baseline progressively widens with an increased number clients (100 - 1000) for the large-scale scale scenarios. These results empirically validate the ideal invariance of our DeepAFL, as shown in Theorem 1. See Appendix E.3.2 for more details.
8

---

Published as a conference paper at ICLR 2026
Representation Analyses. Another key advantage of our DeepAFL is its capability of deep repre- sentation learning. As shown in Tables [ our DeepAFL exhibits a stable improvement in both training and testing accuracy the number of layers T increases. This consistent gain shows its effective ability to learn deep representations, handling the common issue of underfitting in tradi- tional analytic learning. Moreover, while our main results only report our DeepAFL's performance at T E {5, 10, 20), we observe its performance continues to improve as "" further increases. Specif- ically, on the complex CIFAR-I00 and Tiny-ImageNet, our DeepAFL's testing accuracy at T 50 improves by over 1.5% compared to that at T 20. See Appendix E.3.3 for more details.
Efficiency Evaluations. We further give efficiency evaluations to show DeepAFL's superior balance between accuracy and efficiency. As shown in Figures 2 we show that, in addition to its SOTA performance, our DeepAFI. also achieves superior efficiency compared to existing gradient-based baselines. In Tables 9 11 and Figures 466 the total time cost of DeepAFL. for 100 clients shows a minimal marginal increase with each additional layer, adding 1-2s per layer for the CIFAR- 10 and CIFAR-100 100, and 3s per layer for the Tiny-ImageNNe achieve more than 9% performance improvements compared to AFL on the complex CIFAR- and Tiny-ImageNet, with a time cost that is less than twice that of AFL, the primary time- consunnnu backbone's 's forward pass only needs to be performed once. See Appendix(E.33. [E.3.4 for more details.
Thus, when set to 7 = 50, our DeepAFL can
Parameter Analyses. Then, we provide comprehensive parameter sensitivity analyses of our Deep- AFL., including the regularization parameters X, 7, the activation function (-), and the projection dimensions da, dr. Specifically, shown in Tables 3 for the regularization parameters, our Deep- AFL is insensitive to A but more sensitive to For the simple CIFAR-10, we observe satisfactory performance with 7 0.5.. For the more complex datasets, CIFAR--00 and Tiny-ImageNet, the optimal value for is a smaller 0.01. In Table 5 the activation function exhibits a notable im- pact on performance, with difference of about 2%. Here, GELU the optimal activation function, while Softshrink is the poorest yet still beats the model with no activation. As shown in Tables 8 larger dimensions for both da and dr lead to better performance. Nonetheless, overly large dimen- sions can not only incur significant overhead due to their quadratic complexity, but also impact the numerical stability, causing crashing when dę = dF = 2 See Appendix E.3.5 for more details.
Ablation Studies. Finally, we conduct extensive ablation studies for the individual contributions of our Dees key components. Specifically, we compared our DeepAFL and four distinct ablation models across different layers on various datasets, as shown in Tables 12 17 and Figures & 10 First, we observe that the residual skip connection in our DeepAFL has a dual role: it not only accelcrates performance improvement the number of layers T increases, but also enhances the final converged performance. Second, ablating the random projection B, of each layer with an identity matrix shows that the stochasticity provided by B, is vital for DeepAFL. to escape local optima and saddle points. Third, ablating the activation function a() proves the necessity of its nonlinearity in our DecpAFL. Fourth, ablating the trainable transformation 2 renders the ablation model ineffective, indicating that its provided learnability is also indispensable. See Appendix E.3.6 for more details. Meanwhile, we also provide comparisons of our DeepAFL against other deepening strategies in Appendix E.3.7
5 CONCLUSION AND DISCUSSION
In this paper, we propose DeepAFL, as the first attempt in FL to achieve gradient-free representation learning while preserving strong invariance to data heterogeneity. By identifying and addressing a fundamental limitation in existing analytie-ccarnin--ue advantages in both heterogeneity invariance and representation learning. By virtue of its gradient- free nature, our DeepAFL also achieves high efficiency by eliminating costly iterative computations and communications on gradients. Given its theoretical and empirical superiority, we believe our DeepAFL. means a great advancement for the SOTA in the fields of both analytic learning and FL.
approaches, our DeepAFL exhibits dual
For future work, a natural and promising direction is to extend our deep residual analytic models in our DeepAFL to other well-suited fields for analytic learning. Specifically, we will go to aim at continual learning, where all data arrives online and historical data is not accessible, While analytic learning has been proven to be effective in solving the catastrophic forgetting problem of continual learning, applying our AeL to this field remains non-trivial. This is because the features learned from a previous phase may become invalid in a subsequent one if updating the weights layer by layer. As a result, we would need a new approach to incorporate the phase-wise recursions as well. Some further diseussion is provided in Appendix N and our usage of LLMS is declared in Appendix O
9

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported partly by the National Natural Science Foundation of China under grants 62576013 and 62306117, the National Key Research and Development of China under grant 2024YFC2607404, the Jiangsu Provincial Key Research and Development Program under Grants BE2022065-1, BE2022065-3, the Ningxia Domain-Specifie Large Model Health Industry R&D 2024JBGS001, and the GJYC program of Guangzhou under grant 2024D0330005.
ETHICS STATEMENT
The authors have read and adhere to the ICLR Code of Ethics. We have carefully considered the potential ethical implications of our work and believe that proposed approach, DeepAFI., aligns with the principles outlined in the provided ICLR Code of Ethics.
Our research is situated within the FL paradigm, which is specifically designed to address privacy and security concerns by enabling collaborative model training without requiring the sharing of raw, sensitive user data. Our proposed DeepAFL operates exclusively in a decentralized setting, with all local client data remaining on the respective devices. We provide detailed privacy analyses in our paper and discuss how our DeepAFL can be enhanced with existing proven privacy-presenve protocols. The empirical evaluations presented in this paper are based on eetalllstaliidd publicly available benchmark datasets (i.e., CIFAR-10, CIFAR-100, and Imay-ImageNet), This work did involve any human subjects, the collection of new private data, or the use of sensitive information.
Furthermore, a core contribution of our proposed DeepAFL its superior performance in FL, with its dual advantages in heterogeneity invariance and representation learning. To the best of our knowl- edge, the process our DeepAFL does not introduce additional fairness and bias issues. On contrary, its ability to achieve superior and invariant performance across various heterogeneous FL environments can help foster more equitable and generalizable models in real-world applications.
Lastly, in line with the principles of research reproducibility, open-source upon the paper's acceptance. We believe this will allow the broader research community to verify findings, build upon our work, and ensure full transparency of our methodology.
committed to making code
REPRODUCIBILITY STATEMENT
For reproducibility, we have made extensive efforts to provide the necessary details. We give detailed procedures of our approach our paper, including comprehensive descriptions of the algorithmic steps and design choices. For theoretical claims, the complete proofs and analyses included in the Appendix. Codes are available at httpp::/github...
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference Learning Representationss 2016.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3598)11788-1828, 2013. doi: 10. 1..1099PMMM..2013.500
Erik L Bolager, Iryna Burak. Chinmay Datar, Qing Sun, and Felix Dietrich. Sampling weights of deep neural networks. Advances Neural Information Processing Systems, 36:63075-6311 63116, 2023.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning user-held data. arXiv preprint arXiv: 1.04482, 2016.
10

---

Published as a conference paper at ICLR 2026
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Kam Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 1175-1191, 191, 2017.
Di Chai, Leye Wang, Liu Yang, Junxue Zhang, Kai Chen, and Qiang Yang. A survey for fed- erated learning evaluations: Goals and measures. IEEE Transactions on Knowledge and Data Engineering, 36(10):5007-5244 2024. doi: www.1009KKDDE2222333880000
Randall E Cline. Representations for the generalized inverse of a partitioned matrix. Journal of the Society for Industrial and Applied Mathematics, 12(3):588-66) 600, 1964.
Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with ap- plications in pattern recognition. IEEE transactions on electronic computers, (3):326-334, 2006.
Feng Ding, Peter X. Liu, and Jie Ding. Iterative solutions of the generalized sylvester matrix equa- tions by using the hierarchical identification principle. Applied Mathematics and Computation, 197(1):41-50, 2008 ISSN 0096-3003.
Guang-Ren Duan. Generalized Sylvester equations: unified parametric solutions. Cre Press, 2015.
Kejia Fan, Yajiang Huang. Jingyu He, Feijiang Han, Jianheng Tang. Huiping Zhuang, Anfeng Liu, Tian Wang, Mianxiong Dong, Houbing Herbert Song, and Yunhuai Liu. CALM: A ubiq- uitous crowdsourced analytic learning mechanism for continual service construction with data privacy preservation. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(2,, 2025a. doi: 10/3745/77
Kejia Fan, Jianheng Tang, Zhirui Yang, Feijiang Han, Jiaxu Li, Run He, Yajiang Huang, Anfeng Liu, Houbing Herbert Song, Yunhuai Liu, and Huiping Zhuang. APFL: Analytic personalized federated learning via dual-stream least squares. arXiv preprint, 2025b.
Tao Fan, Hanlin Gu, Xuemei Cao, Chee Seng Chan, Qian Chen, Yiqiang Chen, Yihui Feng, Yang Gu, Jiaxiang Geng, Bing Luo, et al. Ten challenging problems in federated foundation models, 2025c.
Eros Fani, Raffaello Camoriano, Barbara Caputo, and Marco Ciccone. Accelerating heterogeneous federated learning with closed-form classifiers. In International Conference on Machine Learning, pp. 13029-13048. PMLR, 2024.
Jialin Guo, Jianheng Tang, Anfeng Liu, Neal N Xiong, Jie Wu, Xiaomin Ouyang, and Jun Huang. COOL: A cloud-fog federated learning system with multimodal isolated client data in edge net- works. IEEE Transactions on Mobile Computing, 2025.
Ping Guo, Michael R Lyu, and NE Mastorakis. Pseudoinverse learning algorithm for feedforward neural networks. Advances in Neural Networks and Applications, 1(321-326), 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Run He, Kai Tong, Di Fang, Han Sun, Ziqian Zeng. Haoran Li, Tianyi Chen, and Huiping Zhuang. AFL: A single-round analytic approach for federated learing with pre-teri models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Chenghao Hu and Baochun Li. Maskerypt: Federated learning with selective homomorphic en- cryption. IEEE Transactions Dependable and Secure Computing, 22(1):221-233, 2025. doi: 10.109//DSC..2243333222
Rui Hu, Yuanxiong Guo, and Yanmin Gong. Federated learning with sparsified model perturba- tion: Improving accuracy under client-level differential privacy. IEEE Transactions Mobile Computing, 23(8):22428825 2024. doi: 10. 10.1109TT...2223333544
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy in classification tasks. In International Conference Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
Songning Lai, Mingqian Liao, Zhangyi Hu, Jiayu Yang, Wenshuo Chen, Hongru Xiao, Jianheng Tang, Haicheng Liao, and Yutao Yue. Learning concepts, remembering the old: Continual learning for multimodal concept bottleneck models. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 12314-12322, 2025.
Samuel Lanthaler and Nicholas H Nelsen. Error bounds for learning with vector-valued random features. Advances in Neural Information Processing Systems, 36:71834 11188-4488611 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Gihun Lee, Minchan Jeong. Yongjin Shin, Sangmin Bae, and Se- Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. In Alice H. Oh, Alekh Agar- wal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Jiaxu Li, Rui Li, Jianyu Qi, Songning Lai, Linpu Lv, Kejia Fan, Jianheng Tang, Yutao Yue, Dongzhan Zhou, Yunhuai Liu, et al. CFSSeg: Closed-fome solution for caass-incremmntll seman- tic segmentation of 2d images and 3d point clouds. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 247-256, 2025a.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021b.
Yichen Li, Yijing Shan, Yi Liu, Haozhao Wang, Wei Wang, Yi Wang, and Ruixuan Li. Personalized federated recommendation for cold-start users via adaptive knowledge fusion. In Proceedings of the ACM on Web Conference 2025, pp. 2700-2709, 2025b.
Yichen Li, Haozhao Wang. Wenchao Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang, Xin Yang, Rui Zhang, Shui Yu, al. Unleashing the power of continual learning on non-centralized devices: A survey. IEEE Communications Surveys & Tutorials, 2025c.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2351 Curran Associates, Inc., 2020.
Xiang Liu, Liangxi Liu, Feiyang Ye, Yunheng Shen, Xia Li, Linshan Jiang, and Jialin Li. FedLPA: One-shot federated learning with layer-wise posterior aggregation. Advances in Neural Information Processing Systems, 111818115558, 2024a.
Yiqi Liu, Shan Chang, Ye Liu, Bo L., and Cong Wang. FairFed: Improving fairness and efficiency of contribution evaluation in federated learning via cooperative shapley value. In IEEE INFOCOM 2024 IEEE Conference on Computer Communications, pp. 621-630, 2024b. doi: 10.1 20000M5522222222111611333333
Cheng-Yar Yaw Low, Jaewoo Park, and Andrew Beng-Jin Jin Teoh. Stacking- deep neural network: deep analytic network for pattern classification. IEEE Transactions on Cybernetics, 50(12):5021- 5034, 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Y Ar- cas. Communication--fficee -efictionnefiii learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273-1282, PMLR, 2017.
12

---

Published as a conference paper at ICLR 2026
Cui Miao, Tao Chang, Meihan Wu, Hongbin Xu, Chun Li, Ming Li, and Xiaodong Wang. Fed- vla: Federated vision- vision-language- language-action learning with dual gating mixture-of-expers for robotic manipulation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6904-6913, 2025.
John Nguyen, Jianyu Wang. Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where begin? on the impact of pre-training and initialization in federated learning. In The Eleventh International Conference on Learning Representations, 2023.
Liangzu Peng, Juan Elenter, Joshua Agterberg, Alejandro Ribeiro, and Rene Vidal. TSVD: Bridging theory and practice in continual learning with pre-trained models. In The Thirteenth International Conference Learning Representations, 2025.
Hongming Piao, Yichen Wu, Dapeng Wu, and Ying Wei. Federated continual learning via prompt- based dual knowledge transfer. In Forty-first International Conference on Machine Learning. 2024.
Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip Torr, Ozan Sener, and Puneet Dokania. RanDumb: random representations outperform online continually learned represen- tations. Advances in Neural Information Processing Systems, 37:37988-3888 -30066, 2024.
Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Bo Zhao, Liping Yi, Alysa Ziying Tan, Yulan Gao, Anran Li, Xiaoxiao Li, Zengxiang Li, and Qiang Yang. Advances and open challenges in fed- erated foundation models. IEFE Communications Surveys & Tutorials, pp. 1-41, 2025. doi: 10. 111109//COMMTT203355555522
Changlong Shi, He Zhao, Bingjie Zhang, Mingyuan Zhou, Dandan Guo, and Yi Chang. FedAWA: Adaptive optimization of aggregation weights in federated learning using client vectors. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30651-30660, 2025.
Jinhyun So, Ramy E Ali, Başak Güler, Jiantao Jiao, and A Salman Avestimehr. Securing secure aggregation: Mitigating multi-round privacy leakage in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9864-9873, 2023.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE transactions on Neural Networks and Learning Systems, 34(12)99587-9603 2022a.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fed- proto: Federated prototype learning across heterogeneous clients. Proceedings of the AAAI Conference on Artificial Intelligence. 36(8):8432 2440, Jun. 2022b.
Jianheng Tang, Huiping Zhuang, Jingyu He, Run He, Jingchao Wang, Kejia Fan, Anfeng Liu, Tian Wang, Leye Wang, Zhanxing Zhu, Shanghang Zhang, Houbing Herbert Song, and Yunhuai Liu. AFCL: Analytic federated continual learning for spatio-temporal invariance of Non-IID data. arXiv preprint, 2025.
Kar-Ann Toh. Learning from the kernel and the range space. In 2018 IEEE/ACIS 17th International Conference on Computer and Information Science (ICIS), pp. 1-6. IEEE, 2018.
Ziyao Wang, Jianyu Wang, and Ang Li. FedHyper: A universal and robust learning rate scheduler for federated learning with hypergradient descent. In The Twelfth International Conference on Learning Representations, 2024.
Ai-Guo Wu, Guang-Rec Ren Duan, and Bin Zhou. Solution generalized sylvester matrix equations. IEEE Transactions on Automatic Control, 53(3):811-55 2008. doi: 10. 10.1119A...2208.999555
Meihan Wu, Tao Chang, Cui Miao, Jie Zhou, Chun Li, Xiangyu Xu, Ming Li, and Xiaodong Wang- EFTVIT: Efficient federated training of vision transformers with masked images resource- constrained clients. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1815-1824, 2025.
13

---

Published as a conference paper at ICLR 2026
Xiyuan Yang. Wenke Huang, and Mang Ye. FedAS: Bridging inconsistency in personalized fed- erated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11986-11995, 2024.
Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. FedFed: Feature distillation against data heterogeneity in federated learning. In Advances in Neural Information Processing Systems, volume 36, pp. 60397-60428. Curran Associates, Inc., 2023.
Mang Ye, Xiuwen Fang, Bo Du, Pong C. Yuen, and Dacheng Tao. Heterogeneous federated learning: State-of-the-an and research challenges. ACM Computing Survey, 56(3), October 2023a. ISSN 0360-0300. doi: 10.1 10.1145/3625558.
Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. FedDisco: Federated learning with discrepancy-aware collaboration. In Proceedings of the 40th International Conference on Machine Learning, volume 202. pp. 39879-39902. PMLR, 23-29 Jul 2023b.
Hao Yu, Xin Yang. Xin Gao, Yan Kang, Hao Wang. Junbo Zhang. and Tianrui Li. Personalized fed- erated continual learning via multi-granulane granularitu prompt. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 4023-4034, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901.
Hao Yu, Xin Yang, Le Zhang, Hanlin Gu, Tianrui Li, Lixin Fan, and Qiang Yang. Handling spatial- temporal data heterogeneity for federated continual learning via tail anchor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Xiang Zhang, Run He, Chen Jiao, Di Fang, Ming Li, Ziqian Zeng, Cen Chen, and Huiping Zhuang. L3A: Label-augmented analytic adaptation for multi-label class incremental learning. In Fecty-second International Conference on Machine Learning. 2025.
Xiaokang Zhou, Qiuyue Yang. Xuzhe Zheng, Wei Liang, Kevin I-Kai Wang, Jianhua Ma. Yi Pan, and Qun Jin. Personalized federated learning with model-contrastive learning for multi-modal user modeling in human-centric metaverse. IEEE Journal on Selected Areas Communications, 42(4):817-835 831, 2024.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou, Data-free knowledge distillation for heterogencous federated learning. In International Conference on Machine Learning, pp. 12878-12889) PMLR, 2021.
Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Correlation projection for analytic learning of classification network. Neural Processing Letters, 53(6):3893-3914, 2021.
Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. ACII: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35: 1600-1141,,, 2022.
Huiping Zhuang, Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. GKEAL: Gaussian kernel
embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7746-7755, 2023.
Huiping Zhuang. Yuchen Liu, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Yi Wang, and Lap-Pui Chau. F-OAL: Forward-only online analytic learning with fast training and low memory footprint in class incremental learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Huiping Zhuang, Zhiping Lin, Yimin Yang, and Kar-Ann Toh. An analytic formulation of convolu- tional neural network learning for pattern recognition. Information Sciences, 686:121317, 2025. ISSN 0020-0255.
Nikita Zozoulenko, Thomas Cass, and Lukas Gonon. Random feature representation boosting. In Forty-second International Conference on Machine Learning, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX FOR PROCEDURES OF OUR DEEPAFL
Algorithm The Training Procedure of our Proposed DeepAFL
Input: The clients' local datasets (D), DD2 ,... DD
Output: The transformation matrices (Ωt } and the analytic classifiers { W
1: for each layer {0, 2, , do
2: // (a) Local Computation for Analytic Classifier Construction (Client side)
3: for each client E 2,, K do
4: if constructing the first layer, i.e., t = 0 then
5:
Extract and construct its local zero-layer feature matrix t6 via 100 Compute its local Feature Auto-Correlation Matrix G using & via
6:
7:
Compute its local Label Cross-Correlation Matrix H using 4 and Y via
8:
Transmit (G, H to the server,
9:
// (b) Information Aggregation for Analytic Classifier Construction (Server side)
10:
Aggregate { K
km and {H; JKmI
km to obtain G and HisK via
12);
11:
Derive the global classifier W, using the obtained G.!. and H via 13
12:
Transmit the global classifier W, to all clients;
I/ (c) Local Computation for Residual Block Construction (Client side)
14:
for each client E [1,2.-.... { do
15:
if layer requirement not satisfied, i.e., t < T then
16:
Compute its local hidden random feature F using & via (14); Compute its local residual matrix R using & and W. via (14
18:
Compute its local Hidden Auto-Corre Correlation Matrix II using F via 15
19:
Compute its local Residual Cross-Correlatio Matrix Yf using F and R via 15
20:
Transmit [II Y the server;
// (d) Information Aggregation for Residual Block Construction (Server side) then
22:
if layer requirement not satisfied, i.e.,
23:
T==1 and { T==1 to obtain II and Y! via (16) Aggregate (II. K
24:
Derive the transformation matrix 22 using III: and Y.KK via 17 and (18)
Transmit the transformation matrix 44+1 all clients;
26:
// (e) Feature Updating for Next Layer Construction (Client side)
27:
if layer requirement not satisfied, i.e., t < T then
28:
for each client & 21,2, K) do Update its local feature matrix & to obtain o via (99
30:
Return: The transformation matrices [ and the analytic classifiers (W,)T.. 1T
Algorithm The Inference Procedure of our Proposed DeepAFL.
Input: Local sample x", transformation matrices (Ω, J7 and analytic classifiers (W.)
Output: Predicted label g'.
Extract and construct local zero-layer feature Tb using xf via (10)
2: for each layer E {1, ... ,T} do
3: Compute its local hidden random feature F, via (14
4: Update its local feature $ to obtain - using S2 and F; via
5: Calculate the predicted score vector & = क W
6: Calculate the predicted label g arg max()'):
7: Return: Predicted label i²'
15

---

Published as a conference paper at ICLR 2026
B APPENDIX FOR VALIDITY ANALYSES
In this section, we thoroughly analyze the validity of our DeepAFL, theoretically establishing sev- eral key propositions. First of all, in Appendix B.I derived the analytical solutions for the least squares problems addressed in this paper, which also constitute the centralized analytical solution. Next, in Appendix B.2 we demonstrate that the deep residual analytic networks derived from our DeepAFL is equivalent to the centralized closed-form solutions and remain invariant to data hetero- geneity. Finally, in Appendix(B.3 we validate DeepAFL's representation learning capacity, proving that the empirical risk of our Aee is monotonically non-increasing with increasing layer depth.
B. DERIVATION OF ANALYTICAL LEAST SQUARES SOLUTIONS
Here, we theoretically derive the analytical solutions to the least squares problems 4 and 7 ad- dressed in this paper. In fact, these solutions constitute the centralized analytical solutions for both the global classifier and the transformation matrix. The detailed proofs are as follows.
Lemma 1: For any least squares problem with the following form:
W arg min Y
w + W/
(26)
w
it yields a distinct analytical (i.e., closed- form) solution, which can be formulated as:
w
+
+
1 T
(27)
Proof. Our proof strategy involves computing the derivative of the loss function (26) with respect to W and subsequently setting it to zero derive the optimal W Specifically, first denote the loss function of the aforementioned least squares problem (26 as follows:
J(W) ||Y
&w X| W[
(28)
Subsequently, to facilitate subsequent differentiation, we expand the loss function (28 as follows:
J(W) ||Y - w + W||
-Tr((Y- - kw) (Y
&W)) + XTr(W W)
(29)
=Tr(Y + 2Tr(Y Φ TT(TTTTT 4 &W) ATT(W W).
where Tr(-) represents the trace of matrix. To minimize the objective function, further compute the derivative of J(W) with respect to W as follows:
OJ (W( OW
24 + 2Φ ΦW - 2AW.
(30)
By setting the derivative to zero, we can obtain:
a T + XI =
(31)
Since T + + AI) is full rank and hence invertible, the closed-form solution for the optimal W can be obtained as follows:
W + +
T
(32)
Lemma 2: For any least squares problem with the following form:
Ω arg min R - FF-FWWJ FF +
(33)
it yields distinct analytical (i.e., closed-form) solution, which can be formulated as:
Ω V((V F T RW T U) @ (71 + diag(A") @ diag(A d))j(A")))U",
(34)
where 2 and @ represent element-wise division and outer product, respectively, while F V and WWT UAWU are spectral decompositions.
16

---

Published as a conference paper at ICLR 2026
Proof. Similar to the proof of Lemma 1, we begin by computing the derivative of the loss function in (33) with respect to and then set this derivative to zero to derive the optimal Ω2". Specifically, denote the loss function of the aforementioned least squares problem (33 as follows:
((2)) |R FΩW F + +
(35)
which can be further expanded into the following form:
(()) || F2W | +
( FΩW) T (R FΩW)) + +Tr((2
(36)
-T(RR T R. - 2Tr(R FRW) + Tr(W F FOW) + +TTr(s
Subsequently, by further differentiating (36), OC(S2 2F RW + FFWW + 22
obtain:
(37)
an
By setting the derivative to zero, we can obtain:
F F'F''WW + + FF
(38)
Since F F and WWT are real symmetric matrices, we can spectrally decompose them to obtain:
F F VA WW UA
(39)
where V V
I and
U U = I, and I denotes the identity matrix. By denoting S = V U and
substituting 39
into
38
we obtain:
F' +700 F RW
SAFVSN'AA + F RWT V W U F U
(40)
A'SA + U.
Since AF and W are both diagonal matrices, we proceed to expand them element-wise. Specifi- cally, let X and AW denote the i-th and j-th diagonal elements of A and AW, respectively. Con- currently, we denote the (i. entry of S and V FRR U by Sig and (V F RW U). respectively. According to (40) we obtain:
X S + TS. (V F NUTWTT UUL
(41)
Meanwhile, can further obtain:
F NTTTUU UUe
S
(42)
MWW +7
Based on (42 we can extend St,j the entire matrix S, yielding:
S= ( RW U) @ (y1 + diag(A A & diag(A)))
(43)
where 1 is the all-ones matrix, while @ and & represent clement-wise division and outer product. Finally, leveraging the orthogonality of V and U. we can obtain L VSUT. Substituting 43 into this expression, we can obtain:
RWTU) n = V(V F RW U) 2 (71 + diag(AF) & diag(A diag(AW))).,
(44)
17

---

Published as a conference paper at ICLR 2026
B.2 OUR DEEPAFL'S INVARIANCE TO DATA HETEROGENEITY
Here, we demonstrate that the deep residual analytic network derived from our DeepAFL exhibits ideal invariance to data heterogeneity, remaining invariant any data partition across clients with different heterogeneity. Specifically, as established in Lemma and Lemma 2, we have derived the centralized analytical solutions for both the global classifier and the transformation matrix. Here, we additionally prove that the solution obtained from our DecpAFL is identical to these centralized analytical solutions. The detailed proofs are as follows.
Theorem (Invariance to Data Heterogeneity): In the FL scenario, let 'D be the full dataset, and P= = ( , D2 , be any heterogeneous partition of D among K clients, where D2 D, for j j and U D; = D. Given fixed seeds for the random projection matrices A and {B, 1T-1 the global model weights {We) and {2, derived from DeepAFL are invariant to any data partition with different heterogeneity, being identical to the centralized analytical solutions on D. 1:
Proof. As established in Lemma I and Lemma 2, for the complete dataset D = {XI: Y J, the centralized analytical solutions for {W.) and {$2r]== can be expressed as:
W.
XI] 1 (T)
(45)
Ω V.[(VV (F R l: W. U,) Ø (71 diag(A diag(A))))
(46)
where (FI-K) FI.k V. and W, W U,ANU? U.T are spectral decompositions, while Ø and @ represent element-wise division and outer product, respectively.
In fact, any heterogeneous partition P can be viewed as a permutation of the sample ordering within the complete dataset D. Accordingly, for any heterogeneous partition P {D), DD ..... D* }. we define the corresponding permuted complete dataset D
{X 1-1:K,, J. which satisfies:
(47) where π is the corresponding permutation matrix and satisfies π T π I. Next, we employ mathe- matical induction to rigorously demonstrate that the global model weights (W.)T= and { obtained by our Dee under any heterogeneous partition P are identical to their centralized analytical solutions, as presented in (45) and (46).
X x 1:K 1
rY
For the base case, we demonstrate that the initial Wo and i obtained via our DeepAFL. precisely satisfy (45) and (46). It can be readily deduced that the complete zero-layer feature matrix similarly satisfies 4 T T According
13 the expression for Wo can be given by:
Wn (
AI -
11KK + {I
(11K) Y
(48)
[[0]:K]]
T
1:
0
+
XI
0
I:K
4-1:K
[(
l:K
AI]
k:k
is evident that Wo coincides precisely with the corresponding centralized analytical solution 45
Consequently, the residual corresponding to each sample matches that obtained through centralized training, differing only in sample order. Thus, the complete residual matrix satisfies R Similarly, the complete hidden random features also satisfy F 188 the Ω is given by:
-.R I:K
FF! Drawing from 17 and
á Vol(V] Y
Un)
(>1 (A)) diag A W J
= Vol(V. K
R
W Up) 2 ( (iag @ diag UJ
(49)
= =Vo((V, [ 0
I:K
π
T
l:
w. U.) 2 (+1 diag(A & diag A W J
=Vo[(V] [F!K)
l:K W
Up) U.) @ (사1 + diag(A @ (A) J)
where the spectral decomposition results are also identical to those in the preceding formula (46): (F]:K) F J . F = (F):) FJ VoAjvV and WoW, = Wo W. U U.AWU. can be observed that the initial 2 derived from our DeepAFL also precisely matches the centralized analytical solution. Thus, the base case holds for the theorem.
U
18

---

Published as a conference paper at ICLR 2026
For the inductive step, assume the preceding (t - 1) layers {Wi)= and [52 satisfy (45) and (4). We further demonstrate that, based on these assumptions, the subsequent layer W; and 1 obtained via our DeepAFL also satisfy (45) and (46). Since the previous (t - layers' global model weights exactly match the centralized analytical solution, the complete feature matrix $1:K and the complete hidden random features F!:K satisfy T.:K D. and F!:K xF!. According to
13 the expression for W, is thus obtained follows:
W. (G
-H,
[()]k)] $1.k AIJ
( y
1:1K
(50)
[( π D
1:K
+
AI
(Ø. 1:K y l:K
[(b)))] Φ biKk AI
Y I:K
is evident that the global classifier W. satisfy (45) and thus the complete residual matrix satisfies
R.KK = rR. Subsequently, analogously to 49 we derive the analytical expression for 82
yielded by our DeepAFL as follows:
Ω2 V.[(V, T. W. U.) O + diag(A diagAia @ diag (A, (
=Vi((V, (F R
R W. U,) (71 (Aag(A,) diag (MM)) JU?
(51)
= =V.[(V]
RR, 1: W, U,) 2 (+1+ + diag (A.) & diag (A)) DJ)T 1: w. U,) @ (71 + diag(A,) @ diag(A, JU?
=VI(VV "I((V(FF)))) R.
where the spectral decomposition results also identical to those in the preceding formula (46) (FI:K) Fi.K (FI.K)Tz T (F..K) T V,AVV, and W,W, W. W,
U,A.U. As result, the transformation matrix Ω derived from our DeepAFL also precisely matches the centralized analytical solution. Thus, the inductive step also holds for the theorem.
In summary, based on the aforementioned base case and inductive step, we establish via mathemati- cal induction that the global model weights (W. and (S2, } derived from our DeepAFL. are invariant to any data partition P with different heterogeneity. In fact, the results obtained from our DeepAFL are identical to the centralized analytical solutions 45) and (46).
B.3 OUR DEEPAFL'S ' CAPABILITY OF REPRESENTATION LEARNING
Here, further demonstrate that the empirical risk monotonically non-increases with increasing layer depth and is guaranteed to converge to a limit, thus validating DeepAFL's representation learn- ing capability. To begin, we analyze the general scenario without considering the regularization term in Theorem 2, proving that under this scenario, the empirical risk derived from our DeepAFL monotonically non-increases with increasing layer depth and converges limit.
Theorem 2 (Capability of Representation Learning):
Let the general empirical risk H(P,, W) (i.e.. the loss function), for given feature representation & and a classifier W be defined as:
H( W) = ||Y - @
(52)
where Y denotes the ground truth labels. Our DeepAFL yields sequence of feature-classifie pairs (t,, We) at each layer t E [0, T). When setting regularization parameters y and À to 0. the sequence of the empirical risks (H(P,, W.)) in our DeepAFL keeps monotonically non-increasing, i.e.,
H.PP,, W.) > H. P(111+ W++1), Vt E [O,T).
(53)
Furthermore, the number of layers T within our DeepAFL increases (i.c., T 20), the sequence of empirical risks is guaranteed to converge to limit H. < H(@o. Wo).
Proof. According Algorithm the construction of each feature-classifier pair (4+411 W proceeds in two alternating steps: (1) fixing the classifier W, to construct the feature representation P441; (2) fixing the updated feature representation Φ optimize the classifier W We will prove that the empirical risks in each these substeps are non-increasing, thereby demonstrating that the sequence of empirical risks (H((), W.) is monotonically non-increasing.
19

---

Published as a conference paper at ICLR 2026
First, let's focus the substep of constructing the feature representation Φ1+1 while keeping the global classifier W, fixed. At this stage, the intermediate empirical risk for the crature-classiffe pair (#++1, Wx) can be expressed as:
H( +1 Wi I 11+1 Welli
(54)
where 1 obtained through our DeepAFL. satisfies L.+1 = 4, + F, Therefore, the afore- mentioned empirical risk in our DeepAFl. can be further expressed
-- W.) = Y ( )
(55)
According to Lemma 2, the 1211 derived through our DeepAFL represents the optimal solution that minimizes the aforementioned empirical risk, thereby satisfying:
H( 111, W,) = || ( F,S(+1) Well & I (Φ: + F.S)).
Aa,
(56)
where, as special case when 2S 0, it holds that:
| - ( + F, IF: = IY - Will H., W.).
(57)
Thus, it follows that:
H( W.) S H(P,, W.).
(58)
Second, we further analyze the substep constructing the global classifier W while keeping the feature representation Φ141 fixed. When the regularization parameter À is set to zero, the W derived through our DeepAFL obtained by minimizing the following empirical risk:
W ++1 arg min || +i141 W arg min H(P 1+1, W).
(59)
According to Lemma 1, the W derived through our DeepAFL. represents the optimal solution that minimizes the aforementioned empirical risk. We can obtain:
H( L+1, W++1) Y
P W IY D W VW,
(60)
where, as a special case when W
W,, it holds that:
Y P Well H(P1+1, Wt).
(61)
Thus, it follows that:
H( (141, W W1+) s H (141, W.).
(62)
In summary, based on (88 and 62 it holds for each feature-classffier pair (P141, We+1) that:
H( W.) 2 H( W.) 2 H( ((#11 W++1), Vt (0,
(63)
Therefore, the sequence of empirical risks (H( W.YLT is monotonically non-increasing. Given that H(P,, W.) > 0 and the sequence is monotonically non-increasing, by the Monotone limit H* > 0. Moreover, since the sequence
Convergence Theorem, necessarily converges begins at H @u. Wo) and decreases at each layer, it holds that 7." < H(Pu, Wu).
Considering our employment of regularization to enhance generalization performance and numerical stability, we further extend Theorem 2 to incorporate the regularized setting. To this end, we begin by analyzing the regularization terms in the regularized empirical risk, thereby defining it for the subsequent theorem, as detailed below:
(1) From the perspective of empirical risk. the empirical risk is function of the global classifier We and the feature representation +,, Accordingly, both should be subject to regularization. The feature representation 4, is obtained through layer-by-layer updates and can essentially be expressed as:
Φ = 4 + (2) 2. 12 F,-2Sh.11 =.. Po Σ F. (64)
where F;-1 is not trainable, providing stochasticity and nonlinearity to the features via fixed acti- vation functions and random projections, whereas Z constitutes the trainable component for rep- resentation learning. Thus, regularizing all { can be seen as a form of regularizing P,. In addition, the global classifier W, is itself trainable, and should be regularized directly.
20

---

Published as a conference paper at ICLR 2026
(2) From the perspective of the model architecture, the global model derived from DeepAFL after constructing the first t layers encompasses the final global classifier W, and all transformation matrices {2; Therefore, regularization terms need to be introduced for both W, and {Ωt M=1-
Building upon the foregoing analyses of the regularized empirical risk, we further provide its explicit definition in 65). Moreover, in Theorem we theoretically prove that the empirical risk of the resulting model is monotonically non-increasing as the layer depth increases.
Theorem 3 (Regularized Capability of Representation Learning): In our DeepAFL, define the regularized empirical risk (()) (i.e., the regularized loss function) for each layer / E [0, 7] as:
(()) Y 4 w.
W
Σ
||
(65)
(2)
(3)
where y >> 0 and À > 0 are non-negative regularization parameters. Moreover, Dt. W., and S2 the feature representation, the analytic classifier, and the learned transformation in our DeepAFL at the layer Here, the term (1) is the original empirical risk H (;,, W.), while the terms (2) and (3) are the regularization in our AeL The sequence of the regularized empirical risks (G(()) 17 in our DeepAFL remains monotonically non-increasing, i.c.,
(t) 2 G(t + 1), Vt E [0,
(66)
Furthermore, the number of layers T within our DeepAFI. increases (i.e., T T -> 20), the sequence of the regularized empirical risks is guaranteed to converge to limit G < ((()).
Proof. Analogous to Theorem 2, we sequentially analyze the construction of the feature represen- tation D2+1 (i.e., deriving the transformation matrix 12++1) and the global classifier W at each layer, thereby demonstrating that the regularized empirical risks {g(t)) in our DeepAFL remain monotonically non con-icreeasing
First, let's focus on the substep of constructing the feature representation 인.-1 and the learned transformation Ω2+1 while fixing the global classifier W, and other transformations {2 For notational convenience, we denote the intermediate regularized empirical risk between layers and
t ++1 (i.e., the risk associated with Pz+1, Wt, and (2t [++1 as (()), which can be expressed as:
(t) = Y
& Pr+1 w.ll + A||W, Hi > 1+1
(67)
where 4.+1 obtained through our DeepAFL satisfies 2 2, + F,52,11 Therefore, the afore- mentioned regularized empirical risk can be further expressed as:
((t) Y (Ø; 77 + A/
Σ
(68)
According to Lemma 2, the Ω derived through our DeepAee.. the optimal solution that minimizes the regularized empirical risk:
1 = arg min Y (Φ: -..-FFA FF, +
(69)
f
Therefore, we can obtain:
Y - (D, + F. W. 7 < Y - (D, + F.S2W 11 UQ,
(70)
where Ω = 0 serves as a special case, from which we further derive:
Y - (Φ, + F.R.41) + 2 S Y W. E.
(71)
By adding A/| W ll: and Σ/= S| |l:: to both sides of inequality 71 obtain:
(1) = Y (Φ. + F W W/|
Σ 2
(72)
< Y W.I. W. + Σ
21 |l (()).
Second, we further analyze the substep of constructing the global classifier W.-1 while keeping the feature representation Φ.41 and transformation matrices {Ω, 1++ fixed. According to Lemma 1, the W e1 represents the optimal solution that minimizes the regularized empirical risk:
W W++ = arg min Y
P W + >/
(73)
W
21

---

Published as a conference paper at ICLR 2026
Therefore, obtain:
| De41 W + W/f will S || - P1+1 + + VW,
(74)
where W = We serves as a special case, from which we further derive:
| 44 W W + W|[Wa 4+1 | S Y
P welli W.ll
(75)
By adding [+1 7| || to both sides of inequality 75 we obtain:
++1
G(l 1) Y
Pe+1 W We+i
11 + IlK W. ++1 I i=1 711
(76)
S e+ W. + X//W I + Σ 1im 7 >112.11 = 9()).
In summary, based on 72 and (6)). it holds for each layer that:
(t)) 2 G(t) 2 G( + 1), Vt (0,
(77)
Therefore, the sequence of the regularized empirical risks (9(1)) in DeepAFL remains
monotonically non-increasing. Given that g(t) 2 0, by the well-known Monotone Convergence
Theorem, it necessarily converges to limit G" 2 0. Moreover, since the sequence begins at G())
and decreases at each layer, it holds that G < (()).
C APPENDIX FOR PRIVACY ANALYSES
In this section, we provide detailed privacy analyses of our DeepAFL., providing multi-layered anal- yses to substantiate our DeepAFL'ss robust privacy preservation, as follows:
and Pross-Correlation Matrices, and numerous studies (Tan et al.] 2022b He et al. (225 Fan et al.) 2025b) have shown the privacy advantages of uploading such information. Specifically, according to [5). the uploaded Cross-Correlation Matrices II and Y are essentially prototype matrices, where each column corresponds to the unaveraged class and residual prototypes for the correspond- ing class. Consequently, the information uploaded within DeepAFL. is akin to that of prototype- based FL methods (Tan et al.) 2022b), which have provided a detailed introduction to the inher- ent privacy benefits of transmitting prototypes. Additionally, existing analytical learning-based FI. methods (He et al. 2025 Fan et al. 2025b) also require the uploading of coo--oorellatio and Matrices similarly. Thus, our DeepAFL shares the same privacy advantages as these existing methods.
First, in our DeepAFL, all clients need only submit their locally computed
Second, since clients not required to upload their local dataset sizes Nr. it is pretty hard to infer the clients' private data (i.e., X Y Φ F.. and RC) from the information they upload. Specifically, in our DeepAFL. each client uploads only its local matrices G E Rdde H² Rd XC II E TR R and r E Rd4xxC Evidently, clients need not upload their local dataset sizes Nu. which also cannot be deduced from the dimensions of the aforementioned matrices. Moreover, the dimensions of all private data are directly tied to No. i.e., X E R.N. x xdw Xde, Y E RX Φb N.Nxxc xda Fk E Ne and R, E NxxxC Therefore, without knowledge of Ng. the dimensions of the aforementioned private data cannot be determined. This results in infinitely many possible instantiations. rendering fundamentally impossible infer the clients' private data.
Third, many existing privacy-preservin preserving techniques can be directly integrated into our DecpAFL, en- abling the server to operate without requiring access to each client's 's specific matrices. Specifically, in our DeepAFL, the server only needs to utilize the aggregated values of the clients' uploaded ma- trices for constructing the global classifier and residual block, without needing the individual client upload results. Therefore, this can be regarded as a form of Secure Multi-Party Computation. Exist- ing Secure Aggregation Protocols (Bonawitz et al., 2016 2017 So et al., 2023) can be employed to aggregate clients' uploaded matrices without accessing their specific matrices. Additionally, tech- niques such as Homomorphic Encryption (Hu & Li 2025) and Differential Privacy (Hu et al. 2024) can also be integrated into our DeepAFL to further preserve client privacy.
22

---

Published as a conference paper at ICLR 2026
D APPENDIX FOR EFFICIENCY ANALYSES
In this section, we provide detailed efficiency analyses of our DeepAFL, focusing on the computa- tion and communication complexities on both the client and server sides. Overall, for constructing all 7' layers, the total computation and communication complexities of each client are O (de + dæ + +))) and (T + df)). while those for the server are ( + of + dz d. + dz and O(T respectively. Notably, although the complexities of our DeepAFL are approximately T times greater than those of AFL to construct a deep network for superior performance, they remain far lower than those of gradient- based methods. 'he detailed analyses are as follows:
Analysis of Computation Complexity: Here, we thoroughly analyze the computation complex- ity within our DeepAFL. Notably, each layer's construction within our DeepAFL. merely requires single-round lightweight analytic computation by the clients and server, obviating the need for multi- round gradient-based iterative updates.
The Client Side: First, for the construction of each layer 1, each client k computes its local Feature Auto-Cone Coro-eCrellaion Matrix G and Label that & E R N, xdg and Y E IN the computation complexity of the above calculations is df + d. Second, each client further constructs its local hidden random features Fř and local residual matrix R, via (14), with complexity of ON + CN ). Third, each client computes its local Hidden Auto-Correlation Matrix II, and Residual Cross- Correlation Matrix Y via (1). Given that F E RNNxx and R E RN the computation complexity of executing (5) can be calculated ON + C. Fourth, the client updates the next-layer feature $ with complexity of (N In summary, the complexity for each client construct each layer is OCCN ( + +N (dz + da + df)). and the overall complexity for constructing all I layers O(TCN& + d4) + 1 (dC + dg + df)).
Matrix H via (1I). Given
The Server Side: First, for the construction of each layer L, the server aggregates all received (G [-= and (H*) ] and subsequently derives the global classifier W. via (12) and 13). Given that G Rdoxds and H e Xd the computation complexity of these calculations is Oa + Kd Cdly + KCds). Second, the server aggregates all received { {II JR and { via (6) with a complexity of OK + KCdp). Third, the server performs spectral decompositions and subsequently derives the transformation matrix 4441 via (7) and (18) with a computation complexity of O( + d CCdß + + + did4 (d; + da +C).. In summary, the complexity for the server to construct each layer is Old and the overall complexity for constructing all T layers is ( de + T(C - () + de)) + Td + do + C) + TKC(d= + do)).
(C) + (dr do +C) KC (Cdd++ dd))) -C)-KCCC
Here, we further thoroughly analyze the communica- tion complexity within our DeepAFL. Notably, DeepAFL requires only a single communication exchange between the client and server for constructing each layer, in contrast to gradient-based methods that depend on multiple rounds of communication.
Analysis of Communication Complexity:
The Client Side: According to [ during the construction of each layer t, each client k is only required to upload its local Feature Auto- Auto-Correlation Matrix G Label Woos-Cooreelatio Ma- trix H Hidden Auto- Correlattnl Matrix II. and Residual Cross-Correlation Matrix T.. Given that G E Rd xda. H E d xC II: E Rde,,d, and Y E TRde xC the client's 's total communica- tion complexity for constructing all T layers can be derived as ( + di) + TC(ds + dr)).
The Server Side: For each layer I,, the server similarly needs only to transmit once to all clients the global classifier W. E Rd xC and the transformation matrix Ω E Rdexds, with a communication complexity of O K du (C + dr)). Consequently, the server's total communication complexity for constructing all T layers can be expressed as OT dr)).
Simplification of Complexity Results: Here, we further simplify the complexity results based on the relative magnitudes of their constituent terms. Specifically, in practical scenarios, the total number of layers T. the number of clients K, and the number of classes C are substantially smaller than the sample size N₂ in each client's local dataset, as well as the feature dimensions dæ and dr. Consequently, the computation complexity for each client and the server can be simplified to OTT (dg + dødr + d))) and (T + de + dr + d))))). Similarly, the communication complexity for each client and the server can be simplified to ( + df)) and ((TKd4d))
23

---

Published as a conference paper at ICLR 2026
E APPENDIX FOR EXPERIMENTAL EVALUATIONS
E.1 ADDITIONAL RESULTS FOR EXPERIMENTAL EVALUATIONS
Table 2: Performance comparisons of the top-1 accuracy (%) among our DeepAFL and the baselines, on the CIFAR-10. The best result is highlighted in bold, and the second-best result is underlined. All the experiments were conducted three times, and the results are shown as & Error- All the improvements of our DeepAFL were validated by Chi-squared tests at the level of <
0.05.
The results of all baselines are directly obtained from the given benchmark in AFL (He al. 2025
CIFAR-10
Baseline
Non-IID-1
Non-IID-2
0.05
FedAvg
2017
64.022..18
60.5210.3
68. 10.13
37.8110.00
FedProx
2020
64.07
60.3910.00
68.46
57.61
MOON
202
63.84.0.03
60.18
57.72 00.17
FedGen
2021
64.14.0.24
60.65
68 28.22
57. 37.0220.18
FedDyn
2021
64.77.2.11
20.35
73.50.0.11
64.07
FedNID
2022
64.6420.02
61.16 a1.1620.39
70.24g0.11
58.7720.18
FedDisco 2023b
63.83go.08
59.901o.a0
65.04_0.11
58.7810.02
AFL. 2025
80.75.0.00
80. 80.7510.00
80.75..0.0
80.75.0.00
DeepAFL (T' 5)
85.20g..05
25.20
20.20_0.05
85, 25.2020.05
DeepAFL (T' 10)
85.99320.09
80.93
$5.93.0000
85.93
DeepAFL. (T 20)
86.43 10.07
86, 86.43 10.07
86.43
86.43 10.07
Improvement f
5.68 p<00.05
5.68 p<00.0
5.68 pc0.00
5.68 p<c0.05
Table 3: Accuracy of our DeepAFL. under varying regularization parameters A.
Datasets
Layers
== 0.01
A 0.05
À A 0.1
A 0.5
x=
A= 5
>= 10
85.14_o.05
85.15
85.14ga.08
85.11 20.0s
85. 85.13
25.17
85.20 20.05
CIFAR-10
T 10 T 20
85.87 2.08
85.90=a.08
85.86
85.92
85.91 20.0s
85.92
85.93
86.34_o.07
26.01
86.37
86.27
86.33
86.3920.08
86.43
64.61
64.61=o.01
a0.02
64.59
64.61
64.622.0.02
64.64g0.01
64.66
CIFAR-100
T = 10
65 0..95_0.07
65.96
65.95
65.94
65.94
65.91
65.91
06.91_0.00
20.06
66.92229.06
66.96
66.94
20.77
66.95
T ==00
60.23_..02
60.24.0.02
60.25
60.24gn.02
60.222g.02
60.26.20.2
60.23
Tmag-em
T = 10
61.34_0.08
61.33
61.35ga.07
61.36.0.07
61.3440.0s
61.32 os
61.32g0.07
62.34_0.02
62.03
62.344g.02
62.32
62.35
62.33
62.34go.01
Table Accuracy our DeepAFL. under varying regularization parameters 7.
Datasets
Layers
0.01
0.05
0.1
? 0.5
5
7
10
5
85.06_o.05
14.00
85.15
_..04
85.10.0.oz
85.1010.01
85.0410.05
10.04
CIFAR-10
T = 10
85.87 0.00
85.90
85,91
_.26
85.8610.10
85.8410.0s
85.7810.0s
85.65
86.21_0.00
86.21
86.32..0.0
86.37
86.3310.0s
86.2610.07
86.23
T
64.72+0.07
64.66=a.02
64.63-g0.o1
64.58
64.49
63.77+0.07
+0.10
CIFAR-100
T 10
65.96
65. 20.06
65.95
65.81
65.69
64.82+0.15
64.26+0.15
66.98=0.04
66.04
66.944n.08
66 66.82+n.07
66.7470.07
66, 66.20+0.00
65.61 65.61
T 60.31
60.28
60.2449.02
60.13+n.ou
59.8949.01
59.0440.04
58.41+0.05
Tiny-ImageNe
T 10
61.37 20.07
61.36 61.3620.07
61.35
61 61.28
61.1910.11
60.4440.0s
59. 30.65
T
62.35 20.01
62.33
62.26-2a.0
62.24gn.os
62.1440.02
61.53
60.98
24

---

Published as a conference paper at ICLR 2026
Table 5: Accuracy of our DeepAFL under varying activation functions af-).
Datasets
Layers
None
GELU
ReLU
LeakyReLU
Tanh
Hardwish
Softshrink
83.2310.10
85.13
84.37
84.4910.12
84.422o.07
84.78
13.7
CIFAR-10
T = 10
83.081o.17
85.89 85.89
85.04
84.5410.49
84.9220.12
85.41
04.13
T 20
$2 20.99
86.3420.05
10.47
83.85
85 25.6220.m
86, 86.0550.00
84 24.25
T
60.82+0.00
64.62
64.26=00.22
20.20
63 133.2820.12
64.02+0.02
62 10
CIFAR-
10
60.82+a0.09
65.98
65.39
65.48+0.10
64 =0.17
65.07=a.20
63. 13.11
T 20
60. 60.83+0.00
66.95
66.6 26.68
66. 06.72+0.10
64.0221.02
66 86.0440.06
64.78
=
56.7 76.77
60.23
60.17
60.16g0.02
58.77 58.77=0.04
59.40ga.04
58 80.0.77
Tiny-ImageNet
T 10
56. 70.0.
61.33
61.35
61.36
59 29.68
60.48+0.00
59 39.8320.04
T 20
56. 20..00
62.33
=2.31
62.31
60.74.o.n4
61.47=a0.01
60. 200000
Table 6: Accuracy of our DeepAFL under varying dimensions dg. For comprehensive and consis- tent analysis, all the experimental results were obtained by fixing the other dimension dp at 1024.
Datasets
Layers
de
27
do
2
de
2
de 22
de
2
de
212
de 2
T 5
81,48_0.30
83.31
84.58.0.08
85.07=0.02
85.48
85.76
80.37
CIFAR-I0
02.28
85.55 84.41
85.96
85.94
85.9440.12
80.77
T 20
82.68
84.95
86.01_0.01
86, 26.28
86.41
86.48
83.1 83.1220.3
T 5
57 17.61_ox 27.61
62.21 62.21
64.06
64.633ooozz
65.39
66 66.4610.1
67.46
CIFAR-100
T
58.20 08.20_o.19
03.45.0.19
65.29
65.93
66.3610.04
67.20
67.1920-44
T = 20
58.51
64.2920036
66.6420.10
66.93
67.39 67.3920:06
67.8520.05
67.9420-02
T = 5
44.94_0.24
20.14
29.29
60.21
60.7710:25
61.84 19
62.75
Ting-ImageNel
T = 10
44.11_o.11
57.51
60.68.0.24
61.43 21.43
61.90
63.13 62.42.0.24
T 20 43. 43.10_o.19
58.09
58.09
61.70
62.33
62.88
63.1 63.11
63.70.0.19
Table 7: Accuracy of our DeepAFL under varying dimensions dp. For a comprehensive and consis- tent analysis, all the experimental results were obtained by fixing the other dimension da at 1024.
Datasets
Layers
dr 27
de 2
de = 2
de 220
de = 2
d: 212
dp 21
T = 5
83.48_0.10
83.76
85.14.n.04 84.24.a.14
66.1620.00
85. 15.55 26.51
CIFAR-10
T
83.58_o.11
24.03
84.84.a.os
85.89.0.00
86.50.n.13
86.42.0.04
84.90
T 20
83.83_a.11
84.4920.00
85.34_0.01
86.31
86.53
85.59
83. 13.40
T = 5
61.2220.00
61.82 61.8220.1
63.1120.11
64166220.o1
66.4922.12
67.81
67
CIFAR-100
T' 61.54_0.10
20.57
64.08220.04
65.9620.0s
67.592g.06
67.9 67.92
66. 12:00
T' = 20
61.92
63. 63.41
63.23-a..1
66.942n.0s
68.20
67. 07.14g0.07
64.10go.10
06.92_0.02
67.55
68.69
60.24.n.02
61.8210.07
63.36
10.82
Tiny-ImageNet
T 10
67.23_
68.30
59.58_a.0s
61.34.0.07
62.89 10.06
63.84
62.89.0.04
20
57.79_...0
59.18
60.8619.03
62.34
63.85 10.11
63.4810.03
60.81 10.08
Table 8: Accuracy of our DeepAFL under varying dimensions de and dp. For a comprehensive and consistent analysis, all the experimental results were obtained by ensuring de = dr d.
Datasets
Layers
d= 2
d = 2"
d= 2
d = 210
d 2
d= 212
d 213
T == 5
77.01gom
80.6220.17
85.04 83.54_a.c9
86. 19.19.
86.54...04
CIFAR-10
T
77.9610.11
11.6
84.24_o.0s
85.77.n.04
86.58
86.47.0.12
T 20
78.69
82.38
85.07
86.22
86.71
84.7620.84
T' 5
49.88go.27
57 07.15g0.10
61.74
641663-n.00
66.86-20.13
68.59
CIFAR-100
T 10
51.36
58. 68.76
63. 00.222.0.2
85.96-20.06
68.20 08.2044..0
69.10
T 20
53.2 53.2140.10
60. 60.26go.07
64.72-0.11
66.9 66.93+a.08
68, 08.75-+0.06
69.12
53.1110.13 44.94_o.23
60.2210.03 67.44_...s
62.29.1..0
63.95 10.07
Tmag-Tm
T 10
44. 14.2010..0
54.47 10.4
58.72
61.1710.01
63. 13.11:.m 13.131goo
64.63 10.10
T 20
13.8310.12
56.1810.11
59.92_..02
62.34
64.08
64.68.1004
25

---

Published as conference paper at ICLR 2026
Table 9: Analyses of our Deep representation learning capability on the CIFAR-IO with in-
creasing network depth. "Training Acc" and "Testing Acc" show the accuracy (%) on the training
and test sets. "Time Cost" denotes the required training time (s). The symbol A represents the dif- ference between two consecutive cases, which reflects the marginal effect of deepening the network.
Layers
Training Acc
Triining
Testing Acc
resting Ace
Time Cost
Time
AFL
83.81
$0.75+00.00
52.36
T 0
85.35
1.54
83.29 2:0.20
2.34
58.02
5.66
T 5
89.03
0.682
85. 85.13+0.00
1. 1.84
65.08
7.06#2.22
10
90.61 +0.05
1.58
85.89+00.00
0.75
71.91 +0.28
6.83
T 15
91.80...00 1.0000
1.19
86.1810.00 1118...08
0.29 10.04
78,86
6.95
== 20
92. 74
0,9 0.94.1.01
86.34400.00
0. 0.16
85.71
6.85.1.11
25
93.49 20.02
0.7540.01
86.40200.00
2.06
19.19
6.86
T 30
94.1020.00 03
0.61
86.4620.00
0.06
99.29 20.33
6.72.t0.19
T = 35
94.63
0.53
86. 86.72+0.00
0.26
105.79
6.50
40
95.08 03
0.4410.02
86.7610.06
0.04
112.4010.21
6.61
T 45
95. 44 03
0..0
16.07
0.01
119 119.00
6.60
T = 50
95.79
0.36
86. 36.72+0..0
-0.05
125.72
6.72.0.08
Table 10: Analyses of our DeepAFL's representation learning capability the CIFAR-100 with increasing network depth. "Training Acc" and "Testing Acc" show the accuracy (%) on the training
and test sets. "Time Cost" denotes the required training time (s). The symbol A represents the dif- ference between two consecutive cases, which reflects the marginal effect of deepening the network.
Layers
Training Ace
Training
Testing Acc
Tesinn Ace
Time Cost
A Time Cost
AFL
61 61.5510.00
58.5610.00
/
50.05
T 0
65. 65.66.2.00
4. 10.04
60.8110.15
2.25 10.15
56.90
6.85
T = 5
74.93.2004
9. 0..74
64.62 64.62+0.00
3.8 3..0s
66.4840.21
9.58
10
79.41 10.08
40.04
65.98
4..6
20.25
8.77
T 15
82. 22.69.20.00
3.28
66.5940.00
0.61
83.50
8.25
T = 20
85. 15.15:20.0
2.45
66.95+0.04
0. 0.36
91.74+0.29
8.24
25
87.21 1:0.02
2.06
67.4010.10
0..5
100.3611..7
8.62.0.11
T 30
88. 8..44...00
1.63.10.03
67. 67.7110.06
0..0
108.37 10009
8.01
T = 35
90.14
1. 1.30
67.97
0.26
116.50+4.19
8.13
40
91.29 2.0099
4.152
68.1940.07
0.22
124.74 124.74110.17
8.24
T 45
92. 12.1512..1
0.86
68.32 11
0.13
133.0620.00
8.32
T = 50
92.9310.06
0.7810.03
68.51 18.00
0.1910.02
141.38 [41.38110.88
8.32.0.03
Table 11: Analyses of DeepAFL's representation learning capability the ImageNNt with increasing network depth. "Training Acc" and "Testing Acc" show the accuracy (%) on the training and test sets. "Time Cost" denotes the required training time (s). The symbol À represents the dif- ference between two consecutive cases, which reflects the marginal effect of deepening the network.
Layers
Training Acc
Training Aoe
Testing Acc
Tosing Ape
Time Cost
! Time Cost
AFL
57.39 00
54.67
125.31 125.3110.3 10.38
T 0
60.30.o.00 02
20.02
56.72 (6.7244.11
2.05 2.05
130. 22
5.41
T = 5
06.8910.06
6.5910.02
60.244..02
3. 3.52
145.73. 44..7340.13
15.01
10
70. 70.14.0.00 1..03
3.2411002
61.34410.13
1.10 1.10
460.75.0.22
15.0220.03
T 15
72. 64
2.50+0.02
61.94
0.60
175. 76..44
15. 15.01
T = 20
74.75
2.11
62.34+
0. 0..00
190.01
14.25
25
76.58 1...0.000 03
1.83++0.01
62.6640.00 05
0.32+
201.6440.12
14.63
T 30
78.31
1.73++0.02
62.9440.00
0.29
219.5620.11
14.92=2.00
T = 35
79.87 1..10.000
1.5610.01
63.2440.16
0.30
234.9910.28
15.53
40
81.32 03
1.45.10.02
63.48 10..18
0.24
249.95.0.13
14.94..0.0
T 45
82. 22.622
1.30
63. 03.65
0.16
264.31
14.36
T 50
83.82
1.20
63.7440..0
0.10
279.68
20.14
15.37
26

---

Published as a conference paper at ICLR 2026
Table 12: Ablation study of our DeepAFL's training accuracy on the CIFAR-I0. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection B,, (3) Ablating the activation function (o.), and (4) Ablating the trainable transformation 1 The value in parenthe- ses (L) indicates the performance drop compared to our full DeepAFL under identical conditions.
Layers
DeepAFL
Ablation (1)
Ablation (2)
Ablation (3)
Ablation (4)
T ==00 0
85.35 85.35
(1 (,0000)
85.35 (1 (.000
85.35 (L 0.00)
85.35 (L 0.000
T = 5
85.23 89.03
(L.3880)
86.8620.0a (U.7.17)
85.35
(1 3.68)
86. 16.13
(1 2.90)
T 10
90.61.0.05
85, 85.2820.05
(J 5.33)
86.97 (L 3.64)
85.35
(L 5.26)
86.30
(L 4.31)
T' = 15
91.80
85.321..07
(J 6.48)
87.03
(L.4.77)
85.3510.02
(L 6.45)
86.3810.04
(L 5.42)
T 20
92.74
85.33
(J 7.41)
87.06 (1 5.68)
85.33
(1 7.39)
86, 86.43+0.04
(L 6.31)
T 25
93.49 10.02
85.34
(J 8.15)
87.07 (1 6.42)
85.3510.02
(1 8.14)
86.4510.03
[1 7.04)
= 30
94.10
85.37
(L (.73)
87.09 (L 7.01)
85.35
(L 8.75)
86.48
(L 7.62)
T 35
94.63-0.03
85.39_0.06
(U.9.24)
87.10 (L 7.53)
85.36
(1 9.27)
86.45.0.04 (L 8.18)
T = 40
95.08 10.03
85.41 (L (.667)
87.10 (L 7.98)
83.35 (L 9.73)
86.47
(L 8.61)
T 45
95.43=0.03
85.40+0.05 (4 10.0)
87.01 (L 8.32)
85.35+0.02 (4 10.1)
86. 86.45+0.02
(4 8.98)
T 50
95.79 L0.00
85.42 (1 10.4)
87.11 (1 8.68)
85.35 [1 10.4)
86.41 86.4710.00 (1 9.32)
Table 13: Ablation study of our DeepAFL's testing accuracy on the CIFAR-10. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection B,, (3) Ablating the activation function (.),, and (4) Ablating the trainable transformation S2,+1. The value in parenthe- ses (L) indicates the performance drop compared to our full DeepAFL. under identical conditions.
Layers
DeepAFL
Ablation (1)
Ablation (2)
Ablation (3)
Ablation (4)
0
83.29
83.2920.20 (1 0.000
83.2920.20 (1 0.00)
83.29 (L 0.00)
83.29 20 (L 0.00)
T 5
85.13
83.03
() (1.2.10)
83.9210.21 (4 1.21)
83.33 (1 1.80)
83.8710.16 (4 1.26)
T = 10
85.89 =5.89
83.0620.07 (J (..3)83
83.99 (1 1.90)
83.34 (1 2.55)
83.95 14 (1 1.94)
T' 15
86.18 10.08
83.0510.08
(4.3.13)
84.00 (4 2.18)
13.3 (L 2.85)
84.08 10.28 (4 2.10)
T 20
86.34_0.06
83.03
(J 3.31)
84.01 (1 2.33)
83.32+a.2a [1 3.02)
84.02+0.22 (1 2.32)
25
86.40.0.06
82.99
(L:3.41)
84.00 20 (1 2.444
83 (L 3:07)
(L 84.08
(.32)
T 30
86.46_0.00
83.02+a.11
(U.3.44)
84.01 (1 2.45)
83.33 (1 3.13)
84.0640.34 (L 2.40)
T = 35
86.72=0.06
82.94
(.78)
83.99 (1 2.73)
83.3320.24 (1 3.39)
84.03 (L 2.69
T 40
86.76_0.06
82.93
(..3))
84.021..25 (1 2.74)
83.33 (1 3.43)
84.00 (L 2.76)
T' = 45
86.77
82.96 15
(1.3.81)
84.01 (1.7.76)
83.3440.23
(1 3.43)
83.91
(L 2.86)
50
86.7210.06
12.97
(1
84.022..28 (1 2.70)
83.3410.24 (1 3.38)
83.97 10.25 (1 2.75)
Table 14: Ablation study of our DeepAFL's training accuracy on the CIFAR-100. We evaluate four distinct ablation models to explore the contributions of our key components in the representations: (1) Ablating the residual skip connection, (2) Ablating the random projection B,, (3) Ablating the activation function (-)., and (4) Ablating the trainable transformation 24+1. The value in parenthe- ses (L) indicates the performance drop compared to our full DeepAFI. under identical conditions.
Layers
DeepAFL
Ablation (1)
Ablation (2)
Ablation (3)
Ablation (4)
T 0
65.66
65.6610.04 (1 0.00)
65.6610.04 (1 0.00)
65.6610.04 (L 0.00)
65.6610.04 (L 0.00)
T' = 5
20.03
20.04 (L (1.9.12)
69.80ga.1
(L.3.1
65.66
(L 9.27)
65.67
(L (..9266)
T 10
79.41 L0.08
66.32
(J 13.1)
(1 70.16
9.25)
65.6610.04
(1 13.8)
65.70 10.08
(1 13.7)
T = 15
82.69 =..09
66.60+a.11
(L 16.1)
70.38 (L 12.3)
63.66+a.oz
(L 17.0)
63.72
(L 17.0)
T 20
=0.15
66.86 (L 18.3)
70.46 (L 14.7)
65.66
(L 19.5)
65. 65.72+
(L 19.4)
T = 25
$7.21#0.00
67. 67.10 12 (0.10.)
70.57 (L 16.6)
65.67
(] 21.5)
65.74 10 (L 21.5)
T = 30
18.84
67.25
(1.21.6)
70.63 (L 18.2)
65.6740.03
(L 23.2)
65.7740.10 (L 23.1)
T 35
90.14=...6
67.40+0.11 (2.77
70.65 10 (1 19.5)
65.6740.04 (1 24.5)
65.7640.08 (1 21.4)
T' = 40
67.5610.12 91.29L0.00
(3.77.
70.69 (L 20.6)
65.6810.0 [L 25.6)
15.7810.08
(L 25.5)
45
92.15
67. 67.70
(1 24.5)
70.73 (L 21.4)
65. 65.6840.04
(1 26.5)
65. 77.07
(1 26.4)
T 50
92.93
67.80
() 25.1)
70.78 (L 22.2)
65.67
(L 27.3)
65. 65.7610.08
(L 27.2)
27

---

Published as a conference paper at ICLR 2026
Table 15: Ablation study of our DeepAFL's testing accuracy on the CIFAR-100. We evaluate four distinct ablation models to explore the contributions of our key components in the representations:
(1) Ablating the residual skip connection, (2) Ablating the random projection B,, (3) Ablating the activation function (o.), and (4) Ablating the trainable transformation 1 The value in parenthe- ses (L) indicates the performance drop compared to our full DeepAFL under identical conditions.
Layers
DeepAFL
Ablation (1)
Ablation (2)
Ablation (3)
Ablation (4)
T T=0 0
60.81:0.18
60.81 (L 0.00)
60.81 (L (.00)
60.81 (1 0.00)
60.81 1s (1 0.000
T = 5
60.91 6..62
62.4
(U.2.19)
60.80 3.82)
60.80 10 (1 3.82)
T 10
65.98
61.17 10 (J 4.81)
62.5520.06
(L 3.43)
60.81 16 (1 5.17)
60. 60.82+o.05
(1 5.16)
T' = 15
66.5910.03
61.4810.06 (U.5.11)
62.55
(Q-4.04)
60.8210.14 (4 (.77
60.85
(L 5.74)
20
66.95
61. 61.82=0.10
(1 5.13)
62. 62.55
(1 4.40)
60. 60.82+0.16 (1 6.13)
60. 60.84 12
(1 6.11)
T 25
67.40 10.10
61.951..11
(J 5.45)
62.6110.00 (1 4.79)
60.8210.18 (1 6.58)
60.83
(1 6.57)
= 30
62.12 60.01
(4.5.59)
62.60 (415.11)
60.82 (L 6.89)
60.77
(L 6.94)
T 35
67.97
62.26 1s (l.7.71)
62.63 II (L 5.34)
60.83 16 (1 7.14)
60.81 (1 7.16)
T = 40
08.19
62.28
5.91)
62.64 (1 5.53)
60.82 (L 7.37)
60.88 (L 7.31)
T 45
68.32
62.37
(L 5.95)
62.63 (1 5.69)
60.834a.16 (L 7.49)
60.89+a.1 (L 7.43)
T 50
68.51 10.09
62.39 (J 6.12)
62.64 (1 5.87)
10.84 (1 7.67)
60.89 (1 7.62)
Table 16: Ablation study of our DeepAFL's training accuracy on the Tiny-ImageNet. We evaluate four distinct ablation models to explore the contributions of our key components in the representa- tions: (1) Ablating the residual skip connection, (2) Ablating the random projection B,, (3) A Ablating the activation function
(-), and (4) Ablating the trainable transformation 44 The value in paren- theses (1) indicates the performance drop compared to our full DeepAFL. under identical conditions.
Layers
DeepAFL
Ablation (1)
Ablation (2)
Ablation (3)
Ablation (4)
0
60.30
60.30 (L 0.00)
60.30 (1 0.00)
60.30 (L 0.00)
60.30+a.02 (L 0.00)
T 5
66.89 10.06
60.84
(L 6.05)
63.21 10.00 (L.3.68)
60.30
(1 6.59)
60.31 10.02 (1 6.58)
T = 10
70.14
61.87
(1 8.27)
63.4620.03
(1 6.68)
60.30
(1 9.84)
60.31
(1 9.83)
T' 15
72.64 L0.05
62.53
(4. 10.1)
13.5910.00 (L 9.05)
60.31 10.01
(L 12.3)
60.3210.02
(4 12.3)
T 20
74.75
62.97
(L 11.8)
63.682a.oz (1 11.1)
60.31
(1 14.4)
60.33
(1 14.4)
T 25
26.58
63.30.0.02 (L 13.3)
63.7420.00 (L 12.8)
60.01
(L 16.3)
60.33
(L 16.3)
T = 30
78.31
63.58 (1 14.7)
63.77 (L 14.5)
60.31
(4 18.0)
60.35 (L 18.0)
T = 35
79.87
63.76
(L 16.1)
63.80 (1 16.1)
60.31
(L 19.6)
60.36 (1 19.5)
T 40
81.32 10.11
63.851..02 (1 17.5)
63.821..02 (1 17.5)
60.31 (1 21.0)
60.35 (L 21.0)
T = 45
82.62
63.982a.0z
(L. 18.6)
63.8420.02 (L. 18.8)
60.31 (L 22.3)
60.35 (L 22.3)
50
83.8210:00 L0.00
14.007.0.01 (1 19.8)
63. 63.8610.01 (1 20.0)
10.31 (1 23.5)
10.36 (1 23.5)
Table 17: Ablation study of our DeepAFL's testing accuracy on the Tiny-ImageNet. We evaluate four distinct ablation models to explore the contributions of our key components in the representa- tions: (1) Ablating the residual skip connection, (2) Ablating the random projection B,, (3) Ablating the activation function (.),, and (4) Ablating the trainable transformation 24 The value in paren- theses (L) indicates the performance drop compared to our full DeepAFL. under identical conditions.
Layers
DeepAFL
Ablation (1)
Ablation (2)
Ablation (3)
Ablation (4)
T 0
56.72 10.14
56.721a.14 (1 0.00)
56.7210.14 (1 0.00)
56.7210.14 (1 0.00)
56.7210.14 (1 0.00)
T' = 5
60.24 20.02
57.39+0.11
(.85)
58. 26 (1 2.08)
56.69+a.13 (1.5.
56.69
(1.5)
T 10
61.34 10.13
58. 58.421o.16
(1 2.92)
58.22 (1 3.12)
56. 16.70 (1 4.64)
56.6910.14
(1 4.65)
T = 15
61.94_..00 20.08
59.01
2.93)
58.20 (1.3.74)
56. 56.70+a.1a (1 5.24)
56.69+a.12 (.25)
T 20
59.21 =2.34
(L 3.13)
58.25 (L 4.09)
56. 20.70 (4 5.64)
56.68 (1 5.66)
T 25
62.66
59.37 14 (L 3.29)
58.25
(L 4.41)
56. 36.7020.14 14 (1 5.96)
56. 75.704o.. 10 (1 5.96)
T = 30
62.94...00 L0.03
59.491..10
(L.3.45)
58.2410.17 (1.4.
56. 20.701 (1 (.24))
56.7010.12 (L 6.24)
T 35
63.24-0.16
59.54+0.07
3.70)
58.27 (1 4.97)
56. 36.7040.14 (1 6.54)
56.72 12 (1 6.52)
T = 40
63.48 10.18
59.661a.08
(.82)
58.3410.18
(L.4.14)
56.7010.14 (L 6.78)
56.73 (L 6.75)
45
63.65
59.72+a.10
(L 3.93)
58.35 (L 5.30)
56. 26.704 (1 6.95)
56. 36.72+a.10 10 (1 6.93)
T 50
63.74....44
59.81 10.11 (.933)
58.39 (L 5.35)
56. 16.70 (1 7.04)
56. 10.08 (L
7.02)
28

---

Published as a conference paper at ICLR 2026
Table 18: Training accuracy comparison of our DeepAFL against the other four deepening strategies
for the analytic networks on the CIFAR-10, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/
activated random feature & label encoding, All the strategies share the same zero-layer features to
ensure fairness. The value in parentheses (4) indicates the performance lags behind our Dee
Layers
DeepAFL
Strategy (1)
Strategy (2)
Strategy (3)
Strategy (4)
T 0
85.35
85.35 (L 0.000
85.35 (1 (,0.00)
85.35
(L 0.000
85.35 (L 0.000
T = 5
85.35 89.03
(J.3668)
85.36
(.66.67
85.68
(1 3.35)
85.68
3.35)
T 10
90.61.0.05
85.35=0.03
(J 5.26)
85.3620.03 (1 5.25)
85, 85.68
(L 4.93)
85.67
(L 4.94)
T' = 15
91.80
85.3510.0a
(J 6.45)
85.35 (L 6.45)
85. 15.00
(L 6.10)
85.6810.08 (L 6.1)
20
92.74
85.33
(J 7.39)
85.35 (1 7.39)
85. a5.00
(1 7.04)
85.67+0.01 (L 7.07)
T 25
93.49 10.02
85.3510.03
(J 8.14)
85.3510.03 [1 8.14)
85.71
(1 7.78)
85.6810.02 02 [1 7.81)
= 30
94.10
85.36
(4 8.74)
85.35 (L 8.75)
85.71
(L 8.39)
85.67
(L 8.43)
T 35
94.63-0003
(1 85.36_0.03
(U.9.27)
85.35 (1 9.28
85.71
(1 8.92)
85.68 (L 8.95)
T = 40
95.08 10.03
85.35 (L (.733)
85.35 (1 9.73)
g3.71
(L 9.377)
85.68
(L 9.400
T 45
95.43=0.03
85.35+0.03 (4) 10.1)
85.35 (4 10.1)
85.71
(4 9.72)
85.69
(4 9.74)
T 50
95.79 10.00
85.36 (J 10.4)
85.35 (1 10.4)
85.7010.01 (1 10.1) 85.6810.00 (1 10.1)
Table 19: Testing accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the CIFAR-10, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature label encoding. All the strategies share the same zero-layer features to
ensure fairness. The value in parentheses (4) indicates the performance lags behind our DeepAFL.
Layers
DeepAFL
Strategy (1)
Strategy (2)
Strategy (3)
Strategy (4)
0
83.29
83.30 (1 0.00)
83.3020.20 (L 0.00)
83.30 20 (L 0.00)
83.30 20 (L 0.00)
T 5
85.13
83.30 () 1.83)
83.31 (4 1.82)
83.6210.20 (4 1.51)
83.61 10.20 (4 1.52)
T = 10
85.89
83.29 (J 2.60)
83.31 (1 2.58)
83.63
(1 2.26)
83.60 20 (1 2.29)
T' 15
86.18 10.08
83.3210.20 (4 2.86)
83.31 10.23 (4 2.87)
83.6210.20
(4 2.56)
83.60 19 (4 2.58)
T 20
86.34_0.06
83.29 (J 3.05)
83.31 (1 3.03)
83.62
(1 2.72)
83. 83.62+0.2z (1 2.72)
25
86.40.0.06
83.2910.24
(1.1.11)
23 (..9)
83.63 (4 2.77)
(L 83.63
2.77)
T 30
86.46_.000
83.31 (U.3.15)
83.29 (l.3.17)
83.64.0.21 (4 2.82)
83.60 (L 2.86)
T = 35
86.72=0.06
83.30
(.44))
23.31
(144)
83.6420.22 (1 3.08)
83.59 20 (1 3.13)
T 40
86.76
83.31
(1
83.32
(L
83.6610.20 (1 3.10)
83.6210.20 (L 3.14)
T' = 45
86.77
83.31
(4.3.46)
83.30
(L.3.47)
83.65 (L 3.12)
83 03.60+o.22 (L 3.7)
50
86.7210.06
13.21 3.44)
13.30 (1 3.42)
83.6410.22 (1 3.08)
83.60 (1 3.12)
Table 20: Training accuracy comparison of our DeepAFL against the other four deepening strategies
for the analytic networks on the CIFAR- 100, including (1) cascades w/ random feature, (2) cascades
w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/ activated random feature & label encoding. All the strategies share the same zero-layer features to
ensure fairness. The value in parentheses (L) indicates the performance lags behind our DeepAFI..
Layers
DeepAFL
Strategy (1)
Strategy (2)
Strategy (3)
Strategy (4)
T 0
65.66
65.6610.04 (1 0.00)
65.6610.04 (1 0.00)
65.6610.04 (L 0.00)
65.6610.04 (1 0.00)
T' = 5
74.93
65.6440.06
(1.9 (.99.9))
65.58
(1.9.35)
66. 18...8
(L (.75)
65.30+a.05 (L 9.63)
T 10
79.41 L0.08
65.64 05.641o.00
(J 13.8)
65.58 10.08 (1 13.8)
66.671..08
(1 12.7)
65.39 10.08
(1 14.0)
T = 15
82.69 =0.09
65.64
(L 17.0)
65.58+a.0m (L 17.1)
(L 67.11
15.6)
65.51 #0.02 (L 17.2)
T 20
85.15 00.03
65. 85.6440.06
(1 19.5)
65.58 (L 19.6)
67.52+a.11
(4 17.6)
65.60 (L 19.6)
T = 25
87.21 20.02
65.6420.05
(1.26.6)
65.57 (L 21.6)
67.86 13 (L 19.4)
65.70 12 (] 21.5)
T = 30
88.84L0.00 10.06
65.641..05
() 23.2)
65.5610.07 (L 23.3)
68. 1410.11 (L (1.20.7)
65.7410.12 (L 23.1)
T 35
90.14=...0
65.6 65.6440.05 (1 24.5)
65.5820.06 (1 24.6)
18.4040.12 (1 (1.77)
65.8440.01 (1 24.3)
T' = 40
91.29L0.00
65.64
[L 25.6)
65.58 (L 25.7)
68.6410.14 (L 22.6)
65.9510.15 (L 25.3)
45
92.15+p..11
65. 05.6440.05
(1 26.5)
65.58 (L 26.6)
68. 68.82+a.1 (1 23.3)
66.044a.17 (1 26.1)
T 50
92.9310.06
65.64
(L 27.3)
65.59 (L 27.3)
68.99 10 (1 23.9)
66.1610.13
(L 26.8)
29

---

Published as a conference paper at ICLR 2026
Table 21: Testing accuracy comparison of our DeepAFL against the other four deepening strategies
for the analytic networks on the CIFAR-100 100, including (1) cascades w/ random feature, (2) cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades w/
activated random feature & label encoding, All the strategies share the same zero-layer features to ensure fairness. The value in parentheses (4) indicates the performance lags behind our Dee
Layers
DeepAFL
Strategy (1)
Strategy (2)
Strategy (3)
Strategy (4)
T T=0 0
60.81:0.15
60.81 (4 0.00)
60.81 (1 (.00.
60.81 (1 0.00)
60.81 15 (1 0.00)
T = 5
60.80 64.62
(..3)82)
60.75 (L8787)
61.23
(1 3.39)
60.57 (4 4.05)
T 10
65.98
60.78 10 (J 5,20)
60.74go.11
(1 5.24)
61.67
(1 4.31)
60.69 (L 5.29)
T' = 15
66.59
60.781a.12 (J (.81.81)
60.74 (L (..5.85)
61.9810.0a
(L 4.61)
60.8510.02
(L 5.74)
20
66.95
10 (J 6.16)
60.73.a.1 (1 6.22)
62.21
(1 4.74)
60.95
(L 6.00)
T 25
67.40 10.10
60.7810.10 (J 6.62)
60.721..11 (1 6.68)
62.31
(1 5.09)
60.99 08 (1 6.41)
= 30
67.71 40.06
60.78 (L 6.93)
60.73.a.11 (L 6.98)
62.53 (L 5.18)
61.12 (L 6.59)
T 35
67.97
60.78
7.19)
60.7220.10 10 (L 7.25)
62.73 12 (1 5.24)
61.1620.00
(L 6.81)
T = 40
68.19
60.78 (J 7.41)
60.74_0.10 (L 7.45)
62.82.a.11 (1 5.37)
21.20
(L 6.99)
T 45
68.32
60.78
(4 7.54)
60.73.a.1 (4 7.59)
62.96+0.1 (L 5.36)
61.27
(4 7.05)
T 50
68.51
10.09
10.77 (1 7.74)
60.71 (1 7.80)
63.0510.11 (1 5.46)
61.3710.08 (1 7.14)
Table 22: Training accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the cascades w/ random feature & label encoding, (3) cascades w/ activated random feature, (4) cascades
including (1) cascades w/ random feature, (2)
w/ activated random feature & label encoding. All the strategies share the same zero-layer features to
ensure fairness. The value in parentheses (4) indicates the performance lags behind our Aee
Layers
DeepAFL
Strategy (1)
Strategy (2)
Strategy (3)
Strategy (4)
0
20.30
60.30
(1 0.00)
60.30 (L 0.00)
60.30 (L 0.00)
60.30+0.02 (L 0.00)
T 5
66.89 10.06
60.20
(L (.6.)
59.84.0.02
7.05)
59.9210.04
(L 6.97)
56. 14.00 (4 10.8)
T = 10
70.14
60.2 60.20=a.o1
(1 9.94)
59.83
(1 10.3)
60.00
(L 10.1)
54.7840.07
(1 15.4)
T' 15
72.6410.05 10.05
60.21 10.01
(4. 12.4)
59.8210.04
(4. 12.8)
60.08
(4 12.6)
14.40
(4 18.2)
T 20
74.75
60.20
(1 14.5)
59.81 (1 14.9)
60.17
(1 14.6)
54.23+a.11
(1 20.5)
T 25
76.58_20.03
60.1910.01
(L 16.4)
59.80.0.02 (L 16.8)
60.25
(L 16.3)
54..12.0.1 (L 22.5)
T 30
78.31
60.21
(1 18.1)
59.81 (L 18.5)
60.33 (1 18.0)
54.0330.12 (L 24.3)
T = 35
79.87
60.21
(1 19.7)
59.81 (1 20.1)
60.44 (1 19.4)
54.06go. 12 (1 25.8)
T 40
81.32 10.11
60.20
(1 (1.11.)
59.80 (4 21.5)
60.521..05 (1 20.8)
53.99 (1 27.3)
T = 45
82.62
60.20 60.20
(4.22.4)
59.80 (L 22.8)
60.622a.o6 (L 22.0)
54.07 (L 28.6)
50
83.82 L0.00
60.201..02 (1 23.6)
59.78 (1 24.0)
60.711..05 (1 23.1)
54.05 (1 20.8)
Table 23: Testing accuracy comparison of our DeepAFL against the other four deepening strategies for the analytic networks on the Tiny-ImageNee including (1) cascades w/ random feature, (2)
cascades w/ random feature label encoding, (3) cascades w/ activated random feature, (4) cascades
w/ activated random feature & label encoding. All the strategies share the same zero-layer features to
ensure fairness. The value in parentheses (4) indicates the performance lags behind our Aeep
Layers
DeepAFL
Strategy (1)
Strategy (2)
Strategy (3)
Strategy (4)
T 0
56.72 10.14
56.721a.14
0.00)
56.7210.14 (1 0.00)
56.72 (1 0.00)
56.72 36.7210.14 10.14 (1 0.00)
T' = 5
60.24 20.02
56.69+a.12
(4.3.55)
56.324o.10
(L.3.92
56.31 10.10 (L.3.33
53.264a.12 12 (1 (1.6.8)
T 10
61.34
10.13
56.61
(J 4.73)
56.261..11
(1 5.08)
56.40 (1 4.94)
52.01 1o.08
(1 9.33)
T = 15
61.94 _0.0s
56.62
(.32)
56.25
(L-5.69)
56.49+a.1u 5.45)
51.744a.12 (L 10.2)
T 20
=n.02
56. 56.64.a.14 (1 (170)
56.23 (U 6.11)
56.61 (L 5.73)
51.63+a.00 (L 10.7)
T 25
62.66
56.62 14 (J.6.04)
56.27=9.0n (L 6.39)
56.66 12 (1 6.00)
51.6620. 1a (L 11.0)
T = 30
62.9410.00 10.03
56.631..00 (1.6.31)
56.23 (L 6.71)
56. 36.751 (L 6.99)
51.4740.18 (L 11.5)
T 35
63.24-0.16
56.61+0.14 (J 6.63)
56.222a..4 (1 7.02)
56. 56.79+0.15 (1 6.45)
51.5140.11 (1 11.7)
T = 40
63.48 10.18
56.621a.12 [L 6.86)
56.261..02 (L 7.22)
56.8310.24 [L 6.65)
51.4710.15 (L 12.0)
45
63.65
56.61
(1 7.04)
56. 56.22+a.08 (L 7.43)
56.99 (1 6.66)
51.42+a.0 (1 12.2)
T 50
63.74....4
56.61 () 7.13)
56.23 (L 7.51)
57.11 (1 6.63)
51.4710.12
(L 12.3)
30

---

Published as a conference paper at ICLR 2026
16
Communication Coet (MS)
120
Computatoonl fetional Cost
Testing Accuracy
64-
10
10
Number of Layers
Number of Layers
Number of Layers
(a) Testing Accuracy v.s. Layers
(b) Computational Cost v.s. Layers (c) Communication Cost Layers
Figure 4: Eccuracy-Efficienee balance our DeepAFL on the CIFAR-10.
100
Communcation Cost (MD)
120
Comp Comppttttonnl Coal
Testing Acturacy
80
800
Layers
Number
Number Layers
(a) Testing Accuracy Layers
(b) Computational Cost v.s. Layers (c) Communication Cost v.s. Layers
Figure 5: Eccuracy-Efficieney balance of our DeepAFL on the CIFAR-100.
Teding Accuracy (96)
192
Communcation Coet (MBB
520
Computetiona Cout
182
120
10
Number Layers
Number Layers
Number Layers
(a) Testing Accuracy v.s. Layers
(b) Computational Cost v.s. Layers (c) Communication Cost v.s. Layers
Figure 6: Accuracy-Efficiengy balance of our DeepAFL on the Tiny-ImageNe
DeepAFI (T 20)
AFL
FadRing
60
DeepAFt [7' -7 20)
ARL
FedAug
55
Asturacy (NS)
(96)
00-
Acoursoy
41.01
35.72
20
CIFAR-100
(b) Tiny-ImageNet
Figure 7: Scalability analyses of our DeepAFL. against baselines with different numbers of clients.
DeepARt - Al atoor (1) Abtation (2) + Actarion (3)
Ablation (4)
DeepAF d AMatoon (1) Anterion (2) Atlation (2) Ablatinn (0)
Accuracy (N6)
Accuracy (%)
88
83
Layers
Layers
(a) Training Accuracy
(b) Testing Accuracy
Figure 8: Ablation study of our DeepAFL. on the CIFAR-10.
31

---

Published as a conference paper at ICLR 2026
69
Accuracy
Accuracc
10
20
10
50
10
30
40
Number of Layers
Number of Layers
(a) Training Accuracy
(b) Tiny-ImageNet
Figure 9: Ablation study of our DeepAFL on the CIFAR-100.
DecptF:
82-
Accuracy
Accurasy
10
10
20
40
30
10
30
Number of Layers
Number of Layers
(a) Training Accuracy
(b) Testing Accuracy
Figure 10: Ablation study of our DeepAFL on the Tiny- ImageNet.
57-
Accurapy
85
89
10
20
50
10
Number of Layers
Number of Layers
(a) Training Accuracy
(b) Testing Accuracy
Figure 11: Comparison of DeepAFL with other deepening strategies on the CIFAR-10.
DespAAi
Strtagy
Stastgy Shalegy
DowppA
Accuracy
66
80
60
10
50
10
Number of Layers
Number of Layere
(a) Training Accuracy
Testing Accuracy
Figure 12: Comparison of DeepAFL with other deepening strategies the CIFAR-100.
DeepARt
Stategy Stategy
66
DowepA
Accuracy (%))
Accuracy
70
55
20
50
30
Number of Layers
Number Layers
(a) Training Accuracy
Testing Accuracy
Figure 13: Comparison of our DeepAFL with other deepening strategies the Tiny-ImageNet..
32

---

Published as a conference paper at ICLR 2026
E.2 IMPLEMENTATION DETAILS OF THE EXPERIMENTS
Here, we detail the implementation details of our experiments. Specifically, the number of clients is set to 100 for all methods. To simulate diverse data heterogeneity scenarios, two common non- partitioning setrings are employed: LDA (L.in et al.) 2020) (as Non-IIID-1) and Sharding (Lin et al.) 2020) (as Non-IIDD-2). Under the Non-IIl-1 setting, the dataset is allocated to all clients via the Dirichlet distribution, with the parameter modulating the heterogeneity level. Under the Non-IID- 2 setting, the dataset is sorted by label and divided into equal-sized shards for distribution among clients, where the number of shards per client controls the heterogeneity level. Smaller values of a and both indicate more heterogeneous data distributions. We set a E {0.1, 0.05] and & E {2, 4} for CIFAR-IO, while setting
{0.1, 0.01} and 8 E (5, 10) for CIFAR-100 and Imay-ImageNet.
To ensure a fair comparison, in our main experiments, the results of all baselines are directly taken from the provided data in AFL (He al.) 2025). Furthermore, for all three datasets (i.e., CIFAR-1O, CIFAR- 100, and ImageNe,,,,,), we adopted exactly the same pre-procsssing procedure as AFL., resizing all input images 224 x 224. Regarding the specific parameters of our DeepAFL, we adopt the following default settings. First, we employ GELU as the activation function and set the projection dimensions to dg = d.- 1024 across all three datasets. Additionally, the regularization parameter À is set to 10 for CIFAR-10, 10, and 1 for both CIFAR- 100 and Tiny-ImageNer Then, for the regularization parameter 7, we set to 0.1 for CIFAR-10, and 0.01 for both CIFAR-100 and Tiny-ImageNed To ensure experimental transparency, we provide the detailed parameter settings for each experiment in the extended analyses of our evaluation in Section E.3
As for the metrics employed in our experiments, we adopt the top-1 accuracy on the testing sets as the primary metric for evaluating the performance of all approaches. In addition, since the top-1 accuracy on the training sets also serves as strong indicator of the representation learning capabil- ity, we extensively employ this metric to analyze how the representations of different models evolve with increasing network depth. Moreover, to assess the efficiency of all approaches, required training time (8) and the volume of transmitted data (MB) as indicators of computational and communication cost, respectively. To analyze the marginal effect of increasing network depth, we use the symbol A to denote the difference between two consecutive cases of the the metric.
measure the
All the experiments in our paper are executed three times per setting, and we report the mean and standard error of the experimental results. Furthermore, all the experimental evaluations in this paper are conducted using PyTorch on NVIDIA RTX 4090 GPUs. Notably, for transparency, the related codes will be made publicly accessible as open-sourced upon the acceptance of this paper, allowing the broader research community to verify our findings and build upon our work.
E.3 DETAILED ANALYSES ON EXPERIMENTAL. EVALUATIONS
E.3.1 MAIN COMPARISONS
In our main results reported in T projection dimensions to dg = dp- = 1024 across all three datasets. The regularization parameter À is fixed 10 for CIFAR-10 and at 1 for both CIFAR- 100 and Tiny-ImageNet. Similarly, the regularization parameter & is set to 0.1 for CIFAR-10 and 0.01 for CIFAR- 100 and Imny-ImageNet. To ensure a fair comparison, the results of all baselines in Tables D and 2 are directly taken from the benchmark data provided in AFL (He et al. 2025). It is worth noting that, since our DeepAFL involves random projections, its performance may vary under different random seeds. To mitigate the influence of randomness, all main experiments in this paper were repeated three times, and the results are reported as Mean + Standard Error. The improvements of our DeepAFL were validated by Chi-squared tests, all of which were found to be statistically significant at the p 0.05 level.
employ GELU as the activation function and set the
From the results, we observe that the performance of DeepAFL consistent across different levels of heterogeneity, which empirically supports the heterogeneity invariance in Theorem 1. Moreover, as the network depth T' increases, DeepAFL consistently achieves significant performance gains, empirically supporting its representation learning capability in Theorems 2-3. It is also worth noting that although we report the performance of DeepAFL under T E (5, 10, 20} in the main results, extended
its performance continues to improve as T increases. This is further confirmed by experiments with T e [o, 50). Specifically, on the more complex CIFAR- 100 and ImageNet datasets, the testing accuracy of DeepAFL. at T 50 surpasses that T 20 by more than 1.5%.
33

---

Published as a conference paper at ICLR 2026
E.3.2 INVARIANCE ANALYSES
Here, we conduct detailed analyses to demonstrate DeepAFL's ideal property of invariance to data heterogeneity, As shown in Tables and the performance of all gradient-based methods deterio- rates markedly as data heterogeneity increases (i.e., as the partitioning parameters a or & decrease). In contrast, due to its ideal property of invariance to data heterogeneity, our DeepAFL maintains stable performance. This property of DeepAFl. can be further extended to invariance with respect to the number of clients. As shown in Figure 7 the performance of DeepAFL. remains entirely consistent across all scenarios, while its advantage over gradient-based methods (c.g., FedAAg) be- comes increasingly pronounced as the number of clients K grows. Quantitatively, at K = 100, DeepAFL achieves performance gains of 10.41% and 18.65% over FedAvg on the CIFAR-100 100 and Inteee,,,,et, respectively, which expand to 25.97% and 32.8% K 1000. Notably, the results of AFL and FedAvg in Figure Z also taken from the given data in AFL (He et al. 2025
E.3.3 REPRESENTATION ANALYSES
Here, we provide a comprehensive analysis of DeepAFL's capability for deep representation learn- ing. To this end, we examine how its training accuracy and testing accuracy evolve the number of layers 7' increases, as shown in Tables 1 As observed, on the simple CIFAR-10 10 dataset, Deep- AFL's performance improves steadily, reaching optimality at T 45. On the complex CIFAR-100 and Siny-ImageNee datasets, it scales without overfitting up to T = 50, yielding over 1.5% improve- ment relative I" = 20. This indicates that for datasets with higher complexity, deeper models can be constructed to enhance representation capacity and achieve superior performance. Moreover, a particularly interesting and evident observation is that AFL exhibits consistently low training accu- racy, which can likely be attributed to the underfitting limitations of its simple single-layer linear model. In contrast, DeepAFL is able to significantly improve training accuracy by deepening the network, showing that its representation learning capability can effectively overcome the underfitting issues in traditional analytic learning. In fact, our DeepAFL only requires minimal computational overhead when increasing depth, with more detailed analyses presented in Appendix E.3.4
E.3.4 EFFICIENCY EVALUATIONS
Here, we present the comprehensive efficiency evaluations of our DeepAFL. To highlight its superior balance between accuracy and efficiency. Figures 3 compare DeepAFL with baselines in terms of computational and communication cost. Specifically, to ensure fair comparison, we selected the baseline with the lowest computational and communication cost (i.e., FedAvg) as the representative benchmark to highlight the superiority of our DeepAFL. Notably, the computational costs of all basclines are directly adopted from the existing results reported in AFL (He et al. 2025). More- over, Tables T and Figures 46 provide further details on how our DeepAFL achieves a balance between accuracy and efficiency through the flexible adjustment of T.
Specifically, shown in Figures 2 compared with gradient-base baselines, our DeepAFL (with = 20) achieves at least a 99.7% reduction in computational cost and a 50.2% reduction in commu- nication cost on the CIFAR-100. On the Tiny-ImageNet, the advantages of DeepAFL. correspond to a 99.6% reduction in computational cost and a 70. % reduction in communication cost. By flexibly adjusting the number of layers T', for example, setting T 5, users can obtain greater efficiency ad- vantages. In summary, although the cost of our Deep is inevitably higher than that of AFL due to the additional deep layers required for enhanced representations, it remains substantially lower than that of gradient- based baselines. This advantage stems from our DeepAFL's gradient-free manner, which avoids the iterative costly computation and communication associated with gradients.
Subsequently, we also report the detailed -fffcccceff balance of our DeepAFL with varying numbers of layers, as shown in Tables I and Figures 4 Encouragingly, the construction of each additional layer in DeepAFL. requires no more than 3s across all three datasets, providing an intuitive demonstration of its efficiency. Compared with AFL, even when building a 50-layer deep network, DeepAFL incurs less than a twofold increase in training time while delivering performance improvements of at most up to 31.38% on the training set and 9.95% on the testing set. It is worth noting that, there is considerable potential for our DeepAFL to further reduce communication cost in the future through compression techniques such as matrix factorization, as its primary communi- cation information is the Auto- Cuto-Correlltion and Cross-Correlation Matrices.
34

---

Published as a conference paper at ICLR 2026
E.3.5 PARAMETER ANALYSES
Here, we provide comprehensive analyses for the parameter sensitivity our DeepAFL, encompass- ing the regularization parameters A, 7 the activation function of-), and the projection dimensions dg. dp. Specifically, we keep all parameters' default settings, varying only the parameter under investigation to assess its sensitivity. The specific analyses for each parameter are detailed below:
First, we explore the regularization parameters À and 7 which regularize the global classifier and the transformation matrix, respectively. As illustrated in Tables sensitive to y. This evidence may stem from the fact that our deep residual network employs only the final global classifier during inference, yet relies on all transformation matrices. Specifically, on the simple CIFAR- 10 dataset, the best performance is achieved when À 10 and r E [0.1, 0.5]. For the complex CIFAR- 100 and Imageeee datasets, the optimal values of À are found be highly dispersed, while the optimal consistently occurs at the smaller value of y = 0.01.
DeepAFL is insensitive to À but
Second, we further analyze the impact of different activation functions on DeepAFL's performance. The detailed experimental results are shown in Table [ The selection of the activation function markedly influences DeepAFL's performance. More specifically, a performance variance of about 2% can be observed among different activation functions, with GELU emerging as the optimal choice and Softshrink as the least effective, Moreover, compared to omitting the activation function entirely, employing GELU yields up to 5% gains, underscoring its critical role. More comprehensive analysis of this significance is available in the corresponding ablation study in Appendix [E366]]
Third, we study the effects of varying projection dimensions dg and de our DeepAFL in Tables 8. As these projection dimensions increase, DeepAFL's performance rises initially before declining. This behavior can be attributed to the fact that, while larger dimensions can enhance the model's expressive power by capturing more information, they also concurrently increase the propensity for overfitting and compromise numerical stability. Notably, the model crashes when de dF = 213, Furthermore, as detailed in Appendix D the complexity of DeepAFL scales at least quadratically with these dimensions, and excessively large values incur substantial overhead, cautioning against the indiscriminate pursuit of larger dimensions. That's 's why we select dp = de = 20 as default.
E.3.6 ABLATION STUDIES
Here, we present the detailed analyses of our ablation studies to dissect the individual contributions of residual skip connections, random projections, activation functions, and trainable transformations in our DeepAFL. Specifically, we construct four ablation models, with each omitting one of these key components, and compare them against our full DeepAFL across varying layer depths on diverse Moreover, the ablation studies are conducted under identical conditions for fair comparisons, with all parameters aligned with those used in the
datasets, as shown in Tables [7 and Figures & 10
main experiments. The detailed analyses are provided below:
First, we focus on analyzing the contribution of the residual skip connections in our DeepAFL. with the corresponding ablation model denoted as Ablation (1). For the simple CIFAR-10, ablating the residual connections prevents performance improvements as the network depth increases, as shown in Tables 12 and Figure indicating that this dataset is highly sensitive to such skip connections. On the CIFAR- 100 and Tiny-ImageNe datasets, performance continues to improve even without skip connections, exhibiting an initial rise followed by convergence, as shown in Tables 17 17 and Figures 1H Meanwhile, the ablation model attains respectable performance on Tiny-Imagel yet exhibits markedly limited gains on CIFAR-I00. These disparities across datasets may stem from their differing complexities, with CIFAR- 10, CIFAR-100, and Tiny-ImageNet containing 10, 100, and 200 classes, respectively. Notably, compared to DeepAFL. the ablation model yields substan- tially inferior performance across all cases. This degradation stems from the fact that, without skip connections, the model encounters an information bottleneck akin to that in gradient-based deep learning. Crucially, the importance of residual skip connections is theoretically grounded, as their ablation would invalidate Theorems and 3. Specifically, the residual skip connections are crucial because they enable special case: if the transformation matrix is zero matrix, update is applied to the representation features. This functionality guarantees that the representation features learned at each layer will be at least good those from the preceding layer. Thus, without the resid- ual skip connections, each layer essentially constructs new representation features from the hidden random features, thereby forfeiting these valuable theoretical guarantees.
35

---

Published as a conference paper at ICLR 2026
Second, we further analyze the contributions of the random projections, which give the stochasticity to our DeepAFL. Ablating the random projections {Br} can be achieved by simply setting them to the identity matrices, and we denote this ablated model Ablation (2). While this ablated model still yields performance improvements across all datasets, its performance growth rate is markedly slower compared to DeepAFI., and it reaches convergence earlier. Specifically, at T 10, the abla- tion model reaches convergence across all three datasets, exhibiting performance gaps of 3.64% to 9.25% on the training set and 1.90% 3.43% on the testing set compared to our DeepAFL. Subse- quently, DeepAFL's performance continues to improve with increasing layers, further widening the performance gap. By T = 50, this gap further expands to 8.68% to 22.2% on the training set and 2.70% to 5.87% on the testing set. In Figures 8 10 to introduce the essential stochasticity, the model tends to converge to local optima or saddle points, even though the projection itself does not alter the feature dimensionality, as dg dp- = =210
we can observe that without random projections
Third, we focus on analyzing the contributions of the activation function in our DeepAFL, which introduces nonlinearity to our DeepAFL. The corresponding ablation model is named Ablation (3). can be observed that the performance of this ablation model remains nearly invariant as the layer depth T increases. Specifically, the performance fluctuation from T 0 to T = 50 consistently falls below 0.05% across all datasets, thereby underscoring the critical importance of the activation function in DeepAFL. This is because, without the activation function, each residual block merely performs a purely linear transformation on the feature representations, fundamentally limiting the model's representative power and preventing further performance improvement with added depth.
Fourth, we study the contributions of the trainable transformation 22+11 which imparts learnability to our DeepAFL. The corresponding ablation model is denoted as ablation (4). As illustrated in Tigures[SHTu) this ablation model exhibits modest performance gains followed by rapid convergence on the simple CIFAR-10, while showing nearly no improvement on the complex CIFAR- and Tiny- ImageNe.. This evidence arises because. without the trainable transformation for learnability, each residual block essentially functions as a fixed random nonlinear feature updater. For the simple CIFAR-10 10, these updaters can fortuitously render random features progressively more discriminative to a limited extent. For complex CIFAR-100 100 and Imn--ImaeeNet, relying solely on random feature updates fails to capture the high-level features, leading to negligible performance improvement.
E.3.7 COMPARING DEEPAFL WITH OTHER DEEPENING STRATEGIES
Here, we further compare our DeepAFL with the other four alternative deepening strategies to high- light its superiority. Specifically, we design four distinct deepening strategies: (1) cascades with random features, (2) cascades with random features and label encoding, (3) cascades with activated random features, and (4) cascades with activated random features and label encoding. Notably, the strategies (1) and (3) here are similar to the naive approaches (a) and (b) in Figure I However, in comparison. strategies (1) and (3) additionally incorporate random projection and activation after the backbone to form the zero-layer features o thereby aligning more closely with our Aee the starting point to ensure fairness in comparison. These two naive strategies are consistent with most existing attempts in the literature to deepen analytic networks (Low et al. (201)). Meanwhile, strategies (2) and (4) are essentially extensions of (I) and (3) through the incorporation of label encoding, which is an established technique in analytic learning that introduces an additional linear mapping for the labels at each layer to facilitate the training of deep analytic networks (Zhuang et al.)
2021 2025). The detailed experimental results are presented in Tables [8| 23 and Figures [3
From the results, our DeepAFL consistently outperforms all other deepening strategies across all datasets, thereby underscoring its superiority. Across all datasets, strategies (1) and (2) achieve very similar performance, exhibiting nearly no improvement with an increasing number of layers. Next, we turn our attention to strategies (3) and (4), which differ in that strategy (4) incorporates additional label encoding. On the CIFAR-10, strategies (3) and (4) yield comparable performance, both showing some improvement followed by rapid convergence. On the CIFAR- -F0, strategy (4) lags substantially behind strategy (3), despite both showing gains with increasing layer depth. This disparity widens further on the Tiny-ImageNet, where strategy (4) experiences a pronounced perfor- mance decline from added depth. The varying effects of label encoding across datasets may stem from differences in the number of classes, as these datasets contain 10, 100, and 200 classes, respec- tively. Taken together, these results suggest that existing deepening strategies are of very limited effectiveness and fall far short of the performance achieved by our proposed DeepAFL.
36

---

Published as a conference paper at ICLR 2026
:+1 Φ F.
W++ (0 P++1 +1)
The Feature Updating
Features
The Least Squares
S t+1
Trainable Transformation
Analytic Classifier
Ωt+1
The Sandwiched Least Squares
We W t+1
a++ F. O (r1 diag(A)) @ (
Figure 14: The training process for each layer our DeepAFL.
H(0,, W:)
Layer-by-layer Decrease in Empirical Risk
Empirical Risk H
H(1 W,)
H(Pr++1 Wo+1)
Layer Depth t
Figure 15: Layer-wise reduction of empirical risk in our DeepAFL.
The Sandwiched Least-Squares Process
Sandwich = FΩW
| IR--FQIII | + rI1
F (Known)
Residual Target R
R (Target)
Ω (Unknown)
Regularization = >111211
W (Known)
Closed-form Solution:
Ω V[(VT F TW O (y1 + diag(AF) x diag (Aag(A))) W T
Figure 16: The sandwiched least-squares process in our DeepAFL.
37

---

Published as a conference paper at ICLR 2026
Table 24: Performance comparisons of the top-1 accuracy (%) among our DeepAFL and two new baselines (i.e., FedAWA (Shi et al., (025) and (Liu et al. 2024a) on CIFAR-10, CIFAR-100. and Imageem The best result is highlighted in bold, and the second- result is underlined.
CIFAR-10
CIFAR-100
Tiny-lmageNet
Bascline
a 0.1
a = 0.05
a = 0.1
a = 0.1
a = 0.1
a = 0.1
FedAWA 2025
79.42%
76.37%
58.94%
50.48%
59.07%
52.33%
FedLPA (024a
49.57%
42.56%
40.04%
22.55%
39.48%
26.67%
AFL [025
80.75%
80.75%
58.56%
58.56%
54.67%
54.67%
Fe. (T 5)
85.20%
85.20%
64.72%
64.72%
60.31 %
60.31 %
DeepAFL (T 10)
85.93%
85.93%
65.96%
65.96%
61.37%
61.37%
DeepAFL (T 20)
86.43 %
86.43 %
66.98% %
66.98% %
62.35 %
62.35%
Improvement 1
5.68 %
5.68 %
8.04%
8.42%
3.28 %
7.68% %
F MORE EXPLANATIONS ON THE DETAILED PROCESS OF DEEPAFL
Here, we provide comprehensive descriptions of our DeepAFI. process to enhance clarity. Specifi- cally, as illustrated in Figures [6) we elaborate on the training process of each layer, demonstrate the layer-by-layer empirical risk reduction, and visually depict the sandwiched least-squares mech- anism that underpins our DeepAFL. Further details are presented below.
First of all, let's focus on the training process for each layer in our DeepAFL. As illustrated in Figure [4] each layer entails the sequential update and computation of the trainable transformation matrix s2 the feature matrix 11+1, and the analytic classifier W Specifically, for each (t + 1- layer, the trainable transformation matrix 22 is initially derived through the sandwiched least squares process, utilizing the preceding analytic classifier W, alongside F, and R,, Subsequently, based on the newly acquired 22,11, the feature matrix is updated to yield o,+1 according to Equation (). Finally, employing the derived feature matrix 1z+1, the analytic classifier is further updated to obtain W++1 using the least squares process, Notably, as a special case, the zero-layer feature matrix Pu obtained directly via feature extraction using Equations (I) and (2), thereby eliminating the need to compute trainable transformation matrix Z2. The aforementioned process is executed layer-by-layer, adhering to the update sequence 40 + Wo H .... WT-1 + Ωt + 4T H
W, until the network construction is complete.
Subsequently, we present intuitive and visual explanations of the monotonic decrease in empirical risk the depth of our DeepAFL increases. The empirical risk H(O,, W.) is determined by the feature matrix 4, and the analytic classifier W.. As previously analyzed, within each layer, Deep- AFL sequentially updates the feature matrix and the analytic classifier to yield improved 4 and W Consequently. the empirical risk undergoes two reductions within each layer, as depicted in Figure 15 We now detail these two reduction steps within each layer. Specifically, the fea- ture matrix + is first updated to obtain Φ,41 via the trainable transformation matrix D4411 which constitutes the optimal solution to Equation (7) for minimizing the empirical risk with respect to feature optimization. Thus, the updated empirical risk H(P141, We) is lower than the initial risk (P., We). Subsequently, with L held fixed, the new analytic classifier W++1 Wi+1 is computed as the optimal solution to Equation (4) for minimizing the local empirical risk. Thus, the final empirical risk H( ((( 2+1, W(+1) is also lower than the risk H ((++1 W.). By executing this procedure layer- by-layer, the empirical risk is progressively reduced, thereby leading to enhanced performance.
Finally, as we term this optimization process for Q2 as the sandwiched least squares problem, we would like to provide detailed illustration of its process. Specifically, the optimization objective in Equation (7) can be viewed as a special case of generalized Sylvester matrix equations (Wu et al., 2008 Ding al. 2008; Duan 2015). This structure is characterized by the unknown variable Q being sandwiched between two known matrices, F and W. In this context, the "sandwich" structure, therefore, refers to this three-matrix product form FRW. The empirical risk minimization aims to minimize the residual, meaning the sandwich term should be as close as possible to the residual target R. Meanwhile, the regularization term 2| e is applied constrain the magnitude of Ω and prevent it from becoming excessively large. Consequently, this particular structure facilitates the derivation of distinct analytical solution, as presented in Equation (8)
38

---

Published as a conference paper at ICLR 2026
Table 25: Accuracy of gradient- gradient-based baselines with varying layers on CIFAR-100)
Layers
T == 1
T = 2
T =
T =
10
T
20
FedAvg
56.62%
55,83%
<5%
<3%
<1%
FedDyn
57.55%
57.22%
56.26%
<3%
<1%
G MORE EXPERIMENTAL RESULTS ON RECENT BASELINES
Following reviewer's suggestion, we implemented two new baselines (i.c., FedAWA (Shi et al. 2025) and FedLPA (Liu et al. 2024a)) across the three benchmark datasets utilized in our main ex- periment: CIFAR-10 10, CIFAR-100, and ImageNet-R. All experimental settings were maintained to be consistent with the other baselines established in our manuscript. Specifically, because FedLPA classified as a one-shot communication FL method, and we empirically observed that its perfor- mance essentially converges within 50 local epochs, we set the local epochs to 50 for this baseline. This setting allows us to mitigate unnecessary computational overhead for FedLPA, thereby more fully demonstrating its potential for efficiency. We present the results of these two new baselines alongside AFL and our DeepAFL. for additional comparison, as shown in Table 24
In terms of accuracy, FedAWA exhibits very strong results, approaching (on CIFAR-10) or surpassing AFL (on CIFAR- 100 and Timy-ImageNet) when a = 0.1. Notably, even though FedAWA outperforms AFL, our DeepAFL still consistently achieves the best performance across all scenarios. Furthermore, as gradient-based one-shot method, FedLPA's performance is compromised by data heterogeneity, it lacks the inherent invariance demonstrated by our DeepAFL. Consequently, despite our best efforts to tune its hyperparameters, FedLPA still performs poorly. Furthermore, as the degree of Non-IID data increases, a pronounced performance degradation is observed for both of the newly introduced gradient- baselines (i.e., FedAWA and FedLPA). This observation further highlights the advantage of the inherent invariance property of our DeepAFI.
In terms of efficiency, using the CIFAR-100 dataset as example, FedAWA requires approximately 10 hours, while FedLPA requires approximately 2.5 hours. In sharp contrast, our DeepAFL com- pletes the task in less than 100 seconds (specifically, 91.74 seconds), achieving speedup exceeding 90x. Notably, even though FedLPA is also classified as a one-shot FL approach, and we have already minimized its number of local epochs to fully reflect its efficiency potential, its overall overhead re- mains significantly higher than that of DecpAFL. This phenomenon is primarily due to FedLPA still being gradient-based based and facing cinecconunnnn backpropagation. Conversely, our DeepAFL. is in a forward-only manner, further highlighting its efficiency advantage beyond its one-shot nature.
H MORE EXPLANATIONS ON OUR DEEPAFL'S EXPERIMENTAL SETUP
Here, we would like to clarify our experimental setup. To maintain fairness in comparisons, the results for all baselines in our paper are directly employed from the benchmark provided in the original AFL paper (He et al. 2025). Specifically, the gradient-based approaches in the original AFL paper adhered a similar setup as AFL, where the backbone network entirely frozen during the FL process, and only the one-layer classifier is trained.
Moreover, to address potential concerns that gradient- bassd baselines might benefit from multiple trainable layers. we conduct additional experiments on FedAvg and FedDyn with multiple trainable layers (T E f2, 5, 10, 20})) on CIFAR-100 under the Dirichlet distribution (a 0.1). For these experiments, the feature dimensions of the additional layers are aligned with those of our DeepAFL (i.e., 1024) to ensure consistency. Detailed results are presented Table 25
Through these results, we can observe that the performance of the gradient-based methods gradually and constantly degrades as T increases. Specifically, when T increases beyond a certain threshold (c.g., T > 10). the excessive number of parameters, combined with the Non-IID data in the FL scenario, may cause the training process to collapse. These findings fully indicate that directly in- creasing the number of trainable layers after the backbone for these baselines, similar to Deep- AFL setup, has a detrimental effect on their performance. Consequently, far from benefiting from multiple trainable layers, these gradient-based baselines exhibit significantly inferior performance under such configurations, thereby validating the propriety of our experimental setup.
39

---

Published as a conference paper at ICLR 2026
I
MORE ANALYSES ON DEEPAFL'S SCALABILITY WITH VIT BACKBONES
Table 26: Analyses of DeepAFL's scalability with the ViT-B-16-14 16-1-1KKK backbone on CIFAR- 100.
Layers
Testing Acc
A testing Are
AAN Ace
Time Cost
À time Cost
AALL.Co
AFI.
75.45%
/
107.19s
/
DeepAFL (I 5)
78.05%
2.60%
2.60%
118.97s
11.78s
11.78s
DecpAFL (I 10)
79.01%
0,96%
3.56%
126.55s
7.58s
19.37s
DeepAFL 15)
79.73%
0.72%
4.28%
133.79s
7.24s
26.60s
DeepAFL (I 20)
80.13%
0.40%
4.68%
141.02s
7.23s
33.83s
DeepAFL (T 25)
80.33%
0.20%
4.88%
148.56s
7.54s
41.37s
DeepAFL (T -30)
80.51%
0.18%
5,06%
156.21s
7.65s
49,02s
Table 27: Analyses of DeepAFL's scalability with the ViT-B-16-I-21 backbone on CIFAR-100.
Layers
Testing Acc
Aresing
AAFLAC
Time Cost
A Time
AXFL
AFL
86.35%
107.19s
/
DeepAFL (I 5)
87.71%
1.36%
1.36%
118.97s
11.78s
11.78s
DeepAFL (T 10)
88.15%
0.44%
1.80%
126.55s
7.58s
19.37s
DeepAFiL (I 15)
88 88.49%
0,34%
2.14%
133.79s
7.24s
26,60s
DeepAFL 20)
88 80%
0,31%
2.45%
141.0ss
7.23s
33.83s
DeepAFL 25)
88 95%
0.15%
2.60%
148. 56s
7.54s
41.37s
DeepAFL. (T 30)
89.08%
0.13%
2.73%
156.21s
7.65s
49.02s
Here, we present detailed analyses of DeepAFL's practical scalability with VIT backbones. Given that our DeepAFL is built upon AFL (He et al.) (025), which uses ResNet- 18 as its primary back- bone, we also employ the aligned backbone in our main experiments. This alignment facilitates a clearer and evaluation by adopting larger ViT models to investigate our DeepAFL's practical scalability.
transparent comparison between the methods. Here, further extend
Specifically, we select versions of the ViT-B-16 backbone: ViT-B-16-I-IK (pre-trained on the ImageNet-IK with approximately 1.28 million image) and ViT-B-16-1-21K (pre-trained on the larger ImageNet-211 K dataset with approximately 14 million images). Owing to the significantly broader pre-training scale, BTT-B-16-I-21K typically yields better performance than VTT-B-16-I-1K Since the backbone sizes of both ViT models are identical (Base-16), the runtime for our DeepAFL on both models remains essentially the same. Furthermore, all experiments are conducted on the CIFAR- 100 dataset with 100 clients using a single NVIDIA RTX 4090 GPU.
The detailed results are presented in Tables 26 27 where "Testing Acc" and "Time Cost" represent the test accuracy and total wall-clock runtime. The A columns denote the step-wise change relative to the preceding row, while the AFF columns indicate the cumulative difference from AFL.
From these results, it is evident that applying DeepAFL to both VIT backbones leads to significant performance improvements with minimal time cost, thereby demonstrating its robust scalability. Specifically, at 30, our DccpAFL achieves an accuracy gain 5.06% over AFL with the ViT-B- 16-I-1K backhone, and improvement of 2.73% with the VTT-B-16--2211 backbone. Remarkably, these substantial improvements are achieved with a runtime of merely 156.21 s, which is only 49.02 higher than AFL. is important consumed by all 100 clients to complete the training process across all layers on a single 4090 GPU. Thus, the average cost per client is less than 1.56 In stark contrast, even the simplest and most efficient FL baseline (e.g., FedAvg) requires a Total Wall-Clock Runtime of over 33,000 s.
that this reported runtime encompasses the cumulative time
Furthermore, although we observe that the performance improvement of DeepAFL exhibits dimin- ishing marginal returns, this phenomenon is entirely expected and normal. First, as the backbone capability strengthens, the feature representations are already highly optimized, making increas- ingly challenging to extract further performance gains beyond such high baseline. Second, while increasing the number layers enhances the fitting capacity DeepAFL, the fixed volume of training data inevitably leads to saturation limit in model performance. Notably, such diminishing returns are also common in gradient-base methods. Therefore, the ability of DeepAFL. to achieve stable and significant improvements across various backbones without increasing any training data volume is highly noteworthy. Especially when considering the negligible runtime cost, the perfor- mance gains delivered by DeepAFL. can be regarded as incurring virtually time overhead.
40

---

Published as a conference paper at ICLR 2026
J MORE ANALYSES ON PARTIAL CLIENT PARTICIPATION
Table 28: Performance evaluation of DeepAFL on CIFAR- 100 under partial client participation.
Consisteut Participation
Inconsistent Participation
Layers
100%
v = 90%
== 80% y= 70% o == 60%
1) == 50%
7 = 100%
1 = 90%
80% 1y y= 70% 1y == 60% 1 = 50%
T= 5
64.72%
64.53%
64.38%
63.92%
62.85%
61.54%
64.72%
64.21%
64.03%
64.05%
62.19%
61.36%
T = 10
65.96%
65.46%
65.26%
64.49%
64.76%
63.17%
65.96%
65.36%
65.05%
64.45%
64.09%
62.93%
T 15
66.56%
66.39%
66.20%
65.38%
65.17%
65.08%
66.56%
66 66.22%
66.17%
65.64%
64.60%
64.98%
T 20
66.98%
66.76%
66.66%
66.47%
66.27%
65.14%
66.98%
66.57%
66.51%
66.32%
65.50%
64.81%
Here, we provide detailed analyses of our DeepAFL's performance in scenarios involving partial client participation, thereby further highlighting its robustness. The detailed analyses are as follows.
First, let's focus on the minor mechanism adjustments required to accommodate partial client par- ticipation. As detailed in Section 3.2] constructing each new layer in DeepAFL entails two rounds of communication. corresponding to the computation of the classifier weights W, and the trainable transformation matrix 14+11 For notational convenience, let denote the complete set of clients, where the total number of clients is |S] = K. Consequently, the aggregation processes for W, (i.e., Equation 12 and S2(+1 (i.e., Equation (T6) can be expressed as:
G c1:K = G = E Ekes G H.KK = H
E
KES H
(78)
KES II, III E KES II r!-k = r - r
(79)
When only a partial set of clients participates in the aggregation for DeepAFL. we denote the subset of clients contributing to the classifier construction at layer t as S.W. and the subset contributing to the trainable transformation matrix construction as S.. Consequently, the aggregated matrices will be constructed using contributions from only these participating subsets:
S.S.
== =E KESW G
ww H.
= Evesp H
(80)
I1 Sn
= -E KES II r = Exesp r.
(81)
Subsequently, the server utilizes these aggregated matrices from the partial participants to compute W, and l2+1 as before. Notably, since only a subset of clients contributed data, the effective full dataset should be redefined as the union of data held by all participating clients. Thus, the invari- ance property of our DeepAFI. needs to be redefined as being identical to the centralized analytical solution over the effective full dataset comprising the data of the participating clients. In summary, our DeepAFL. can handle partial participation easily and effectively without substantial procedural adjustments, while maintaining its analytical advantage of being invariant to data heterogeneity.
Second, we conducted experiments to evaluate the performance of our DeepAFL under varying client participation rates 1j. For simplicity, we assume that the participation rate remains consistent across aggregation processes within the same layer, i.e., 17 = 1/|S| | Furthermore, the client subsets S.W. and S.D corresponding to 17 are randomly sampled from S. Specifically. to ensure comprehensive evaluation, we consider two distinct cases: (1) Consistent Participation and (2) Inconsistent Participation. These cases dictate whether the client subsets for the two aggregation processes within the same layer are identical distinct, i.e., Yt, S. = S.a or Vt, S.W of St7 Intuitively, the case (2) presents a greater challenge for DeepAFL, as the inconsistency within a single layer may heighten the risk of model instability.
Building upon these two cases, and using full participation (1) = 100%) as the control group, we conducted a broad range of analyses with 77 ranging from 90% to 50% to thoroughly assess the per- formance variation of our DeepAFL on CIFAR-100, As shown in Table 28 our DeepAFL exhibits substantial robustness to partial client participation in both cases. Specifically, for a high participa- tion rate of 2 70%, the maximum accuracy degradation observed at T = 20 remains below 0.7% across both cases. Furthermore, even under the extremely challenging scenario of f) = 50%, where up to half of the clients drop out, DeepAFL still maintains high accuracy at T = 20, achieving 65.14% and 64.81 % for both cases. Even at low layer count (T = 5) and the most severe dropout rate (17 50%). the performance of our DeepAFL (61.54% and 61.36%) still significantly outper- forms the AFL's accuracy (58.56%) achieved under the condition of 100% client participation. This underscores the strong robustness of DeepAFL. in handling partial client participation scenarios.
41

---

Published as a conference paper at ICLR 2026
K
MORE DISCUSSIONS ON DEEPAFL'S ROBUSTNESS TO NOISES
Table 29: Performance evaluation of DeepAFL on CIFAR-100 under noisy training environments.
Layers
= 0%
x 10%
T= 20%
T= 30%
7 40%
7 == 50%
DeepAFL (I 5)
64.72%
64.20%
63.63%
63.25%
62.51%
60.91%
DeepAFL. 10)
65.96%
65.43%
65.09%
63.96%
62.98%
60.99%
DeepAFL (T 15)
66.56%
66.11% %
65.66%
64.29%
63.15%
60.97%
DeepAFL (I 20)
66.98%
66.38%
66.11%
64.69%
63.35%
60.45%
Here, we further demonstrate that our DeepAFL, utilizing the MSE loss, is less sensitive noise compared to muddient-based methods employing the CE loss. Specifically, we elaborate on this point from both logical and experimental perspectives, as detailed below.
From the logical perspective, a crucial factor that makes method sensitive to noise is overfitting. Specifically, if a method tends overfit, then when there is even slight noise in the input data, will tend to fit this noise. In contrast, the analytic learning methods using MSE loss are inherently less susceptible to overfitting, typically exhibiting a tendency toward underfitting instead, Indeed, addressing this underfitting limitation by augmenting the model's fitting capacity through deep rep- resentation learning is the primary motivation for proposing our DeepAFL.
Furthermore, Tables I in Appendix E.1 list the training and testing accuracies for AFL and our DeepAFL across various datasets, corroborating that analytic learning methods are not prone to overfitting, Taking the CIFAR-100 dataset an example, the training set accuracy for AFL is only 61.55%, which is quite low, especially compared to its testing set accuracy of 58.56%. This phe- nomenon suggests that AFL suffers from severe underfitting. Moreover, when gradually increase the depth T from O to 50 in our DeepAFL, both training and testing accuracies rise steadily. The fact that testing accuracy improves commensurately with training accuracy confirms that DeepAFL avoids overfitting, even at depths of up to 50 layers.
More specifically, DeepAFL's ability maintain superior representation learning without overfitting stems from the fact that its analytic classifier with MSE loss is linear. Because of this, its model complexity is quite low, making it very difficult overfit and thus robust to noise. In contrast, CE loss is designed to induce steeper gradients to facilitate backpropagation optimization. While this characteristic accelerates training. it simultaneously renders gradient- baseed methods more prone to overfitting and hypersensitive to data noise. Consequently, noise robustness constitutes the inherent methods (with CE loss).
advantage of our DeepAFL. (with MSE loss) compared to gradient-
From the experimental perspective, we further verify the noise robustness of our DeepAFL. by ex- plicitly simulating noisy training environments. Specifically, we simulate the noise in the training data by randomly flipping the labels of a proportion of the training data. The selected samples have their labels randomly changed to another class label. Using т = 0% as the control group, we con- duct extensive analyses with E (10%, 20%, 30%, 40%, 50%) to thoroughly assess performance variations of our De on CIFAR-100. The detailed results are reported in Table 29
These results demonstrate that DeepAFL. exhibits substantial robustness data noise. Specifically, under moderate noise levels (r < 20%), the maximum accuracy drop for our DeepAFL (at = 20) less than 1%. Furthermore, even under the extremely challenging scenario of T 50%, where half of the training labels are corrupted, DeepAFL maintains high accuracy. Crucially, even at a low layer count (T' 5) and the most severe noise rate
50%), the performance of our DeepAFL (i.e.. 60.91 %) still significantly outperforms the AFL's accuracy (i.e., 58.56%) achieved under the condition of 100% accurate data labels with no noise. Since AFL represents the state-of-the-an baseline on Non-IID-1 for CIFAR-100, these results also imply that our DeepAFI. even with 50% label corruption, outperforms all gradient-based baselines under ideal label accuracy.
Additionally, detailed in Appendix [ we also examine the robustness of DeepAFL under partial client participation. Similar experiments conducted with client dropout rates ranging from 10% to 50% show that DeepAFL maintains highly stable and superior performance. Notably, DeepAFL's performance under label flipping is slightly weaker than under client dropouts. This is expected, as client dropouts simply reduce training data volume, whereas label flipping introduces the additional negative effect of data poisoning. Moreover, the performance gap between these two scenarios remains small, which further substantiates DeepAFL's superior robustness.
42

---

Published as a conference paper at ICLR 2026
L MORE DISCUSSIONS ON DEEPAFL'S USE CASES
Here, we provide detailed discussions on DeepAFI's use cases. Specifically, our DeepAFL. requires a given backbone to serve as an effective feature extractor for training, but it does not mandate how this backbone is obtained. Indeed, leveraging pretrained models or foundation models as the back- bone in FL has emerged as very common and popular research practice in recent years (Nguyen) et al. 2023; Piao et al. 2024 Yu et al. 2024 2025). Accordingly, we elaborate below on several common ways for obtaining such pretrained backbones, corresponding to different use cases of our DeepAFL, as well as the pervasiveness of pre-trained backbones in FL and the necessity of FL even with such backbones. The detailed discussions are presented follows:
First, we want to highlight that our DeepAFL. requires a pretrained backbone for feature extraction, but does not mandate how this backbone obtained. For the sake of brevity, we do not discuss this in detail within the main text. Herein, we introduce three common ways for obtaining pretrained backbones, which collectively illustrate DeepAFL's various use cases:
Supervised Pre-training: This is the most prevalent case and constitutes the primary set- ting for the experimental comparisons presented in our paper. In this scenario, our Deep- AFL can be viewed as a collaborative fine-tuning approach for FL, where clients adapt a shared public backbone private data with different feature distributions.
Self-Supervised Pre-training: With the emergence of large-scale pre-trained models, there is frequently insufficient labeled data for supervised pre-training. In such cases, self- supervised backbones have gained prominence (such as auto-encoders via reconstruction, and DINO series via contrastive learning). These self-supervised models, however, inher- ently lack an integrated classifier or regressor, possessing only robust generalizable feature extraction capabilities without direct predictive functionality. Therefore, deploying such backbones in FL necessitates training a matching classifier or regressor. In this context, our AFl serves as a direct and highly efficient FL training approach to obtain this classifier or regressor, effectively leveraging the backbone's 's established capabilities.
Domain Adaptation: If the pre-trained backbone presents a domain shift relative to the FL system's 's target data, the server can also perform initial feature domain adaptation by fine-tuning the backbone (using a set of domain -omain-specific or compliant public data). Once this adaptation is complete, the backbone is then frozen, and our DeepAFL can be imple- mented for subsequent FL training. This case can be viewed as an engineering strategy to mitigate the issues arising from a cross-domain backbone. Given that domain adaptation can be performed centrally by the server and DeepAFL incurs only negligible time costs, the aggregate overhead across both stages is still considered to be very small.
In summary, DeepAFL can not only leverage backbones obtained through supervised pre- wre-training as demonstrated in our experiments, but also effectively utilize other types of backbones for differ- ent use cases. Specifically, when using backbones from self-supervised pre-training, our DeepAFL effectively solves the meaningful problem of constructing speciffe classifier in FL. Further- more, when faced with the potential for domain gaps, this problem can be mitigated through domain adaptation as described above, which can be viewed
engineering strategy.
Second, we further discuss the pervasiveness of pre-trained backbones in FL and the necessity of FL even with such backbones. Indeed, incorporating pre- me-ttrieed backbones is widely embraced
practice in recent research to introduce prior knowledge and stabilize FL (Nguyen et al. [023] Piao
al. 2024 Yu et al. 2024 2025). Specifically, a key commonality between our work and these studies is the focus on using foundation models for conducting FL, rather than using FL for building foundation models. Notably, many existing pre-trained model-based FL works are mechanistically constrained to necessitate large foundation models. In contrast, DeepAFL accommodates backbones of varying sizes and capabilities, which renders particularly suitable for resource-constraine edge environments compared to related works. Moreover, we demonstrate that a pre-trained backbone does not negate the need for FL. Leveraging results from the original AFL paper on CIFAR-I000 (a = 0.1 and K 100) (He et al. 2025), ) we observe that purely local training without global aggregation yields maximum and average test accuracies of merely 16.36% and 12.04%. Conversely, traditional FL method FedAvg and AFL achieve 56.62% and 58.56%, while our DeepAFL (T 20) further elevates performance to 66.9 66.98%. These results conclusively demonstrate that despite the availability of pre-trained backbone, FL remains indispensable for achieving high performance.
43

---

Published as a conference paper at ICLR 2026
M DEEPAFL'S REPRESENTATION LEARNING
In this section, we present detailed discussions on the representation learning capabilities of Deep- AFL. Specifically, first outline how the capacity of DeepAFL aligns with fundamental character- isties of representation learning. We then analyze the intermediate features produced by DeepAFL. as empirical evidence supporting these capabilities, The specific discussions are provided below.
1.1 ANALYSES ON THE CONCEPT OF REPRESENTATION LEARNING
Here, we demonstrate that the capabilities of DeepAFL align with the fundamental characteristics of representation learning. To this end, we first outline these fundamental characteristics and then conduct point-by-pond analysis showing how our DeepAFL satisfies each one. Subsequently, we further substantiate our claim by analyzing the architectural similarities between DeepAFL and the Multi-Layer Perceptron (MLP). The detailed analyses are presented below.
First of all, drawing upon the seminal work of Bengio et al. (Bengio et al. 2013), we can concep- tually distill two fundamental characteristies of representation learning: (1) Automated Feature Extraction: This entails learning feature transformations (which are often non-linear) directly from the data, thereby avoiding the complexity and reliance on expert knowledge associated with man- ual feature engineering.
(2) Utility for Downstream Tasks: The extracted features are beneficial and useful for adapting to specific downstream tasks, meaning they provide utility when building predictors (e.g., classifiers), ultimately leading to enhanced model performance.
In fact, AFI directly satisfies these two foundational characteristics: (1) The feature trans- formations involve trainable parameters Ω, which are automatically learned from the data via the "sandwiched" least squares. Moreover, the use of the activation function (+)) ensures necessary non-linearity in the layer-wise transformations. (2) The feature transformations show evident and significant benefits for our downstream task (i.e., classification). Specifically, when coupled with an analytic classifier, the accuracy of our DeepAFF consistently improves on the training and test sets as T increases (as shown Tables I of our paper). In summary, based on these conceptual analyses, our DeepAFL. indeed aligns with the fundamental concept of representation learning.
Subsequently, to further clarify our DeepAFL's alignment with fundamental representation learning concepts, we further draw an analogy between our DeepAFL. and the MLP. Given that the MLP is recognized as the most fundamental DNN, which indisputably achieves representation learning, this analogy serves to demonstrate that our DeepAFL similarly achieves these capabilities.
Specifically, since our DeepAFL incorporates a residual block structure, we also introduce skip connections into the MLP, which does not impair its representation learning capability (but rather makes easier to train). At this point, what the residual block of the MLP learns is nothing more than subjecting the features from the previous layer an affine transformation W W++1 followed by an activation function (.). Here, the affine transformation W++1 constitutes the learnable parameters of the MLP, as follows:
1 = a W W)
(82)
Consequently, when multiple layers are stacked, the MLP functions as continuous alternation of affine transformations and non-linear activation functions. Crucially, clear parallel emerges between our Dee and the MLP, as DeepAFL similarly involves an alternating stack of affine transformations and non-linear activation functions:
( o (B)B,))
(83)
The primary distinction is that the learnable parameters W++1 in the MLP are applied prior to the activation function, while the parameters 2 in our DeepAFL. are applied after the activation. This localized difference is effectively diminished when multiple layers are stacked. Furthermore, more complex DNNs also share this philosophy of feature re- re-weighting via affine transformations and non-linear activation functions. For instance, a CNN can be broadly viewed as an MLP with weight sharing and local connectivity. Therefore, the inherent structural similarity between DeepAFL and the MLP confirms that Aee is aligned with the core concepts of representation learning.
Based on the analyses above, Dee not only satisfies the fundamental characteristics of rep- resentation learning but also exhibits a strong structural similarity to the MLP. This dual validation confirms that DeepAFL. aligns with the core concepts of representation learning.
44

---

Published as a conference paper at ICLR 2026
M.2 ANALYSES ON INTERMEDIATE FEATURES
Table 30: Separability analyses of DeepAFL's intermediate features on CIFAR-100.
Layer
Training CSR
Testing CSR
Training IFS
Testing IFS
Training DM
Testing DM
DeepAFL (T - 0)
1.583
1.529
3.25
3.13
3.21
3.09
DeepATLL (T-
1,565
1,520
3,20
3.11
3.16
3.07
DeepAFL 10)
1.555
1.516
3.17
3.10
3.14
3.06
DeepAFL (T 15)
1,546
1.513
3.15
3.09
3.12
3.05
DeepAFL (T 20)
1,539
1,510
3.13
3.08
3.11
3.04
Here, we present experimental analyses of the intermediate features generated by DeepAFL to em- pirically validate its representation learning capabilities. Specifically, we first conduct additional ex- periments designed to assess the separability of the intermediate features produced by our DeepAFL, thereby substantiating the model's advanced learning capacity. Furthermore, we analyze DeepAFlI's utility for the downstream task to further validate its representation learning capabilities.
First, let's focus on the additional experiments for investigating the separability of intermediate features produced by our DeepAFL, which are conducted using the following three metrics:
Seanetttess-Sppratttoon Ratio (CSR): This metric evaluates the ratio of the average intra-class squared distance (i.e., compactness) to the average inter-class squared distance (i.e., separation). A lower CSR value indicates better separability.
Inverse Fisher Score (IFS): It is defined as the reciprocal of the Fisher Score, which is widely adopted metric to distinguish different classes. The IFS is calculated by the ratio of the intra-class variance inter-class variance. A lower FS value implies better separability.
Discriminative Measure (DM): is metric derived from the core principles of the Linear Discriminant Analysis (LDA), quantifying the ratio of within-class scatter between-class scatter. A lower DM value signifies better separability.
In summary, all the additional metrics unified such that lower value indicates better represen- tation separability. Using these metrics, we conducted additional experiments on the CIFAR-100 dataset analyze the intermediate features generated by our DeepAFL. The detailed experimental results are presented in Table 30 From the results, it is evident that all these metrics show better separability as the layer count of our DeepAFL increases on both the training and test sets, which further substantiates our DeepAFL's representation learning capabilities. Furthermore, we observe that the metrics on the test set generally perform better (i.e., yield lower values) than those on the training set. This is likely because the training set contains a much more samples, leading to greater inherent noise and, consequently, slightly inflated metric values.
Meanwhile, we can also observe that DeepAFL exhibits a diminishing marginal returns phenomenon as the layer count increases, but this entirely expected and normal since we did not increase any training data volume. This phenomenon is also common in gradient-based methods. For instance, simply increasing the depth of DNN can not lead to continuous accuracy improvement up to 100% without encountering bottleneck. Even the high performance achieved by LLMs is result of scaling the model alongside vast increases in training data volume (i.e., the scaling law).
Second, as discussed in Appendix M.I. the utility for the downstream task (i.e., classification accu- racy) is the most direct and fundamental metric to validate the representation learning capabilities of DeepAFL. This perspective also aligns with the classical work by Yoshua Bengio et al. (Alain & Bengio 2016). which asserts that better linear separability is indicative of more meaningful repre- sentation. In fact, in Tables II we have reported the training and testing accuracies achieved by our DeepAFL when utilizing intermediate features from various depths (i.c., DecpAFL with differ- ent numbers of layers). Let's take the results on the CIFAR- dataset as an example. The baseline AFL struggles with severe underfitting, achieving a training accuracy of only 61.55%. In stark con- trast, our DeepAFL (at T 20) propels the training accuracy to 15.15%. representing a massive absolute improvement of 23.6%. This improved representation translates directly to generalization, yielding a substantial 8.39% increase in test accuracy. Crucially, these gains are achieved with high efficiency. The total runtime for our DeepAFL is only 91.74 which is marginal increase of less than 42 s compared to AFL (50.05 s). These results show that DeepAFL learns significantly more discriminative and useful representations than AFI..
AppendixMM.I1
45

---

Published as a conference paper at ICLR 2026
N ADDITIONAL DISCUSSION
Beyond the discussion in our main text, we provide additional discussion here. A key feature of our DeepAFL is its capability to perform gradient-free representation learning after the feature ex- traction backbone. This stands in contrast traditional methods that typically apply representation learning directly to the backbone model. A natural question arises: what are the differences and advantages of our approach compared direct representation learning on the backbone?
Direct representation learning on the backbone typically involves two main strategies.
The first strategy is to fine-tune the backbone during the FL training process. This aims to adapt the model to the data distribution within the FL system. However, this approach re- quires enabling gradient-based training in the FL training, which can be severely impacted by data heterogeneity. Moreover, there is a high risk of damaging the knowledge acquired during pre-training, known as catastrophic forgetting. As many existing works have shown that fine-tuning the backhone with gradients during FL training may be counterproductive, it is precisely why we developed DeepAFL for gradient-free representation learning.
The second strategy is to build more complex backbone during the pre-training process, leveraging massive pre-training datasets enhance its robustness. Our approach is orthog- onal to this strategy, as our DeepAFL. is designed to further improve the representations during the FL. training process, given any fixed, pre-trained backbone. This training- proccsss representation learning is both necessary and meaningful, as the pre-trained backbone will inevitably exhibit a domain shift when applied to a new FL system.
In addition, another noteworthy aspect of our work is the effectiveness of residual connections in our DeepAFL, especially since its gradient-free nature seems be at odds with the original gradient- based motivation for these connections in ResNet. We believe that it is a profound theoretical ques- tion that warrants future research. Here, we would like offer several intuitive explanations for it:
First, our comprehensive ablation studies show that the model without residual connections struggles to improve its training accuracy the number of layers increases, let alone testing accuracy. This situation mirrors the challenges faced by gradient-based deep networks be- fore the introduction of ResNet. Deep networks without residual connections can similarly suffer from an information bottleneck as they continually update feature representations. By adding skip connection, each layer's representation learning is built upon the previous layer's 's foundation, enabling continuous and cumulative improvements in representations.
Second, we prove in Theorems 2 and 3, our Deep representation learning capabil- ity is primarily reflected in the non-increasing nature of its empirical risk with added layers. This beneficial property is ensured by the use of skip connections. When network features converge, each residual block can be just set to zero, at minimum, to achieve an unchang- ing empirical risk, thereby guaranteeing that the overall empirical risk of our DeepAFL will not increase. Without this design, the ablated model would no longer be able to guarantee these theorems, severely compromising its representation learning capability.
Third, the philosophy of our DeepAFL intrinsically aligns with that of Gradient Boosting- Specifically, under the chosen MSE loss function, the negative gradient of the loss function is precisely the residual. Therefore, by continuously fitting the residuals from previous lay- ers, our DeepAFL can be seen as learning help unify the understanding of both gradient- -asend and gradient-free learning approaches.
This perspective may further
In summary, we believe our DeepAFL represents a significant first step. There is substantial potential for future theoretical and practical exploration based on this foundation, which we hope will further advance various fields, including FL, analytic learning, and representation learning, etc.
O STATEMENT ON THE USAGES OF LARGE LANGUAGE MODELS
In adherence to the ICLR 2026 policy, we report the use of Large Language Models (LLMs) during the preparation of this paper. Here, the only usages of LLMs were to aid and polish the writing and illustration. Specifically, the LLMs were utilized to improve sentence structure, enhance clarity, ensure grammatical accuracy, and optimize the illustration. Furthermore, all the scientific con- tributions (including the hypothesis formulations, the theoretical analyses, the experimental designs, the data analyses, and the final conclusions, etc.) are the original work of the human authors.
46


Reviews:
[{"review_id": "BSdlNUkbw8", "reviewer": "Reviewer_f9VW", "summary": "The paper aims to address the shortcoming of analytic federated learning: lack of representation learning capability. The idea is to incorporate skip connection into the multi-layer version of the analytic federated learning. The initial feature is extracted from a pre-trained backbone, and then randomly projected and activated to get the zero-th layer feature. The feature is refined at the next layer by adding a residual block consisting of a nonlinear feature transformation and a learnable weight. The nonlinear feature transformation is the same as those for the initial feature, i.e., random projection for stochasticity and activation for nonlinearity. Given the feature at each layer, both the classifier and weight have a closed-form solution to the regularized least-squares. This way, the classifier keeps improving and is shown to converge eventually along the chain of layers. This idea is then adapted to the federated learning setting with only auto-correlation and cross-correlation matrices of relevant client data uploaded to the server, which ensures the invariance of the global model to data heterogeneity as analytic federated learning does. It is mentioned that it is hard to infer client information from the communicated data and the data encryption technique can be integrated to enhance privacy. Extensive experiments are conducted to demonstrate the superiority of the proposed algorithm in terms of both accuracy and efficiency across various settings.", "questions": "1. what if only partial participation of clients?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of adding skip connection to the multi-layer version of analytic federated learning is great.\n2. The design of the residual block is nice.\n3. The framework addresses well the representation learning issue of the analytic federated learning while keeping all advantages like closed-form solutions and invariance to data heterogeneity.\n4. The algorithm is guaranteed to converge\n5. Extensive experiments are provided to demonstrate the potential of the algorithm.", "weaknesses": "The privacy argument seems weak. Once data encryption techniques are introduced, the computational cost will be increasing as well.\n\nMinor: \nIn Line 373-374, \"and AFL (He et al., 2025). Furthermore, we include the analytic learning-based method AFL (He et al., 2025) as a baseline to further highlight our advantages\", where \"AFL (He et al., 2025)\" is repeated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pty00vWdp", "reviewer": "Reviewer_mCa1", "summary": "This paper presents DeepAFL, a deep analytic federated learning framework that extends the original Analytic Federated Learning (AFL) approach from a single linear layer to multiple residual analytic layers. Each layer is trained via a closed-form “sandwiched” least-squares solution, completely avoiding gradient-based optimization. The authors provide theoretical proofs showing (1) invariance to data heterogeneity when random projections are shared and (2) a monotonic reduction of empirical risk as the network depth increases. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate consistent accuracy improvements and notable reductions in computational and communication costs.", "questions": "1. Clarify theoretical assumptions, including shared random seeds, the impact of regularization, and the influence of finite precision.\n2. Provide quantitative computational profiling (runtime, FLOPs, GPU-hours) to verify the claimed efficiency improvements.\n3. Include comparisons with other analytic or gradient-free deep learning methods beyond AFL.\n4. Add a clear visual diagram explaining the “sandwiched” least-squares process for improved readability.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces the first federated learning framework that achieves deep representation learning without gradients while maintaining closed-form analytical updates.\n2. The idea of stacking analytic layers in a residual manner is a meaningful and elegant extension of AFL.\n3. Reported efficiency gains-around 99% reduction in computation and 50% reduction in communication compared to gradient-based methods-are impressive.", "weaknesses": "1. While theoretical invariance is well-supported, the practical scalability to very large models such as ViTs or LLMs has not been evaluated.\n2. The related work section omits recent multimodal and representation-based FL methods like FedRep [1] and FedU² [2].\n3. No wall-clock runtime, GPU-hour, or FLOP comparisons are provided to substantiate the efficiency claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxp64xuXJT", "reviewer": "Reviewer_ofER", "summary": "This paper proposes DeepAFL, an extension of Analytic Federated Learning (AFL) that introduces analytic residual blocks and deeper analytic layers to enable federated representation learning without gradient calculation and communication. The method alternates between solving for the classifier and transformation matrices​ in closed form, using aggregated feature–label correlation matrices across clients. This paper provides theoretical proofs for heterogeneity invariance and monotonic empirical risk reduction, and the analysing of privacy and efficiency. Experiments evaluate DeepAFL on CIFAR-10, CIFAR-100, and Tiny-ImageNet under Non-IID settings, showing 5–8% accuracy gains over gradient-based FL baselines (FedAvg, FedGen, FedDisco, etc) and the original AFL.", "questions": "1. Could the authors clarify the real motivation and use case of DeepAFL? If all clients already share a strong frozen backbone, what problem is this method actually addressing? Would it make more sense to frame DeepAFL as a collaborative fine-tuning approach, where clients adapt a shared public backbone to private data with different feature distributions?\n\n2. The current experiments assume that the backbone provides perfect and well-aligned features. Could the authors evaluate DeepAFL under domain-gap conditions, where the backbone has never seen the clients’ data or task, to test whether the proposed analytic layers can compensate for feature misalignment?\n\n3. Please include results under IID partitions or with a centralized analytic baseline to verify whether DeepAFL’s improvements come from genuine analytic efficiency rather than its immunity to Non-IID effects.\n\n4. What exactly does “representation learning” mean in this work? If the backbone is frozen and the residual blocks only reweight existing features, are new representations actually being learned? Some analysis of intermediate features, such as layer-wise visualization, separability metrics, or similarity, would help substantiate this claim.\n\n5. Theoretical analyses currently assume ideal, fixed backbone features. Do the invariance and convergence results still hold when the features are imperfect, noisy, or mismatched with the clients’ data? If not, discussing or extending the theory to such realistic cases would greatly improve the paper’s relevance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well-written and clearly structured, with equations and derivations easy to follow. The figures and notations are well explained.\n\n2. Extending analytic learning to a deeper residual form and formulating closed-form residual updates is technically neat and easily understood.\n\n3. This work provides sound theoretical analyses to support the heterogeneous invariance and representation learning capabilities.\n\n4. Due to no backpropagation, this work achieves quite optimization cost reduction, which is demonstrated by experiments.", "weaknesses": "### **1. The experimental setup is methodologically unfair.**\n\nDeepAFL aggregates global feature–label statistics and computes a closed-form model that the authors explicitly state is identical to centralized analytic learning, hence it is inherently immune to data heterogeneity. In contrast, gradient-based baselines (FedAvg, FedProx, MOON, etc.) naturally degrade under Non-IID settings. Evaluating all methods on strongly heterogeneous partitions, therefore, gives DeepAFL an artificial advantage: the method bypasses heterogeneity by design while the baselines must deal with it. \n\nA fair comparison should instead be conducted under IID conditions or against a centralized analytic baseline, so that any remaining advantage would reflect DeepAFL’s genuine strengths in efficiency or analytic formulation rather than the absence of gradient-related heterogeneity effects.\n\n### **2. The representation learning ability is not convincing.**\n\nWhile the paper repeatedly emphasizes that *DeepAFL achieves deep analytic representation learning*, the method it introduces does not actually justify that claim. According to Section 3.1, each residual analytic block is defined as  \n$g_{t+1}(\\Phi_t) = \\sigma(\\Phi_t B_t)\\Omega_{t+1},$  \nwhere $B_t$ is a random projection matrix, $\\sigma(\\cdot)$ is a nonlinear activation, and $\\Omega_{t+1}$ is computed analytically via least squares. This design simply reprojects and reweights the *fixed backbone features* rather than creating new or more abstract representations. All transformations occur within the same frozen feature space extracted by the pretrained backbone, without modifying or adapting the backbone itself. Consequently, DeepAFL indeed enhances the expressiveness of the analytic mapping in this fixed space, but it does not improve the representational capacity or transferability of the underlying model. In other words, the pretrained backbone remains the real bottleneck of representation learning, and DeepAFL does not attempt to address it.  \n\nMoreover, the theoretical and empirical evidence provided does not demonstrate genuine feature learning. Theorem 2 merely proves that the empirical risk decreases monotonically with layer depth, a sign of better fitting, not of learned representations. Similarly, the “representation analyses” in Section 4.2 only report accuracy gains with increasing layers, which reflect improved regression capability but not new feature semantics or cross-domain adaptability.  \n\n### **3.The overall problem setting is conceptually misguided.**\nDeepAFL assumes that all clients share a frozen, pre-trained backbone such as an ImageNet-trained ResNet-18.\nHowever, if this backbone has never been trained on data related to the clients’ tasks, its extracted features may already be poorly aligned with the actual data distributions.\nThis feature-domain mismatch is exactly the challenge that federated learning is meant to address—adapting a shared model to diverse client data without violating privacy.\nThe authors themselves note that “a domain shift between the backbone’s pre-training data and the FL training data can further impact AFL’s performance”, yet the proposed method and theory ignore this issue.\nAll experiments assume that the backbone provides perfectly relevant and transferable features, effectively sidestepping the main difficulty that would justify using FL.\nTo be meaningful, DeepAFL should be analyzed and validated in settings where the backbone features are imperfect or mismatched with client data, since that is where its claimed advantages would actually matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXubtJRaW7", "reviewer": "Reviewer_ZKVc", "summary": "This paper presents DeepAFL, a framework designed to achieve invariance to data heterogeneity in federated learning (FL) while reducing computational cost. Unlike existing analytical federated learning (AFL) methods that are limited to a single layer, DeepAFL extends the idea to multiple nonlinear layers, thereby enabling richer representation learning. Specifically, building upon pre-trained features, DeepAFL updates the model in a layer-wise manner, where it learns an analytical classifier for each layer and subsequently learns a transformation to adapt the layer’s features for improved alignment with the class labels. In the FL setting, local clients transmit information matrices only once to the server, which then aggregates the final model. Experimental results show that DeepAFL outperforms both gradient-based and analytical FL approaches, demonstrating strong invariance to data heterogeneity and improved computational efficiency.", "questions": "Besides the weakness shown in the above section, please also see the following questions: \n\nQ1: DeepAFL uses ResNet-18 as a pre-trained backbone and trains $T$ additional layers on top of it. Do the gradient-based approaches also follow a similar setup, or do they fine-tune the ResNet-18 backbone? If not, would gradient-based methods potentially perform better, or be more stable, if they likewise kept the pre-trained backbone fixed and only trained additional layers? \n\nQ2: Since each intermediate layer in DeepAFL is trained with an auxiliary classifier that shares the same objective and class labels, all layers are encouraged to align closely with the same targets without awareness of how subsequent layers will utilize their features. Could this design lead to redundant representations across layers? For instance, if a given layer already achieves linear separability, the following layer may have little room to learn additional useful information. Would it be possible to incorporate a reconstruction-based objective, such as an autoencoder with MSE, to encourage more diverse and informative feature learning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a well-motivated problem that constitutes a significant and timely advancement for analytical federated learning. \n\n- The proposed DeepAFL demonstrates promising performance compared to existing gradient-based and analytical FL methods, while maintaining strong computational efficiency and invariance to data heterogeneity. \n\n- The experiments are comprehensive and examine multiple aspects, including hyperparameter settings, activation functions, and ablation analyses.", "weaknesses": "- To obtain an analytical solution, DeepAFL relies on the mean squared error (MSE) loss; however, MSE can be sensitive to noise in the data when applied to classification tasks. In realistic federated learning (FL) scenarios, such as hospital applications, clients may contain data annotated by different labelers, leading to slight inconsistencies due to varying labeling standards. The experiments in the paper primarily use clean and well-organized datasets, such as CIFAR and TinyImageNet. Therefore, it remains unclear whether DeepAFL can achieve competitive performance under noisier, real-world conditions compared to gradient-based FL approaches that employ cross-entropy loss.\n\n- Although the paper highlights the importance of representation learning for analytical FL, it lacks visualizations or analyses to demonstrate whether the learned features are semantically meaningful. Moreover, the generalization ability of these features remains unexplored, for example, their cross-domain generalization capability.\n\n\n- The gradient-based FL baselines used for comparison are mostly from studies prior to 2023, lacking evaluation against more recent and advanced approaches such as FedAWA [1] and one-shot FL methods like FedLPA [2].\n\n\n[1] 2025 CVPR FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors\n\n\n[2] 2024 NeurIPS FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)