Beyond Student: An Asymmetric Network for Neural Network Inheritance
Authors: Yiyun Zhou
Jingwei Shi
Mingjing Xu
Zhonghua Jiang
Jingyuan Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Knowledge Distillation (KD) has emerged as a powerful technique for model compression, enabling lightweight student networks to benefit from the performance of redundant teacher networks. However, the inherent capacity gap often limits the performance of student networks. Inspired by the expressiveness of pretrained teacher networks, a compelling research question arises: is there a type of network that can not only inherit the teacher’s structure but also maximize the inheritance of its knowledge? Furthermore, how does the performance of such an inheriting network compare to that of student networks, all benefiting from the same teacher network? To further explore this question, we propose InherNet, a neural network inheritance method that performs asymmetric low-rank decomposition on the teacher’s weights and reconstructs a lightweight yet expressive network without significant architectural disruption. By leveraging Singular Value Decomposition (SVD) for initialization to ensure the inheritance of principal knowledge, InherNet effectively balances depth, width, and compression efficiency. Experimental results across unimodal and multimodal tasks demonstrate that InherNet achieves higher performance compared to student networks of similar parameter sizes. Our findings reveal a promising direction for future research in efficient model compression beyond traditional distillation.

Published as a conference paper at ICLR 2026
BEYOND STUDENT: AN ASYMMETRIC NETWORK FOR NEURAL NETWORK INHERITANCE
Yiyun Zhou
Jingwei Shi 9- Mingjing Xu" + Zhonghua Jiang
Jingyuan Chen
Zhejiang University Shanghai University of Finance and Economies Swansea University
(yiyunzhou, jiangzhonghua, jingyuanchen1ezju. edu. shijingwe @@su.. @ww.eesuteu.....ee.eeeeuuuuueeee edu. cn mingjing. xu@swansea. ac.uk
Demo Code:
2001/IneerNetteeem
ABSTRACT
Knowledge Distillation (KD) has emerged as powerful technique for model com- pression, enabling lightweight student networks to benefit from the performance redundant teacher networks. However, the inherent capacity gap often limits the performance of student networks. Inspired by the expressiveness of pretrained teacher networks, a compelling research question arises: is there a type of network that can only inherit the teacher's 's structure but also maximize the inheritance of its knowledge? Furthermore, how does the performance of such an inheriting network compare that student networks, all benefiting from the same teacher network? To further explore this question, we propose InherNet, a neural net- work inheritance method that performs asymmetric low-rank decomposition on the teacher's 's weights and reconstructs lightweight yet expressive network with- out significant architectural disruption. By leveraging Singular Value Decompo- sition (SVD) for initialization to ensure the inheritance of principal knowledge, InherNet effectively balances depth, width, and compression efficiency. Experi- mental results across unimodal and multimodal tasks demonstrate that InherNet achieves higher performance compared to student networks of similar parameter sizes. Our findings reveal promising direction for future research in efficient model compression beyond traditional distillation.
INTRODUCTION
In recent years, Knowledge Distillation (KD) (Hinton et al. 2015) has gained widespread attention and recognition in both academia and industry as an efficient model compression technique (Zhou et al., 2018 Xu et al. 2024b). Its core idea to transfer knowledge from large, pretrained teacher network to a lightweight student network, thereby improving the student's performance without significantly increasing additional computational cost. Initially, KD is mainly applied to unimodal tasks, such as visual tasks (Hinton et al. 2015 Jin et al. (023) and textual tasks (Jioo et al. 2019 Kang et al. 2023). With the advancement of multimodal learning, KD has also been extended multimodal scenarios for modsl knowledge transfer and fusion (Dai et al. 2022 Yang et al. 2024).
+
Trainable Parameters:
Pretrained Weights
A-
X
(a) LoRA approximates the weight updates AW.
(b) NNI approximates the
pretrained weights W.
Figure 1: The difference between LoRA and NNI.
"Equal contribution.
1 "orresponniig author.

---

Published as a conference paper at ICLR 2026
However, despite its effectiieness, widely accepted view is that the performance of student net- works is typically inferior to that of teacher networks due to limited capacity (Gou et al. [021 Zhou et al.] 2025b). This observation draws parallel to Ceaaneter-Effiiient Fine-Tuming (PEFT) Mann-
grulkar et al., 2022 Xu et al. 2023] Zhou et al., 20251), especially the LoRA method (Hu et al.)
2022). As illustrated in Figure (a), given the pretrained weights W RR=x" LoRA approximates the weight updates AW using a pair of low-rank matrices: A E Rmxr (initialized with Gaussian noise) and B xx*"" (initialized with zeros), where r < min(m, n). This approach significantly reduces the number of trainable parameters from m x n to r(m + n). However, LoRA generally still exhibits a performance gap compared to full tinettunin (Zhang et al. 2023b Ding et al. 2023
Han et al. 2024).
Given the strong performance of teacher networks in KD, the difficulty of designing student networks to adapt to the teacher network (Gu & Tresp, 2020; Liu et al., 2022), and the pa- rameter efficiency of low-rank approximations in LoRA, we are naturally led a more affinitive model compression approach: Neural Network Inheritance (NNI). Instead of training a separate stu- dent network, NNI directly inherits the capacity of the teacher network in a low-rank form. As shown in Figure T (b), NNI achieves model compression by performing low-rank decomposition on each layer of the teacher network and replacing the original weights with their decomposed counter- parts. Crucially, unlike LoRA which approximates weight updates AW, InherNet approximates the full pre-trained weights W, preserving the teacher model's expressiveness more directly.
Upon reviewing prior work, we find that general and widely applicable method for NNI is lacking. Existing related efforts are rather fragmented. mainly focusing on early Convolutional Neural
Network (CNN) architectures (Jaderberg al., 2014; Zhang et al., 2015) Xu et al. 2019 Yang
et al. 2020). and are mostly confined to vision tasks, with limited applicability to other modali- ties- particularly the increasingly prominent multimodal tasks. Moreover. these methods (Lebe-
dev et at.) 2014 Hu et al.] (024) often rely on traditional tensor decomposition techniques (e.g., CP decomposition and MPO decomposition), which break a single layer into multiple sub-layers for compression, often making networks deeper and narrower-raising risks of vanishing or exploding gradients and compromising stability and performance. More importantly, there re- mains a lack of investigation into the teacher inheritance under KD, especially in terms of sys- tematically comparing inherited and student networks in performance and efficiency.
To this end, we propose InherNet- novel and general method for efficiently inheriting the knowledge and structure of teacher networks in KD. For knowledge inheritance, unlike prior decomposition--asse methods (Jaderberg et al. 2014), InherNet leverages Singular Value Decom- position (SVD) for initialization (SVD offfine single-pass computation makes the training time negligible), allowing the decomposition of only two layers without excessively deepening the net- work. For structure inheritance, InherNet is architecture-agnostic and applies low-rank approxi- mation to convolutional or linear layers. Previous studies have shown that Mixture of Experts (MoE) models can significantly enhance performance across various tasks (Zhou et al. 2022 Tian et al., 2024 Gao et al., 2024) Cai et al., 2024). Inspired by this, InherNet adopts a similar design, enabling both deepening and widening of the network to mitigate the potential risks of gradient vanishing or explosion. This design enables InherNet to remain structurally aligned with the teacher network while also inheriting its representational capacity more faithfully. Notably, our experiments reveal a compelling insight: InherNet converges significantly faster than traditional student networks. We attribute this phenomenon to the extended SVD-based initialization and provide detailed the- oretical analysis to support this claim. Additionally, each layer of the teacher network inherited by InherNet uses fixed asymmetric design (i.e., single dimensionality reduction matrix and multiple dimensionality expansion matrices). We provide empirical evidence and theoretical support for this fixed design in Appendix
To summarize, our contributions are in four aspects:
We propose InherNet, novel method that explicitly inherits both the knowledge and struc- ture of powerful teacher networks in KD. Unlike implicit knowledge distillation, InherNet constructs wider, deeper, yet lightweight network without the challenges of heterogeneous knowledge transfer.
We uncover and analyze three pivotal insights regarding InherNet, as presented in Sec. 3.4 These findings illuminate the underlying principles of neural network inheritance and offer clear directions for future research in this area.
2

---

Published as a conference paper at ICLR 2026
InherNet
SVD-driven Knowledge Inhenihance
Expert-style Structural
Reshaped
Reshape
R gacisss, HRg
Vi
Expand
19
pia Min
Router
Uprt
Miro
Widren U Marloporr
4D Reshape
1. Decomposed
Convolutional Layer Decomposition
W g. & 1,,",
Multipicatioon
geloos B Marcts
Figure 2: Overview of the proposed InherNet (e.g., decomposition a convolutional layer). Inher- Net consists of two parts: (a) Knowledge Inheritance and (b) Structure Inheritance. Note that the low-rank decomposition of linear layer is similar, with the key being to satisfy the properties of the 2D SVD operation.
We provide extensive and rigorous theoretical analysis to explain the empirically observed phenomenon of InherNet's accelerated convergence with comparable parameter sizes. Our proofs formally attribute this efficiency advantage to its SVD-based initialization, estab- lishing a solid theoretical foundation for its design.
We conduct systematic experiments across various network architectures and modal tasks, including both unimodal and multimodal scenarios. The results consistently demonstrate that InherNet outperforms traditional student networks in both efficiency and effectiveness.
2 METHODOLOGY
In this section, introduce our proposed InherNet method, which consists of knowledge inheri- tance and structure inheritance, as shown in Figure Next, we will provide theoretical analyses of the method's convergence guarantees and parameter efficiency.
2.1 INHERNET
2.1.1
KNOWLEDGE INHERITANCE
For a linear layer, consider a pretrained teacher network with a weight matrix W E Rm InherNet employs SVD to factorize W into compact low-rank approximation. Specifically, we have:
W UVV U.E.V...
(I)
where U, E RR Σ, E Rexr, and V, E RRx represent truncated matrices derived from the top-r singular values and vectors. Therefore, the original layer is approximated by two consecutive low-rank layers, U,Ei and V.T. The approximation is supported by Theorem 2.1
Theorem 2.1 (Eckart- Boung-Mirsky theorem). Given a weight matrix W RRxx with singular value decomposition U.VT, the optimal runk-r approximation in terms of Frobenius norm is given by: W, - [arr] Moreover, the approximation error is minimized, specifically:
min(m.n)
|W W
Σ
dW()).
where (W) denotes the i-th singular value of W (Golub et al. 1987 Eckart Young 1936 Golub Van Loan 2013)

---

Published as a conference paper at ICLR 2026
For convolutional layer, shown in Figure K E xxNxxxxxxXX XKa, where n, c, kws kn, represent the number kernels, the number of input chan- nels, and the width and height of the filter, respectively. Our work focuses the channel decom- position method (Zhang et al. 2015) to decompose the convolution layer. Specifically, K is first reshaped into 2D matrix K E (ex and SVD initialization is applied. The resulting decomposed layers are reshaped to have convolutional kernel weights Wrdown IR [R~xxxlx1 and W E R' xex Rma xkn
2 (a), formally, there exists a convolution kemell
2.1.2 STRUCTURE INHERITANCE
Unlike previous low-rank decomposition methods (Jaderberg et al., 2014 Zhang et al. 2015 Zhou al. 2025c), we adopt an expert-style inheritance structure inspired by the Mixture of Experts (MoE) paradigm, which has been widely proven effective across various domains (Tian et al. 2024
Gao et al. 2024 Cai et al., 2024). As illustrated in Figure 2 (b). in order strike good balance between performance and efficiency, we design an asymmetric expert-head structure. Specifically, given input features X IR"", the output Y E IR" of InherNet is formulated as:
Y= Σ (x) Wh (Widoum (X)) h=1
where denotes the number of expert heads, W'dow and W represent the "down" projection of the backbone branch and the "up" projection of the h-th expert head, respectively. Wodow and W
x1/2
are initialized using [.arr] 1///2 (sraar and
from Eq, and the matrix slicing notations follow the same convention as in PyTorrc. G(X) represents adaptive gating fusion weights computed as:
Hf
G(X) = softmax (W*())).
(4)
where W E RR is learnable parameter.
We also provide empirical and theoretical support for InherNet's fixed asymmetric structural design in Appendix [ Note that InherNet inher- ently benefits from SVD and gating mechanism, theoretically enabling excellent convergence and parameter efficiency, as rigorously analyzed in Sec. 2.22 and Sec. 2.3 By directly inheriting struc- tural knowledge through decomposition and adaptively aggregating expert heads, InherNet provides an efficient and expressive solution, demonstrated by our experiments in Sec.
The InherNet's procedure can be found in Appendix B.
2.2 THEORETICAL ANALYSIS CONVERGENCE GUARANTEES
We analyze the convergence properties of InherNet, focusing its structured low-rank decompo- sition, multi-head gating mechanism, and SVD-based initialization. Our goal is to establish that, despite the compression from low-rank projections, InherNet has stable and efficient optimization under stochastic gradient descent (SGD).
Let C())- [ ((( v)] be the expected loss, where (I, P denotes data sampled from the underlying distribution. Let 0 e IRd denote the parameters of an InherNet, and let g(t) be the model parameters at iteration & updated via stochastic gradients.
We impose the following assumptions, which are common in the analysis of stochastic optimization for nonconvex problems (Patel et al.) [2022] Xu et al. 2024a Lei et al. 2019 Pham al. 2020):
Assumption 2.1 (Lipschitz Continuity). The expected loss gradient is L-Lipschitz continuous:
||V[()) VL((')| s L|le 8'11. 00 E Ed. Assumption 2.2 (Bounded Variance). The stochastic gradient (o((oo())))) is unbiased and has bounded variance:
(5)
E [(Vo((o(<))))) = VL()), E [iVoe- < a²
Assumption 2.3 (Bounded Representations). There exist constants B,, By > such that ||>||| S B, and [=]] B, for all (x. y) ~ P.
4

---

Published as a conference paper at ICLR 2026
2.2.1 GRADIENT DECOMPOSITION UNDER INHERNET
InherNet models each module a low-rank gated composition:
fx Σ Gx; W
(W ( (x)d)o(())))
where Wrdww the shared projection, Wh is the h-th expert head, and G+(() is a gating weight. Lemma 2.2 (Gradient Decomposition). Under the InherNet parameterization, the gradient of the loss splits naturally into contributions from each head and from the gating network. The total gra- dient V Vel(fo(x),)) decomposes into:
H
VolSo(z).y) - GGn(T))))))) (o. + A.
(8)
h=1
h=1
where On and 0g are the parameters of expert head and the gating network, respectively, and f represents the gradient of the loss with respect to the output of the h-th head, with
An 01a(2) fa(()) = W (Wrdom()))
This gradient routing allows InherNet to specialize experts during training, directing gradient flow to the most relevant parameters for each input. It is beneficial when compressing teacher networks.
2.2.2 EFFECT OF SVD-BASED INITIALIZATION AND CONVERGENCE GUARANTEE
Let W denote the pretrained weight from a teacher network. InherNet initializes each module using truncated SVD. The initialized parameters are: Wrdown U, W -,
Proposition 2.3 (Stability via Orthonormal Initialization). If U, and Vr are initialized with or- thonormal columns, then under Assumption 2.7] the effective gradient Lipschitz constant reduced from L to L' z L/K, where is the condition number of W.
This improved conditioning reduces curvature imbalance during early training and stabilizes con- vergence, particularly when the teacher model is well-trained (Wang al. 2024 Lim & Ye 2024 Saxe et al. 2013).
Theorem 2.4 Conv-Convex Convergence). Under Assumptions 2.1 2.3 and adopting a diminishing
learning rate schedule 1/. 3 the sequence of parameters (00) generated by InherNet satisfies the following convergence rate:
T Σ E 1V
VT
The convergence rate matches classical result for non-convex optimization problems (Xu et al, 2024a). Because of the SVD-based orthonormal initialization and adaptive gating mechanism, In- herNet achieves convergence rate of O(1/VT)) with practical scaling as OLLKk). The improved convergence behavior stems from Proposition 2.3 where the Lipschitz constant for gradient updates is reduced from L to L' = L/k. This reduction in curvature imbalance commits lower gradient variance and faster gradient descent early stages, as empirically verified in Sec. 3.4
2.3 THEORETICAL PROOF OF PARAMETER EFFICIENCY
We analyze parameter efficiency of InherNet, establishing theoretical guarantees on both param- eter reduction and representational power preservation. Our analysis formalizes the efficiency- expressivity tradeoff through three metrics:
Definition 2.1 (Parameter Efficiency (PE)). For network fo parameterized by 0 RP, define PE(fa) Representational Caps Representational capacity measures the network's 's approximation
ability and |0| denotes parameter count.
Definition 2.2 (Approximation Error). Given a target function f and a neural network fo. the approximation enror is E(Jo, f*) (eeeDllloo(a [*(x))]]), where D is the data distribution.
5

---

Published as a conference paper at ICLR 2026
Definition 2.3 (Expresiivity-t-PPrrmeeee Ratio (EPR)). The EPR of network fo with respect to function class F is (fo. = el E(Jo.f") measuring the inverse minimum achievable approximation error per parameter.
2.3.1 COMPRESSION BOUNDS FOR INHERNET
We first quantify the parameter reduction achieved by InherNet relative to standard network, which directly improves the PE defined in Definition 2.1
Theorem 2.5 (Parameter Reduction Bounds), Consider layer W E Rmxx. The InherNet param- eterization with rank r and H heads uses (m a n) + (r + +1) trainable parameters, yielding compression ralio:
17. 72
m n
(1+0(
(10)
Hr (m)
H (m + +n)
Thus, achieves an approximately p-fold increase Definition 2.1 rank-H subspace preserves the layer's representational capacity.
under the assumption that
To ensure low Approximation Error (Definition 2.2 despite compression, we establish:
Lemma 2.6 (Spectral Energy Preservation). Let W E <Rmx have singular values 01 202 2
mmin(m,x) If the rank r is selected such that:
Ei=1 o
21
(11)
min(m.n) of
Di=1
then by Theorem 2.1 the Frobenius-norm error of the optimal rank-r approximation W, is bounded by: I|W W S I|!. Consequently, the Definition 2.2 between an InherNettcompressee layer and the original layer is bounded by under the L2 norm squared.
Proposition 2.7 (Knowledge Preservation Rate). Let fw be a teacher network with weight matrices [Wi) and fo be the corresponding InherNet with rank r. The functional similarity between fw and fo is
by:
L
Sim(fw fr) 2 1
Σ ai 1=1
1
1=1
(12)
Eist
mmi (min)())
where ay represents the layer's 's influence on output, and Ts are the singular values of Wi. This bound is monotonically increasing with but not with H, establishing that rank is the primary metric of inherited knowledge preservation.
2.3.2 REPRESENTATIONAL POWER PRESERVATION
InherNet also maintains the representational power of the original network, preserving beneficial EPR (Definition 2.3)
Theorem 2.8 (Representational Power Preservation). Fw be functions representable by a stan- dard network with weights (Wi)-r- For any f* Fw and 8 0, there exists InherNet network with appropriate ranks {ri} and head counts {Hi} such that the Approximation Error (Definition 2.2 satisfies:
E(finhereet f²) 8
(13)
while the parameter count is reduced by M(min; min(my,mi)) relative to the standard network. Con- r.
sequently, the InherNet architecture attains an EPR (Definition 2.33 higher than that the standard architecture by at least the same factor.
This preservation stems from two complementary mechanisms: Layer-wise low-rank approx- imation: Capturing the dominant spectral components of each weight matrix; 2. Multi-head spe- cialization: Allowing different heads specialize on different input subspaces.
Corollary 2.9 (Universal Approximation). InherNet retains the universal function approximation property of standard neural networks while using asymptotically fewer parameters. For any con- tinuous target function, sufficiently large InherNet network approximate it arbitrarily well (to Approximation Error + 0) with significantly higher Parameter Efficiency and EPR than standard network of comparable accuracy.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of top-1 Accuracy (%) on the CIFAR-100 validation set between InherNet and previous ffte-of-the-ar knowledge distillation methods. Best results are in bold, and the second-best are underlined.
ResNet32x4
VGG13
WRN-40-2
WRN-40-2
ResNer56 72.34
ResNerl 74.31
ResNetl 10
Teacher
79.42
74.64
75.61
75.61
74,31
KD Type
ResNet8x4
VGG8
WRN-40-1
WRN-16-2
ResNet20
ResNer32 71.14
ResNer20
Student
72.50
70.36
71.98
73.26
69.06
69.06
CRD
75.51
73.94
74.14
75.48
71.16
73.48
71.46
OFD
74.95
73.95
74.33
75.24
70.98
73.23
71.29
Feature
ReviewKD
75.63
74.84
75.09
76.12
71.89
73.89
71.34
SimKD
78.08
74.89
74.53
75.53
71.05
73.92
71.06
CAT-KD
74.65
71.62
71.37
73.33
72.98
73.54
74.92
70.66
73.08
70.67
KD+CTKD
73.39
73.52
73.93
75.45
71.19
73.52
70.99
Logit
DKD
76,32
74,68
74.81
76.24
71.97
74.11
71.06
MI.DD
77.08
75.18
75.35
76.66
72.19
74.11
71.89
MoKD+Logi Std.
75.22
74.32
72.27
Small
75.68
76.04
72.67
74.13
74.13
InherNet
Large
78.53
75.16
76.39
76.39
73.67
75.88
75.88
Proposition 2.10 (Diminishing Returns of Multiple Heads). For given layer with rank r, the marginal reduction in approximation error from adding the (H + 1)-th head satisfies:
(r, H) Err, H +1 H2 E(r,
C
(14)
where e is a constant depending on imput distribution characteristics, and (r, H) denotes the ap- proximation error with rank r and H heads. This establishes that benefits from additional heads diminish at rate of (
The theoretical results explain empirical insights Sec. 3.4 (1) rank plays the primary role in
preserving teacher knowledge as formalized in Proposition 2.7 and (2) while multiple heads outper-
form a single head, adding more heads yields diminishing returns as quantified in Proposition 2. 2.10 Detailed theoretical proofs and extended discussions are provided in Appendix
3 EXPERIMENTS
We provide all implementation details (including detailed introduction to the datasets and base- lines) of the experiments, along with the parameter sizes of the teacher and student networks and InherNet (introducing two variants, InherNet Small and InherNet Large), in Appendix C.1
3.1 VISION: IMAGE CLASSIFICATION
In this task, we conduct extensive experiments to validate the advantages of the proposed InherNet over traditional KD methods. The experimental setup is follows:
Datasets. We use two widely adopted image classification datasets CIFAR-100
Krizhevsky &
Hinton 2009) and ImageNet (Russakovsky et al. 2015).
Baselines. We compare against several mainstream KD methods, including logit-based methods:
KD (Hiton al. 2015). CTKD (Li al, (023b), DKD (Zhao et al. 2022), MLKD (Jin et al.
2023 and Logit Std. (Sun et al. 2024); as well as feature-basee methods: CRD
(Peng et al. 2019)
OFD (Heo et al. 2019). ReviewKl (Chen et al. 2021). SimKD (Chen al. 2022 and CAT-
KD
Guo et al. 2023).
3.1.1
RESULTS ON CIFAR- 100
Table compares the top-1 accuracy of InherNet with previous state-of-the-art KD methods. All methods used 240 epochs, except for MLKD for 480 epochs. The choice of 240 epochs is because all methods, including InherNet (even with faster convergence due to SVD initialization, as validated by Insight in Copendix[C.4laaaa
Figure converge within this period, except for MLKD. With 240 epochs, MLKD has not yet converged; using ResNet56 as the teacher and ResNet20 as the student, MLKD even achieves lower accuracy than the student's. This result is consistent with previous

---

The experimental setup is Published as a conference paper at ICLR 2026
studies Jin al. (2023); Sun al. (2024). In contrast to such unfair evaluations, exclude the unfair baselines in our other analyses. The main findings are as follows:
Among all baselines, the recently proposed MLKD and Logit Std. achieve the best performance. This may be due to the unfair comparison settings in previous works (Jin et al., 2023 Sun al. 2024), where these methods are trained for more epochs. as they converge more slowly and per- form poorly with fewer epochs (e.g., with 240 epochs, MLKD only achieves 60.05% accuracy when using ResNet56 as teacher and ResNer20 as student). In contrast, other KD methods use fewer epochs but show limited improvement.
Under fair experimental settings (excluding MLKD and Logit Std.), significant performance gap remains between student and teacher networks. This is not only due to the limited capacity and architecture of the student network. but also because of the underlying objective of KD-using the teacher network as reference to guide the student network toward its performance limit.
The proposed InherNet strikes balance between efficiency and effectiveness. With fewer epochs, the lightweight InherNet achieves competitive performance compared to the state-of-the-ar KD method Logit Std., while the slightly larger InherNet Large consistently outperforms other methods and significantly narrows the performance gap with the teacher network. We attribute this the proposed extended SVD initialization, which inherits most of the teacher's forward knowledge, and the asymmetric MoE inheritance structure, which enables robust generalization during training. Moreover, InherNet empirically shows faster convergence (Sec. (3.4) and bet- ter parameter efficiency. backed by theoretical convergence guarantees (Sec. 2.2 and parameter efficiency proofs (Sec. 2.3
3.1.2 RESULTS ON IMAGENET
In Apppendix C.2
we provide experimental results on ImageNet, showing that InherNet Large con- tinues to demonstrate leading performance the large-scale dataset, while InherNet Small achieves a good balance between efficiency and effectiveness. All methods are trained for 100 epochs.
3.2 LANGUAGE: GLUE BENCHMARK
In this task, we evaluate InherNet's performance on multiple text tasks. follows:
Datasets. To remain consistent with prior research (Raffel et al. 2020). we train and evaluate
InherNet's performance the GLUE benchmark (Wang al. 2019).
Baselines. Our experiments are based on the T5 model (Raffel et al. 2020 implemented Hug- gingFace (Wolf et al.) 2020). Due to the lack of widely known and reproducible T5-based KD methods, we use the traditional KD method (Hinton et al. 2015). with teacher-student setup of T5-Base/T5-Smm with parameters 222M/60M.
Beyond the T5 experiments, we also apply InherNet to a larger decoder-only LM, LLaMA-2-7B Raf- Tel et al. 2020); Touvron al. (2023). For such autoregressive LMs, conventional token-level KD is difficult to deploy due to the 32K dimeennnsi vocabulary softmax, strict teacher-forcing require- ments, and tokenizer mismatches between teacher and student, so we do not include standard KD baselines here. Instead, we compare self-distileed I.LaMA-2-7B baseline (teacher and student shar- ing the same weights, offering no compression) with an InherNet model constructed from the same teacher. As detailed in Appendix C.3] the resulting 4.2B-parameter model slightly outperforms the 7B teacher on GSMSK while maintaining competitive performance on MMLU-style evaluations, indicating that our inheritance mechanism scales to large Transformer LMs without relying KD.
Table 2 shows the performance of InherNet and the distillation method on GLUE tasks. As shown in the table, there is still performance gap between the distilled student network and the teacher network. However, the InherNet method we proposed narrows this gap through explicit inheritance, which distinguishes it from the traditional implicit learning approach used in KD.
8

---

Published as a conference paper at ICLR 2026
Table The performance (%) of InherNet and the distillation method on GLUE tasks. For MRPC, QQP, SST-2, MNLI, RTE, and QNLI, we report Accuracy. For CoLA, we report the Matthews
correlation coefficient. For STS-B, we report Pearson and Spearman correlation coefficients.
Method
MRPC
QQP
SST-2
MNLI
RTE
QNLI
COLA
STS-B
T5 15-Base
89.21
86,54
90.71
54.93
77.98
90.13
57.00
89.04 / 88.88
T5-Small
88.38
81.05
87.78
53.67
72.71
87.97
48.12
85.55 86,50
T5- KDDlllll-K
88.23
84,92
89.01
53.81
74.31
88.76
49.31
88.05 87.85
InherNet
88.74
84.69
89.07
56.34
72.93
90.29
50.63
86.70/ / 86.36
Table 3: Comparison of cross-modal retrieval on the CC3M validation set and zero-shot classi-
fication on ImageNet validation set performance (%) between InherNet and the student networks distilled with CLIP-KD trained on CC3M dataset.
12T
T21
Classification
Method
R@0 1
R@5
R 10
R@1 1
R
R@ 10
top-1
top-5
ResNet-101
30.58
53.95
63.49
29.31
54.13
63.31
15.70
32.75
ResNet-18
31.03
56.41
65.82
30.78
55.74
65.35
15.93
33.83
RfficientNee-B00
33.65
58.72
68.28
32.65
58.21
67.92
17.30
35.75
MobileNetV3
29.07
53.43
63.59
28.10
53.11
63.35
14.59
31.68
InherNet
32.17
57.17
67.82
33.01
59.88
68.72
17.39
36.65
3.3 VISION + LANGUAGE: MULTI-MODAL EVALUATION
In this task, we validate the effectiveness of InherNet in multimodal scenarios and compare it with the recent CLIP-KD (Yang et al. 2024). The experimental setup is as follows:
Dataset. We Conceptual Captions 3M (CC3M) (Sharma et al. 2018) for visual- language pre- training. Additionally, we use the ImageNet validation set for zero-shot classification, with the evaluation protocol consistent with CLIP-related works (Wei al. 2022 Li et al. 2023a).
Baselines. In this task, ResNet- 101 serves as the teacher network, while ResNet-18 (He et al. 2016), EfficientNet-OO (Tan & Le 2019) and MobileNetV3 (Howard al. 2019) used students.
Table shows the cross cross-modal modal retrieval performance on the CC3M validation set and zero-shot classification performance ImageNet validation set for InherNet and its student network distilled through CI CI.IP-KD trained on the CC3M dataset. The proposed InherNet generally outperforms other baseline methods in various multimodal tasks, with its performance consistently significantly higher than that of the teacher network, further validating the effectiveness of InherNet in multimodal scenarios.
3.4 INSIGHTS ON INHERNET
Based on the image classification task, conduct analytical experiments under the teacher-student setup of ResNet56/ResNet20 We provide experimental details and results for all insights in Ap- pendix C.4 The observed insights are follows.
Insight 1. Distillation benefit small-scale InherNet, but harms the performance of large-scale InherNet. It is aligned with our theory: as the rank grows, the InherNet fr converges in function space toward the teacher fw in Proposition 2.7 In the high-rank regime, the task loss alone can drive it that generalize slightly better than the teacher. In contrast, the KD loss penalizes any deviation from the teacher's logits, acting as an overly strong regularizer. More details are shown in Appendix B.4
Insight 2. Rank significantly affects model performance; using multiple expert heads performs better than a single head.
Insight 3. Compared to traditional distillation methods, InherNet converges significantly faster.
3.5 ABLATION STUDY

---

Published as a conference paper at ICLR 2026
We construct three variants of InherNet Lage with the teacher network ResNet56 CIFAR-100 to investigate the critical roles of different components. Specifically, "w.o. svd" denotes the absence of SVD initialization, "w.o. gate" removes the gating mechanism, and "w. sym.. adopts symmetric structure (with two branches and two expert heads, similar to LoRA+MoE (Liu al.] 2024)). The experimental results are shown in Table 4 from which we observe the following: (1) SVD initialization contributes the most to performance improvement. This is not only because allows the network to inherit a substantial amount of beneficial knowledge from the teacher network, but also because it stabilizes training, accelerates convergence, and helps the model find the optimal solution more quickly and accurately, as illustrated in Insight 3 of Sec. 3.4 (2) The gating mechanism enables effective dynamic routing in most cases, and it is especially crucial for lightweight inherited networks. (3) Under the same parameter size, the symmetric architecture underperforms compared asymmetric ones. This is because the asymmetric architecture introduces a structural inductive bias (Parton & Engelbrecht, 2020), which enhances the diversity among experts.
Table 4: Ablation study of different variants on InherNet.
Variants
ResNet32x4
WRN-40-2
ResNet56
ResNet110
w.o.
76. 68
75.42
69.35
74.13
w.o. gate
78.17
76.08
73.22
75.64
sym. 77.92
75.98
73.28
75.45
InherNet
78.53
76.39
73.67
75.88
RELATED WORK
We summarize previous research of knowledge distillation (KD) (Hinton al. 2015) and low-rank decomposition (Denton et al. 2014); extensive discussion and comprehensive comparisons appear in Appendix D
5 CONCLUSION
This paper introduces InherNet, a novel approach beyond student that directly inherits both the structure and knowledge of teacher networks. By applying low-rank approximation and leveraging singular value decomposition, InherNet efficiently compresses models while maintaining the ex- pressiveness and performance of the teacher network. Our method demonstrates faster convergence compared to traditional student networks and offers a more stable and effective solution for model compression.
ACKNOWLEDGMENTS
This research was partially supported by grants from the "Pioneer" and "Leading Goose" R&D Program of Zhejiang under Grant No. 2025C02022, the National Natural Science Foundation of China (No. 623602307,,,,, and Shanghai Rising-Star Program (23QA 1409000), Additionally, this work was partially supported by the UK Engineering and Physical Sciences Research Council (EPSRC) [EP/W524694/1].
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
This work is guided by the ethical principles outlined in the ICLR Code of Ethics, prioritizing re- sponsible research practices, scientific rigor, and the broader well-being of society. We recognize the global community of stakeholders in machine learning research and aim to ensure our contributions are beneficial to society while minimizing potential risks. Our research adheres to high standards of integrity, transparency, and reproducibility, with methods and results presented honestly and ac- curately. We have carefully considered the wider implications of our work, addressing potential concerns related to privacy, safety, and fairness, and have consulted with domain experts to miti- gate any unintended consequences. All data used in this study has been handled line with ethical approvals, safeguarding privacy and confidentiality. We are committed to promoting inclusivity, avoiding bias, and ensuring that our findings are both accessible and socially responsible.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information 2016.
bottleneck. arXiv preprint urXivv1612.000
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.066 2407.0624, 2024.
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In AAAI, 2020.
Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In CVPR, 2022.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21100 14168, 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instructiontuned 1lm. 2023.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on clip via vision-language knowledge distillation. arXiv preprint urXiv:2203.0633 2022.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211-33 1-288, 1936.
11

---

Published as a conference paper at ICLR 2026
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint 0825:22402.0866., 2024.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Gene H Golub, Alan Hoffman, and Gilbert W Stewart. A generalization of the young---ou matrix approximation theorem. Linear Algebra and its applications, 88:317-327, 1987.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision. (9(())1189--181 2021.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the vin vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 6913, 2017.
Jindong Gu and Volker Tresp. Search for better students to learn distilled knowledge. arXiv preprint arXiv:2001.11612 2020.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. arXiv preprint arXiv:2306. 2023.
Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In CVPR, 2023.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficien fine-tuning for large models: A comprehensive survey, arXiv preprint arXiv:2403. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. 2020.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.025:: 2015.
Andrew Howard. Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference computer vision, pp. 1314-1324, 2019.
Chengming Hu, Xuan Li, Dan Liu, Haolun Wu, Xi Chen, Ju Wang, and Xue Liu. Teacher-student architecture for knowledge distillation: A survey. arXiv preprint arXiv:2308. 04268, 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Xiaolin Hu, Xiang Cheng, Peiyu Liu, Wei Liu, Jian Luan, Bin Wang, and Yong Liu. Dota: Weight- decomposed tensor adaptation for large language models. arXiv preprint arXiv:2412.20891, 2024.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 1405:344 2014.
Zhonghua Jiang, Kui Chen, Kumxi Li, Keting Yin, Yiyun Zhou, Zhaode Wang, Chengfei Lv, and Shengyu Zhang. Acckv: Towards efficient audio-video Ilms inference via adaptive-focusion and cross-calibration kv cache optimization. arXiv preprint arXiv:2511. 1,1 2025a.
12

---

Published as a conference paper at ICLR 2026
Zhonghua Jiang, Kunxi Li, Yiyun Zhou, Sihao Liu, Zhaode Wang, Shengyu Zhang, al. Purekv: sparse attention for vision-language
Plug-and-play kv cache optimization with large models. arXiv preprint arXiv:2510.25000 2025b.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 2019.
Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Junmo Kang, Wei Xu, and Alan Ritter. Distill or annotate? cost-efficient fine-tuning of compact models. arXiv preprini TrXiv:2305.01455 2023.
Christof Koch and Shimon Ullman. Shifts in selective visual attention: towards the underlying neural circuitry. In Matters of intelligence: Conceptual structures in cognitive neuroscience, pp. 115-141. Springer, 1987.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.
Yunwen Lei, Ting Hu, Guiying Li, and Ke Tang. Stochastic gradient descent for nonconvex learn- ing without bounded gradient assumptions. IEEE transactions on neural networks and learning systems, 31100)4334444400 2019.
Kunxi Li, Yufan Xiong, Zhonghua Jiang, Yiyun Zhou, Zhaode Wang, Chengfei Lv, and Shengyu Zhang. Flowmm: Cross-modal information flow guided kv cache merging for efficient multi- modal context inference. arXiv preprint arXiv:2511.055: 2025.
Quanjiang Li, Tianxiang Xu, Tingjin Luo, Yan Zhong, Yang Li, Yiyun Zhou, and Chenping Hou. Thivey-driven label-specific representation for incomplete multi-view multi-label learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking, In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23390-23400, 2023a.
Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In ACCV, 2020.
Zheng Li, Jingwen Ye, Mingli Song, Ying Huang, and Zhigeng Pan. Online knowledge distillation for efficient pose estimation. In ICCV, 2021.
Zheng Li, Xiang Li, Lingfeng Yang. Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In AAAI, 2023b.
Lek-Heng Lim and Ke Ye. Simple matrix models for the flag, grassmann, and stiefel manifolds. arXiv preprint arXiv:2407.13448 2024.
Sihao Lin, Hongwei Xie. Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. CVPR, 2022.
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets Imss: Parameter efficient fine-tuning for multi-task medical applications, 2024.
URL httph:://///aaaeeee /ttps:///axvv..ooo http:/awwvvvv.o..aaaaaaaoo httpg/bbs/231.... 18339
Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingting Ding, and Liang Li. aresttrrch knowledge distillation. In Proceedings of the Asian conference on computer vision, pp. 3396- 2022.
13

---

Published as a conference paper at ICLR 2026
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.5101,,22 2017.
Xiangwei Lv, Jingyuan Chen, Mengze Li, Yongduo Sui, Zemin Liu, and Beishui Liao. Grasp the key takeaways from source domain for few shot graph domain adaptation. In Proceedings of the ACM on Web Conference 2025, pp. 2330-2340, 2025.
Sourab Mangrulkar, Sylvain Gugger, I. Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-ae parameter-efficiest fine-tuning methods. https://githhtn t/github...b...... com/
huggingface/peft) 2022.
Siyuan Mu and Sen Lin. comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprini
2025.
James R Norris. Markov chains. Number 2. Cambridge university press, 1998.
Callum Parton and Andries Engelbrecht. Mixtures of heterogeneous experts. In Proceedings of the 2020 4th International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, pp. 1-7, 2020.
Vivak Patel, Shushu Zhang, and Bowen Tian. Global convergence and stability of stochastic gradient descent. Advances in Neural Information Processing Systems, 35:36014-36022 2022.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In (CCV,2019)
Nhan H Pham, Lam M Nguyen, Dzung T Phan, and Quoe Tran-Dinh. Proxsarah: An efficient algorithmic framework for stochastic composite nonconvex optimization. Journal of Machine Learning Research, 10)1 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763, PmLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Rajarshi Saha, Varun Srivastava, and Mert Pilanci. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36:18828- 18872, 2023.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37:88981 1189998888901..88 2024.
Kazuyuki Samejima, Kenji Doya, and Mitsuo Kawato. Inter-module credit assignment in modular reinforcement learning. Neural Networks, 16(7):985-994, 2003.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2018
Shangquan Sun, Wengi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 15731-15740, 2024.
14

---

Published as a conference paper at ICLR 2026
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, al. Convolutional neural networks with low- rank regularization. arXiv preprint arXiv:1511.0605 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net-
works. In International conference on machine learning, pp. 6105-616 14. PMLR, 2019.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565 9584, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Wang Wang, Mingjing Xu, Zhen Cao, Jingzi Guo, Chong Liu, Haowei Zhang, and Xiaoling Zhang. Unified data synthesis for automated 3d visual inspection and digital twinning of bridges. Au- tomation in Construction, 182: 2026.
Wenxiao Wang. Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748, 2024.
Yixuan Wei, Yue Cao, Zheng Zhang, Zhuliang Yao, Zhenda Xie, Han Hu, and Baining Guo. icar: Bridging image classification and image-text alignment for visual recognition. arXiv preprint arXiv:2204. 10760, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In EMNLP: System Demonstrations, 2020.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.121445 2023.
Mingjing Xu, Peizhong Ju. Jia Liu, and Haibo Yang. Psmgd: Periodic stochastic multi multi-gradient descent for fast multi-objective optimization. arXiv preprint 11X1v:2422.10611, 2024a.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint wx.iv:2402.13112 2024b.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruming for efficient deep neural networks. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing Computing-NeurlPS NeurlPS Edition (EMC2-NIPS), pp. 14-17. IEEF, 2019.
Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In Proceedings of the JERE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15952-15962, 2024.
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen.
Learning low-rank deep neural networks via singular vector orthogonality regularization and sin- gular value sparsification. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pp. 678 778, 2020.
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprint A.V.:2402.12215: 2024.
15

---

Published as a conference paper at ICLR 2026
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Dacao Zhang, Kun Zhang. Shimao Chu, Le Wu, Xin Li, and Si Wei. More: A mixture of low-rank experts for adaptive multi-task learning. arXiv preprint arXiv:2505.22694 2025.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3713-3722, 2019.
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Jialu Liu, Michael Bendersky, Marc Najork, and Chao Zhang. Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation. arXiv preprint w.XXvv2305.00011 2023a.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint Xiv:2230. 10792, 2023b.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine 2015.
intelligence,
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp- 11953-11962, 2022.
Qihuang Zhong, Liang Ding, Shen, Juhua Liu, Bo Du, and Dacheng Tao. Revisiting knowledge distillation for autoregressive language models. arXiv preprint arXiv:2402.11890, 2024.
Baohang Zhou, Ying Zhang, Yu Zhao, Xuhui Sui, and Xiaojie Yuan. Multimodal graph-based varia- tional mixture of experts network for sero-shot multimodal information extraction. In Proceedings of the ACM on Web Conference 2025, pp. 4823-4831, 2025a.
Guorui Zhou, Ying Fan, Rumpeng Cui, Weijie Bian, Xiaoqiang Zhu, and Kun Gai. Rocket launching: A universal and efficient framework for training well perfoonng light net, 2018. URL https: //arxiv. haargg//bb/1/// 1708. 04106
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Yiyun Zhou, Wenkang Han, and Jingyuan Chen. Revisiting applicable and comprehensive knowl- edge tracing in large-scale data. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pp. 235-251. Springer, 2025b.
Yiyun Zhou, Zheqi Lv, Shengyu Zhang, and Jingyuan Chen. Cuff-kt: Tackling learners' real-time learning pattern adjustment via tuning-free knowledge state guided model updating. In Proceed- ings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 4144 4155, 2025c.
Yiyun Zhou, Zheqi Lv, Shengyu Zhang, and Jingyuan Chen. Disentangled knowledge tracing for alleviating cognitive bias. Proceedings of the ACM on Web Conference 2025, pp. 2633-2645, 2025d.
Yiyun Zhou, Mingjing Xu, Jingwei Shi, Quanjiang Li, and Jingyuan Chen. Collaborative rep- resentation learning for alignment of tactile, language, and vision modalities. arXiv preprint arXiv:2511.11512 2025e.
Yiyun Zhou, Chang Yao, and Jingyuan Chen. Cola: Collaborative low-rank adaptation. arXiv preprint arXiv:2505. 2025f.
16

---

Published as a conference paper at ICLR 2026
7 APPENDIX
We present the following additional information to include and clarify details that could not be fully covered the main text.
Content
Fixed Asymmetric Design
Empirical Evidence
Theoretical Support
Theoretical Analysis Supplement
B.1
Algorithm Overview (Detailed Procedure)
Convergence Guarantees
Parameter Efficiency
Experimental Details
Implementation Details
Results on ImageNet
Additional Experiments on Large Transformer Language Models
Experimental Details Insights on InherNet
Related Work
D.1 Knowledge Distillation
D.2 Low-Rank Decomposition
The Use of Large Language Models
A FIXED ASYMMETRIC DESIGN
A. EMPIRICAL EVIDENCE
We attempt invert InherNet into InherNet (i.e., multiple single-dimensionaliyy reduction matri- ces followed by a dimensionality expansion matrix). We compare the performance of InherNer and InherNet across various tasks, shown in Tables InherNet consistently underperforms compared to InherNet across all tasks. The performance gap especially pronounced in text-related tasks, which may be attributed the fact that, such tasks, the linear layers adapted for inheritance more effectively preserve and replicate the knowledge from the teacher network. contrast, convolutional neural networks, due to their higher dimensionality and the necessity of reshaping operations, may suffer greater information loss. We plan to explore further optimization strategies for inheriting convolutional neural networks as a direction for future work.
and 7 From the results, we observe that
Table 5: Comparison top-1 Accuracy (%) on the CIFAR-100 validation set between InherNer and InherNet.
ResNet32x4 79.42
VGG13 74.64
WRN-40-2 75.61
WRN-40-2 75.61
ResNer56 72,34
ResNet110 74.31
ResNet110 74.31
Teacher
Method
ResNet8 72.50
VGGS
WRN-40-1
WRN-16 16-2 73.26
ResNet20
ResNet32 71.14
ResNet20
Student
70.36
71.98
69.06
69.06
Small
75.19
75.84
74.17
76.53
71.39
71.68
71.68
O
Large
78.79
71.26
75.24
76.17
73.38
74.95
74.95
Small
77.57
75.68
76.04
76.04
72.67
74.13
74.13
InherNet
Large
78.53
75.16
76.39
76.39
73.67
75.88
75.88
17

---

Published as a conference paper at ICLR 2026
Table
The performance (%) of InherNet and InherNet on GLUE tasks.
Method
MRPC
QQP
SST-2
MNLI
RTE
QNIJ
COLA
STS-B
TS-Buse
89.21
86.54
90.71
54,93
77.98
90.13
$7.00
89.04 / 88.88
InherNet
87.36
83.50
87.69
56.27
72.86
89.74
50.07
86.55 / 85.73
InherNet
88.74
84.69
89.07
56.34
72.93
90.29
50.63
86.70 / 86.36
Table 7: Comparison of cross-modal retrieval on the CC3M validation set and zero-shot classifica- tion on ImageNet validation set performance (%) between InherNet and InherNet.
12T
T21
Classification
Method
R@1 1
R@5
R@ 10
R@1
R@S
R@ 10
lop-1
lop-5
ResNet-101
30.58
53.95
63.49
29.31
54.13
63.31
15.70
32.75
InherNet
28.76
57.46
64.17
30.39
59.25
65.81
17.24
36.02
InherNet
32.17
57.17
67.82
33.01
59.88
68.72
17.39
36.65
A.2 THEORETICAL SUPPORT
pretrained teacher network's weight W E mR can be optimally approximated as W R E,E,V," derived from the SVD, where r < min(m, nt) A critical design choice arises when integrating this low-rank factorization with MoE principles to handle heterogeneous data distribu- tions. We analyze two possible asymmetric architectures:
One-Down-Many-Ups A single, shared dimension-reduuingg projection Widowwn feeds into multi- ple specialized dimension-increasing expert heads (W h==1 as shown in Figure
Many- Many-Downs-One-l Multiple specialized projections [Wdown)s ! Jhh1 are aggregated and passed
through a single shared reconstruction layer W'up
We found that the first configuration is theoretically superior, and we ground our analysis in the Information Bottleneck (IB) principle (Koch & Ullman 1987 Alemi al. 2016 et
The IB principle frames learning as a problem of compression. Given an input X and a target output Y, the goal is to learn compressed representation, or bottleneck, Z. that is maximally informative about Y while being minimally informative about X. This trade-off is formalized by minimizing the Lagrangian:
Lu (p(((=)) I(X;
YT)
(15)
where -::-) denotes mutual information and ,3 is a Lagrange multiplier controlling the trade-off between compression ((;;; Z)) and prediction (Z The optimal representation Z is thus minimal sufficient statistic of X for predicting Y.
A.2.1 INFORMATION- THEORETIC ANALYSIS OF ASYMMETRIC ARCHITECTURES
We now apply the IB framework to analyze the two competing architectures. Let the input features be X and the final output be Y.
The One-Down-N Man-Doww-Many Configuration as a Unified Bottleneck This architecture, as defined in InherNet, is given by:
H Y Σ Ga(X) WhP (wdoww (X)) h=1
(16)
Let us define the bottleneck variable as the output of the shared projection: Z é ()down(X), The IB objective for this architecture is learn the parameters of Wrowwn (which define the mapping p(=[zr)
that minimize Li from Eq. 15
Theoretical Analysis. The one-down-man many mas architecture imposes single shared bottleneck: all information from X to Y must pass through the same diw-diennsional Z. This means Z must serve as the representation for the entire mixture of H experts. Consequently, to maximize the predictive information 7(Z; Y), Z is forced encode only those features of X that salient for
18

---

Published as a conference paper at ICLR 2026
predicting Y across all expert domains, while filtering out any spurious task-specific details (Ye et al. 2024 Zhou et al.) [025e]d Lv et al. 2025| Zhou et al. (025a).. In effect, the model lears to focus on the joint task distribution p(ylar )= = Ei=1 p(h[z) plyl h), where p(h[z) is given by the gating function G (X). Since the architecture essentially forms Markov chain X Z - Y (Norris) 1998). the Data Processing Inequality guarantees that / [2; Y) S Y); The IB objective drives Z to approach this upper bound on predictive power while minimizing ((;;
yielding minimal sufficient representation of X.
This unified bottleneck encourages Z become a robust, canonical representation of shared knowl- edge that all experts can rely on. Each expert head W then performs a simpler, decoupled task: mapping the common r-dimensional representation Z specialized output subspace in IR". No- tably, although Z has dimension r. the seee-down-man--eee mapping. Because each W., can independently span an output subspace of dimension up to r, the combined model can cover the union of these subspaces. Through gating, different inputs activate different experts, allowing the overall output to lie in a rich space of dimension up to r rH, far greater than any single low-rank projector could achieve. The architecture dedicates most of its pa- rameters to the expert-specifii "up" transformations, thereby boosting expressiveness where needed while keeping the shared encoder compact.
design is not limited to a single rank-r
Another major advantage of the unified bottleneck is in gradient flow during training. The er- gradients from all expert outputs propagated back into the single projection W This shared pathway means that every expert's feedback refines the same encoder W'dowa, aligning the experts on learning common representation. Such unified credit assignment avoids conflicting up- laproe-impooveeentt for one expert cannot come at the expense of degrading the shared features for another (Samejima al. 2003 Mu & Lin 2025). This synergy makes optimization more stable and efficient. Besides, our use of SV based initialization provides an informative starting subspace for Z, aligned with the directions of maximal variance and information in the teacher. This initialization further accelerates convergence and helps Z capture the most relevant features of X in Sec. S.c.
The Many-Downs-One-Up Configuration as Competing Bottlenecks The inverse architecture formulated as:
Y = Wup Σ Gx( Widow (X))
(17)
Here, have H distinct bottleneck variables, Zn Wh Wddwa(X), which are aggregated to form an intermediate representation, Zage - DH h= (X) Z,, before the final reconstruction.
Theoretical Analysis. The many-downs-one-up configuration is fundamentally limited from an informatioo-theoreet perspective. According the Data Processing Inequality, for any Markov chain A + B + C (Norris) 1998), we have [(A; C) S ((A; In our case, the processing chain of interest is:
(Z) Zn) Zuge +Y
(18)
Therefore, I((ZI.... Zz):Y) v 2 I(Zugg: Y). The aggregation = ) GX(X) Zn is lossy operation that conflates information from the experts. It mixes specialized features from different expert channels into a single vector before reconstruction, making impossible for the shared Wuup layer to discern the contribution of each expert. Moreover, this architecture imposes a strict capacity bottleneck: all expert outputs must funnel through the same rank-r reconstruction matrix W As WuP can only produce outputs in an r-dimensional subspace of TR. the final prediction Y is confined to a fixed low-dimensional subspace regardless how diverse the individual Zn, are. In contrast to the one-down- many-mannnyaaaa uups design, here the presence of H distinct projections Wrdwwn does not increase the overall representational vank-any benefit of having multiple experts is squandered by the single shared projecjec..on.
Training this architecture also poses a severe credit assignment problem (Tedus et al. 2022 Zhang al. 2025). The single reconstruction layer Wu must learn to invert a complex mixture Zags back to Y without access to the individual expert outputs Zr. During backpropagation, the gradient of the ac dL (X) 8%
loss with respect to 7agg is shared by all experts. By the chain rule one can show 3%,
for each expert h. Thus, each Woown only receives fraction (X) of the total feedback signal,
19

---

Published as a conference paper at ICLR 2026
weighted by its gating activation. When multiple experts active, their parameter updates become entangled and often conflicting, since the model cannot clearly know which expert was responsible for which aspect of the prediction error. This ambiguity in credit assignment leads to unstable op- timization and hampers the learning of effective specialized representations. In practice, the easiest path for the model is collapse the experts into effectively single expert to eliminate the ambi- guity. For example, it drives all Zn to become nearly identical or favor one expert exclusively, so that Wup deals with a single dominant bottleneck. Such outcomes defeat the purpose of having Il experts in the first place, confirming the fundamental difficulty of the sany-downs-one-pg design.
A.2.2 CONCLUSION OF ARCHITECTURAL DESIGN
The "one-down-mane successful but also theoretically sound when analyzed through the lens of the Information Bottle- neck principle. It creates unified optimization objective that promotes the learning of a minimal, sufficient, and shared representation. In contrast, the imany-downs-one-ut structure introduces in- formation loss through premature aggregation and creates a challenging credit assignment problem, hindering effective specialization. Therefore, conclude that enforcing single, shared informa- tion bottleneck before expert-level specialization is a superior design principle for building InherNet.
architectural design, as employed in Figure 2 not just empirically
B THEORETICAL ANALYSIS SUPPLEMENT
B.1
ALGORITHM OVERVIEW (DETAILED PROCEDURE)
Algorithm [ details the complete procedure for implementing InherNet, our SVD-Driven Neural Network Inheritance method. The algorithm begins by performing truncated SVD on a pretrained teacher weight matrix, extracting the most significant singular components to form a low-rank ap- proximation. This factorization then used to initialize two sets of projection layers: the backbone branch, responsible for lowering the dimensionality, and multiple expert heads, which are designed an asymmetric and MoE style to reconstruct and enrich the output. An adaptive gating mech- anism dynamically aggregates the outputs from the expert heads, ensuring the inheritance of the teacher's s structural and semantic information. This procedure enables efficient knowledge transfer, faster convergence, and improved parameter efficiency in training.
Algorithm 1 InherNet: SVD-Driven Neural Network Inheritance
Require:
Pretrained teacher weight W E m=x,, rank r, number of heads H, input Ensure: Output Y
1: Perform truncated SVD: WA & U,E.VTT
2: Set backbone branch: Wdown (.) 4- U,21!
3: for h = 1,2,.... H do
4: Set /--th head: (-P(-)) + 3 - /TV
5: end for
6: Compute backbone branch output: X, = Wdoww (X).
7: for = 1,2,.... H do
8: Compute expert head outputs: (1a) W'''((,)..
9: end for
10: Compute gating weights: G(X) softmax (W3(X)
Aggregate outputs dynamically: Σ G₁ (X). (h))
12: return
B.2 CONVERGENCE GUARANTEES
We now provide a detailed convergence analysis for InherNet, highlighting how its low-rank factor- ization, multi-head gating, and SVD-based initialization affect optimization under SGD. Despite the structural compression, we will show that InherNet achieves convergence rates on par with standard neural networks under mild conditions. We begin by expanding on the assumptions used in Sec. 2.2 then derive how InherNet's s architecture influences gradient descent, and finally present convergence guarantees alongside the impact of the adaptive gating mechanism.
20

---

Published as a conference paper at ICLR 2026
B.2.1 GRADIENT DYNAMICS UNDER INHERNET PARAMETERIZATION
Recall that an InherNet module low-rank, multi-head network with an input-dependent gating mechanism. Concretely, the module's output can be written as a weighted sum of H "heads":
fo(x) - E (n(); WT (W (x)),
(19)
h=1
where Widown is rank-r projection applied to input each Wh is the h-th head's weight matrix (mapping the r-dimensional projected input to the output space of dimension d'), and Gn(a; (g) is the gating weight for head h (with gating network parameters 0g). By construction, E (r;
for any æ. Let 0n denote the parameters of the h-th head and fl the parame- ters of the gating mechanism, so that 0 = (0i...., ,n 0g) represents all model parameters.
Lemma B.1 (Gradient Decomposition). Under the InherNet parumeterization. the gradient of the loss splits naturally into contributions from each head and from the gating network. In particular, for a single data point (z, y) with loss ((f((x)))) y), the total gradient satisfies:
11
Vol[o(())
Σ
Gala; 0g) Vo,
b=1
(20)
Σ
Vo Gi(x;)
where Vo,&& E is the gradient with respect the parameters of head h, and D.. 00(fo(m)))) is the Ffulz)
scalar "error signal" for head (the partial derivative of the loss with respect to the output of head h, where fa(x)) = Wht (Wodww (Wdown(x)))
Proof. This follows directly from the chain rule applied to the composite model fe(x) = E Gx( (r)), where (x:) Wit (W'dowaz))) The derivative Vol breaks into two parts: one through each head's parameters 0, (weighted by Gj),, and one through the gating parameters 8g (accumulating the head output sensitivities A,). This two-term decomposition holds for any Inher- Net module (whether the low-rank module is linear or convolutional), since all such modules share the same gated low-rank structure.
L
This lemma illustrates how the adaptive gating and multi-head structure direct the gradient updates. In particular, even under a low-rank constraint, the presence of the gating network preserves a rich set of descent directions during training, thereby maintaining the model's expressivity in the param- eter space. InherNet can still update each head's parameters and the gating weights independently, ensuring that compression does not eliminate useful gradient directions. This adaptive gradient rout- ing allows the model to specialize the expert heads during training by directing larger gradient flows to the most relevant heads for each input pattern.
B.2.2 EFFECT OF INITIALIZATION ON CONVERGENCE
We next examine the role of SVD-based initialization the optimization landscape. Let W E mm*x denote a pre-trained weight matrix that an InherNet module aims to compress. InherNet uses truncated SVD of W to initialize its low-rank factors. Specifically, suppose W x U, E, V, is the rank-r approximation of W, where U, E mmrr and V, g= have orthonormal columns (containing the top left and right singular vectors), and E, R is the diagonal matrix of the top-r singular values. Then the InherNet module is initialized as: Wrdown = U, x//2 and W = H //VV for h 1, ...., . H, so that W'dwa +... Woup W initialization. Each head thus starts with an equal share of the teacher's weight (1/H of W), and the factor matrices Ur. V, orthonormal in their columns.
Proposition B.2 (Orthogonality Induced Stability). Under Assumption 1. an orthonormal SVD- based initialization leads to a better conditione optimization landscape. In particular, if the factor matrices U, and V, are initialized with orthonormal columns as above, then the effective gradient
21

---

Published as a conference paper at ICLR 2026
Lipschitz constant of [(0) is reduced from L to L' L/K, where K is the condition number of W (the ratio of its largest singular value to smallest singular value).
Proof. When the low-rank factors are initialized orthogonally, the curvature of the loss with re- spect the factorized parameters is significantly more balanced. In fact, the Hessian of L with respect to the factorized parameters (W/down, WT ..... WH) tends to have a block-diagonal diagonal struc- ture with reduced inter-parameter coupling. This arises because the initial parameter space can be viewed as lying on a product of Stiefel manifolds (due to the orthonormal columns). As result, the largest eigenvalue of the Hessian the effective Lipschitz constant governing the gradient's 's rate of Shange-drope to roughly L/K. In other words, the initial orthogonal factorization "pre-conditions" the problem by attenuating the problematic directions high curvature in the weight space. This im- proved conditioning aligns with prior theoretical results: orthogonal initialization has been shown to accelerate convergence in deep linear networks and to stabilize training in wwerpparameterized models.
Remark B.I. This result explains why the truncated SVD initialization confers optimization stability, especially in the early stages of training when the factorized weights remain near-orthogonal. By effectively reducing the Lipschitz constant, the initial gradients face gentler landscape, which can lead to faster progress per iteration. Our use of an orthogonal low-rank initialization is thus not only a parameter-efficien choice but also an optimization-friendee one, improving the curvature (conditioning) of the problem during the critical beginning of training.
B.2.3 CONVERGENCE RATE ANALYSIS
With the above insights, we now analyze the convergence rate of SGD on InherNet. We consider optimizing the expected loss L()) using stochastic gradient descent. Let g(9) denote the model parameters at iteration f, and (z19, (()) be the random sample drawn at iteration t. The SGD update is:
g(++1) = 800 T7 Vollfoco (z!0), y/)).
(21)
where >> 0 is the learning rate at iteration t. We assume a diminishing step size schedule (1) that satisfies the standard conditions for convergence: Σ 1/- = oo (to ensure we continue making progress) and E ni oo (to control the noise over many iterations).
Theorem B.3 (Convergence of SGD for InherNet). Under Assumptions 1-3, suppose we run SGD on (()) with a diminishing learning rate satisfying the above conditions. Then for any T 2 1, the iterates (0(4) yT satisfy the guarantee:
2 -[((
Lo E ΣT
E [(
min ISSS
S
(22)
Σt=1
Here L- info C(0) is the optimal (minimam) loss value. In particular, with typical choice of me
(e.g. Th. x 1/Vt), this bound implies that
min ISST
PIIVC(0 =((LVVT,
(23)
matching the standard convergence rate for nonconvex SGD.
Proof. The proof relies on the smoothness and bounded variance assumptions. By L-smoothness of (Assumption 1), a one-step descent lemma holds: for any iteration
((0(c+1)) < L(O(t)) ne Tn(1- (0-1--4 Lmn
|oC((()))
(24)
L Ln + [[ 11
where (0)) :- Vol(Sore (z(()), (()) is the stochastic gradient at step t. This inequality follows from standard smoothness bound (the RHS is obtained by applying the gradient update and Young's inequality the Taylor expansion of L around ((1)). Now taking expectation both sides with
22

---

Published as a conference paper at ICLR 2026
respect the randomness step and using Assumption 2, which ensures 1 |l o² (and Big(') = VL((())), we get:
VC(A
Lmfe [[((0((+))]] E[c(O())] nn(1
[oC()()))
2
(25)
L o². 2
Rearranging and summing this inequality over 1 to T yields:
T
E(1
Lm
E voc000))
s E - ((((+11)
2
(26)
+ L
> m
Since L. < ((A(++1)) (the loss cannot go below the global infimum), we have E Ei((O(1)
[(O(T+1))] < ((((1)) - Also note the identity Σ 7t(1 (1- -11-- Σ The let .T = Σi= 71 (1 - Lxt),, The left-hand side above is at least Pr min El| (C((())
E n/. Now,
since each term in the sum contains the non-negative factor (1 Lm: Therefore,
P
min ISST
[[| s L(B())
l- + Lo²
Σ
n
(27)
2
Finally, using Fr 2 Σ m/ - E 1/ and dividing both sides by Fr, obtain the stated result.
This theorem establishes that, despite its structured low-rank parameterization, InherNet network trained with SGD enjoys the same asymptotic convergence guarantees (convergence to stationary points of L) standard, full-rank neural network under the given assumptions. In other words, introducing the InherNet architecture does not degrade the theoretical convergence rate of SGD. For example, with a learning-rate schedule 7/- = n/ /t, the bound above implies
1 T
B - O(L/TT),
(28)
matching the classic non-convex SGD rate. This rate achieved with potentially better constants: due to the improved conditioning from Sec. 2.2 the effective Lipschitz constant is L'' < T./K + OE)) se L/K, where к is the condition number of W and represents higher-order terms that diminish as training progresses. Indeed, in the early phase of training, one expect speedup by a factor of x in convergence (since the gradient norm can decrease faster with smaller L'), consistent with our empirical findings (see Sec. 3))
Remark B.2 (Convergence Different Optimization Regimes). Our analysis primarily addresses non-convex optimization scenarios, these are most relevant for training neural networks. Nonethe- less, the convergence guarantees provided by InherNet naturally extend to other standard optimiza- tion scenarios:
For general convex objectives, InherNet achieves the standard SGD rate of O(1/T), but with improved constants due to orthonormal initialization.
In p-strongly convex settings. InherNet achieves an exponential convergence rate of O(e- for a learning rate y), with significant improvements arising again from orthonormal parameter initialization.
B.2.4 ADAPTIVE GATING'S S IMPACT ON CONVERGENCE
Finally, we consider the role of the gating network in SGD's convergence behavior. Thus far, our analysis has treated the gating weights (x 0g) as given. In practice, these gating weights
23

---

Published as a conference paper at ICLR 2026
during training, the gating network learns to emphasize the most useful heads for each input. We argue that this adaptivity can further improve convergence by reducing the variance of the stochastic gradients.
adaptive:
Proposition B.4 (Variance Reduction via Adaptive Gating). Under Assumption the adaptive gat- ing mechanism In particular, if G., (g) tends to assign higher weight to heads that contribute more to reducing signal- to-noise ratio), then the variance of the stochastic gradient under this adaptive weighting is strictly lower than under uniform weighting Gn, 1 In the best case, adaptive gating can improve the convergence stability (variance reduction) by up to factor of H relative to uniform gating.
reduce the variance of gradient updates compared to static uniform gating.
the loss (i.e. the gating weights are positively correlated with each head's "importance"
Proof. Consider the per-sample stochastic gradient g [o y). This can be viewed as the
weighted sum of head-specific gradient components: g Σm= GA Gh. where 9h == Voln denotes h==1
the gradient contribution from head (and drop (2; (,)) for brevity in G₁,). Because 2n GA. 1,
the variance of y (conditioned on given model state 0) satisfies
"
Varigl = Var
Gh 9h
(29)
A=1
By Jensen's 's inequality, for any random variables Zn we have Var(En wnZn) S En wh Var( when 2, Wh 1. In our case, the gating weights Ga, serve as the convex weights. Intuitively, Jensen's 's inequality implies that the variance of a weighted average is minimized when we put as much weight as possible on the lowest-variance components. The adaptive gating chooses G2 (x) in data-dependent way that tends to up-weight heads with more reliable, higher signal (or lower variance) gradients, while down-weighting the less reliable ones. In contrast, a uniform gating (GA, 1/H for all h and all æ) cannot adapt to per-head reliability and thus often incurs higher overall variance by averaging in unnecessary noise from less important heads. In extreme cases, one head provides significantly lower- variance (or more informative) gradient than others, focusing the weight on that head (adaptive gating) versus evenly spreading weight can reduce the gradient variance by factor approaching H. Thus, the adaptive gating strategy yields strictly lower variance in the stochastic gradient, which in turn leads to more stable SGD updates and potentially faster convergence in practice.
To conclude, our theoretical analysis demonstrates that InherNet retains strong con- vergence guarantees despite its compressed, structured design. In addition, enjoys two notable benefits compared to a standard network:
Summary:
Enhanced optimization stability due to the orthogonal low-rank initialization, which improves the problem's 's conditioning (Sec. 2.2).
Reduced gradient variance during training thanks to the adaptive gating mechanism, which focuses learning on the most relevant components (Sec.
Together, these properties ensure that InherNet can be trained as efficiently and robustly as an un- compressed model (Li et al.] [0255] Jiang et al.) 2025a b), while leveraging its architecture to accel- erate and stabilize the optimization process.
B.3 PARAMETER EFFICIENCY
Here we provide an expanded theoretical analysis of the parameter efficiency of InherNet, offering complete proofs and additional insights that complement the main text.
B.3.1 PARAMETER EFFICIENCY METRICS AND DEFINITIONS
The definitions presented in the main text elaborated here for completeness. Definition B.1 (Parameter Efficiency). For a neural network fa parameterized by 0 E RP that approximates target function f*, the parameter efficiency (PE) of fe is defined as
Representational Capacity(.fe)
PE(f))
[0]
24

---

Published as a conference paper at ICLR 2026
where Representational Capacity measures the network's ability approximate functions in given function class, and |0] denotes the total number of parameters. our experiments, we use evaluation performance proxy for representational capacity and record the parameter count during training. Definition B.2 (Approximation Error). Given a target function f from a function class F and a neural network fe, the approximation error is defined
E(fo, EE-D [II fo(x) (
where the data distribution.
Definition B.3 Frxxprssivitt-tooParaaaa Ratio (EPR)). The EPR of network so with respect to a function class is defined as
[P.
|0] 8
i.e. the inverse of the minimum achievable approximation error (over functions in F) per parameter. higher EPR indicates more expressivity (lower error) per parameter.
B.3.2 COMPRESSION BOUNDS FOR INHERNET
Theorem B.5 (Parameter Reduction Bounds). Consider layer W E RRxxn, The InherNet param- elerization with rank r and heads yields a compression ratio:
171
m.
(+0(-
H Hm(m++ (m)+)) + (r++1) H (m mn
Hr
This corresponds p-fold increase in Parameter Efficiency (Definition B.1 assuming the layer's representational capacity is largely preserved.
Proof. A layer W mmx has free parameters. Under the InherNet expert-head parameteri- zation with rank r and H heads:
Each head h contains "down" matrix Whloon and an "up" matrix W.. contributing in total H [rm + ar] = Hr (m + +n) trainable parameters.
The gating network maps the r-dimensional codes into И weights via
W E R Hxr (parameters H r) and bias E ER"
adding another Hr + = H (r++ 1) parameters.
Hence the total adapter-specific parameters are
H (m + n) + H (r 1),
and the compression ratio relative to the original m n is
(1+0(mZ) (
H r (m + n)
(r+1))
Hr (m +n)
mn
H Hr (n)
By Definition B.I assuming this rank-H1 subspace preserves the layer's expressivity, InherNet thus achieves an fold gain in parameter efficiency.
Lemma B.6 (Spectral Energy Preservation). Let W E RRmxn have singular values a
2
02
... 2 nij If the rank r is selected such that:
El o?
21-c
(30)
E
miin(m.)) o²
then by the Eckart- Young-Mo Mirsky theorem in Sec. approximation W,, bounded by:
the Frobenius-norm error of the optimal rank-r
IW W. 1 < W
(31)
25

---

Published as a conference paper at ICLR 2026
Consequently, the Approximation Error (Definition B.2 between InherNett-oompressed layer and the original layer is bounded by e under the squared Euclidean norm.
theorem, truncating the SVD of W to its top singular values yields the optimal rank-r approximation in Frobenius norm. The squared error of this approximation
Proof. By the Morkart-Youg-Miisse
is Σ 2ir+11
(m..)) of. Under the given condition, the relative error is
mi.. 11++1
of
E
o2
S c.
Emin(m.n) (m) o?
Ei=1
m)) d;
Thus I|W - W,1. S [|
neural network, this matrix approximation enror directly trans- lates to bounded increase in the network's 's overall error.
Proposition B.7 (Knowledge Preservation Rate), Let fw be a teacher network with weight matrices [Wi} and fr be the corresponding InherNet with rank r. The functional similarity between fw and fo is
by:
E.
of
Sim(fw, ft). f+)
Σ 1=1
1
(32)
D
of.
where represents the layer's influence on network output, and ø1, are the singular values of W.. This bound monotonically increasing with r but not with H, establishing that rank the primary determinant of InherNet knowledge preservation.
Proof. For a single layer I, the error introduced by low-rank approximation is bounded by ei
Ei= of
miinemy of as established in Lemma B.6 The total loss in functional similarity can be expressed Dis
of these layer-wise errors, where weights aj reflect each layer's contribution to the network's output. The gating mechanism in multi-head designs can at best maintain this bound by selecting the optimal head for each input, but cannot improve the fundamental spectral approximation determined by rank r. Therefore, similarity improves monotonically with rank but not necessarily with the number of heads.
weighted
B.3.3 REPRESENTATIONAL POWER PRESERVATION
Theorem B.8 (Representational Power Preservation). Let Fw be functions representable by stan- dard network with weights (Wif= For any f E Fw and 6 0, there exists an InherNet network with appropriate ranks {ri) and head counts {Hi} such that the Approximation Error (Definition B.2) satisfies:
E(StnherNets f 8
(33)
while the parameter count is reduced by Mmnnn; min(mamil) relative to the standard network. Con- Hi
sequently, the InherNet architecture atlains an EPR (Definition) B.3) higher than that the standard architecture by at least the same factor.
Proof. We construct finherNet two stages:
Layer-wise low-rank approximation. For each layer with weight matrix Wi, by Lemma B.6 can choose a rank ri such that |[;- Will < a [|W] where Wi is the rank-7; approximation. Each e 0 can be made arbitrarily small by increasing ri.
Multi-head specialization. Using H, > heads for layer further reduces the approximation error, since each head can specialize on subset of the input distribution via the gating mech- anism. For an input æ, let h (z) be the index of the head that the gating network selects. The effective weight applied to is then (i=(h((ar) Because inputs are partitioned among H, heads, the approximation error per head is reduced. Formally, one can ensure
Wi (a)) (m))
will < Hi [Will
(34)
since the gating network routes each x to the head whose low-rank weight best approximates W; that input's region.
26

---

Published as a conference paper at ICLR 2026
Applying these two steps across all layers and choosing {ai such that the errors compose to at most 6, we guarantee E (fiiheeNeet fr) I 8. The parameter reduction factor follows from Theorem B.5 and the EPR improvement follows directly from Definition B.3
Corollary B,9 (Universal Approximation). InherNet retains the universal functtin approximation property of standard neural networks while using asymptotically fewer parameters. For any con- timuous target function, a sufficiently large InherNet network can approximate it arbitrarily well (to Approximation Error standard network of comparable accuracy.
0) with significantly higher Parameter Efficiency and EPR than
Proposition B.10 (Diminishing Returns of Multiple Heads). For a given layer with rank the marginal reduction in approximation error from adding the (H + 11)thh head satisfies:
C E(r, H) Er, H H++1+ H2 E(r,1)
(35)
where e a constant depending on input distribution characteristics, and (rr H) denotes the ap- proximation error with rank r and H heads. This establishes that benefits from additional heads diminish rate of )
Proof. The gating mechanism in InherNet effectively partitions the input space among H heads. When adding an (H + 1)-th head, the new head can at best specialize on a subset of inputs that previously sub-optimally handled. Under reasonable assumptions about input distribution and the effectiveness of gating. the probability of an input being significantly better handled by the new head decreases quadratically with H. Specifically, if model the input space as being divided into regions, with each head specializing can only improve regions where the existing heads perform poorly. The volume of such regions decreases proportionally to T leading to the stated bound.
approximately y of the space, then the (H + 1)-th head
B.4 ADDITIONAL ANALYSIS: INTERACTION BETWEEN INHERNET AND KD
To better understand how KD interacts with InherNet, we conduct an set of experiments on CIFAR-100 using ResNet-56 as the teacher. We construct InherNet models with ranks r E (8, 12, 64, 128}, and train each model both with and without KD. The detailed results are shown Appendix C.4as as Insight (Figure
For aggressively compressed models (small ranks, e.g., S. 32), adding KD improves performance. For example, at r = 32, the top-1 accuracy increases from 74.4% (InherNet only) to 75.6% (Inher- Net + KD). In this regime, the InherNet model has limited capacity, and the teacher's logits provide a strong inductive bias that compensates for the information lost in the truncated spectrum. For larger ranks (e.g.. 2 64), the behavior changes. At 64, the KD term brings additional in- crease but decrease, and at r = 128 it also actually degrades performance: the accuracy drops from 75.8% without KD 74.1% with KD.. In other words, the InherNet becomes more expressive, KD transitions from being beneficial to being detrimental.
This phenomenon is closely aligned with our theoretical analysis. Proposition 2.7 shows that, as the rank r increases, the InherNet fr converges in function space toward the teacher fw. and the approximation error is controlled by the tail of the singular-value spectrum. For sufficiently large the InherNet model already provides good or even better performance than the teacher, and the task loss alone can drive solutions that generalize better than the original teacher.
When we add KD, we optimize
Liotal = Lesk (fr(z).)) v) + CKD (fv())()).fivxz))
(36)
where Lusk = Ace (cB(ff(x).)) encourages correct predictions, and
LKD = ARD 7² KL (PW) (7)
pf.T)
(37)
encourages the InherNet model to align its logits with those of the teacher, with temperature
and softened distributions (T) and p. For small r. this additional constraint is helpful: the teacher acts as strong prior that guides the low-capacity InherNet toward better solution. For large
r. however, the KD term acts as an overly strong regularizer, penalizing any deviation from the teacher's logits even when such deviations would improve the model performance. This explains
27

---

Published as a conference paper at ICLR 2026
why the "InherNet + KD" curve eventually falls below the "InherNet only" curve r grows in Figure
From a practical guideline, this analysis yields simple suggestion for using KD with InherNet:
Aggressive compression (small rank. Use InherNet with KD. The InherNet provides a good low-rank subspace, while the KD loss helps recover discarded information.
High-fidelity inheritance (large rank). Prefer InherNet-only training. The InherNet model al- ready approximates or outperforms the teacher, and forcing it to stay close to the teacher logits can restrict its ability to find solutions that generalize better than the teacher.
B.5 ON LAYER-WISE VERSUS NETWORK-LEVEI COMPRESSION
This subsection provides additional discussion on whether the layer-wise SVD step in InherNet could lead to suboptimal compressed models.
In InherNet. the decomposition step is applied per layer: for each teacher weight matrix W, perform a truncated SVD and construct low-rank expert module as initialization. This is the only place where SVD is used in a layer-wise way. After this initialization, all inherited layers are trained jointly, end-to-end, under the same task loss (and, when used, the KD loss). No layer is frozen after compression. Consequently, inter-layer correlations stage; instead, they are re-optimized globally during training, and the final solution is determined by network-level optimization dynamics rather than by the initial SVD alone.
not fixed at the decomposition
Network-level guarantees despite layer-wise SVD. Although SVD is performed independently each layer, our theoretical analysis is carried out the network level.
(i) Layer-wise spectral control. Lemma 2.6 states that if the rank of layer is chosen then the optimal rank-r approximation W, satisfies TW W, I e||W In other words, the layer-wise approximation error can be made arbitrarily small by increasing and the truncated SVD gives the best possible rank-r approximation in Frobenius norm.
(ii) Knowledge preservation for the whole network. Proposition 2.7 lggregates these layer-wise er- rors into a functional similarity bound between the teacher fw and the inherited network fr where Or measures the influence of layer 1 on the final output. This bound shows that, although the ap- proximation is performed locally at the level of individual layers, its effect on the network function is controlled globally via a weighted sum over layers: layers that are more influential (larger oz) are allowed smaller spectral truncation error, while less influential layers can tolerate more compression.
(iii) Representational power under compression. Theorem 2.8 further shows that the rep- resentational power of the original network is preserved under compression. Formally, shows that for any function f E Fw representable by the original network, V5 > 0, (ri, Hi} such that E [[inherNet (*) ("") S 8, while the total parameter count is reduced by 2( min, min(my.me)). The constructive proof proceeds in two stages: (1) layer- low-rank ap- r, proximation, and (2) multi-head specialization. Importantly, chooses {ni} and { so that the composition of all layer-wise errors yields a network-level approximation error at most 6. Thus, the interaction between layers explicitly accounted for through depth, and layer wise SVD does not inherently force the compressed network into suboptimal representational regime.
Taken together, these results imply that, as long as each ri is chosen to keep the spectral tail small, there exists a compressed InherNet that approximates the original network arbitrarily well while using fewer parameters. So, the layer-wise nature of the SVD step is compatible with strong global approximation guarantees.
Empirical evidence against systematic suboptimality. layer-wise SVD led the optimization to consistently suboptimal local minima, one would expect the compressed models to be always worse than the teacher across tasks. Our experiments do not support this. At the compression budgets used in the main text, the combination of layer-wise SVD, joint end-to-end training, and multi-head specialization produces inheritors that are at least competitive with and sometimes strictly better than their teachers.
28

---

Published as a conference paper at ICLR 2026
Table The parameter sizes of the teacher network, student network, and the proposed InherNet on the CIFAR-100 100 dataset.
ResNer32x4
VGG11 9.46M
WRN:40-2
WRN-40-2
ResNer56 861.6KK
ResNetl
ResNet110
Teacher
7.43M
2.26M
2.26M
1.74M
1.74M
ResNet8x4
VGG8
WRN-40-1
WRN-16-2
ResNer20
ResNer32
ResNet20
Student
1.23M
3,97M
569.78K
703.28K
278.32K
472.76K
278.32K
InherNet
Small
907. 007.19K,en
3.82M
540.55K
540.55K,
212.05K,
417.12K
2
Large
1.76M,
6.91 2,0
3.01M
1,01M,
388.88K,
773.18K,-.2
417.12K,
For example, on CIFAR-100 in Sec. 3.1.1 the InherNetrarge compressed from ResNet-110 achieves 75.88% top-1 accuracy, surpassing the ResNet-110 teacher (74.31%). In multimodal CLIP-style benchmarks in Sec. 3.3 the inherited model reaches 36.65% zero-shot ImageNet accuracy, signif- icantly outperforming the ResNet-101 101 teacher (32.75%) These results indicate that the layer-wist SVD initialization does not trap the model in systematically suboptimal region of the loss land- scape; instead, it provides a good starting point within low-rank subspace, from which global training recovers even improves the teacher's performance.
C EXPERIMENTAL DETAILS
C.I SETTINGS
C.I.1 VISION: IMAGE CLASSIFICATION
In this task, we conduct extensive experiments to validate the advantages of the proposed InherNet over traditional KD methods. The experimental setup is as follows:
(Krizhevsky & Hinton 2009) and ImageNet (Russakovsky et al. 2013). CIFAR-100 contains 100 classes, with 50,000 training images and 10,000 validation images. ImageNet includes 1,000 classes, with ap- proximately 1.28 million training images and 50,000 validation images.
Datasets. We use two widely adopted image classification
Baselines. We compare against several mainstream KD methods, including logit-based methods:
KD (Hinton et al., (013).. CTKD (Li et al.] 20236), DKD (Zhao et al., (022). MLKD (Jin et al.
2023), ,, and Logit Std. (Sun al. 20244; well feature-based methods: CRD
(Peng al. 2019).
OFD (Heo et al. 2019), ReviewKD (Chen et al. 2021). SimKD (Chen et al.
2022). and CAT-
KD (Guo al. 2023).
Implementation Details. Following previous works (Chen et al. 2023 (Sun al.)
2021 Zhao et al. 2022 Jin et al,
use SGD optimizer with initial learning rate of 0.05, momentum of 0.9, and weight decay of 0.0005. On ZIFAR-100, all methods train for 240 epochs except MLKD, which uses 480 epochs. On ImageNet, all methods train for 100 epochs. For InherNet. the default number of experts H is set to 3, and the rank is adjusted based on the student network. All experiments are repeated four times, and we report the average performance. The experimental settings on CIFAR- 100 and ImageNet are shown in Table | and Table TO] respectively. Table g presents the parameter sizes of the teacher network, student network, and the proposed InherNet on the CIFAR-100 dataset.
annd
C.1.2 LANGUAGE: GLUE BENCHMARK
In this task, we evaluate InherNet's performance on multiple text tasks. The experimental setup is as follows:
Datasets. To remain consistent with prior research (Raffel et al. 2020). we train and evaluate InherNet's performance the GLUE benchmark (Wang et al.) 2019). The benchmark includes datasets used to evaluate natural language understanding systems. Since the WNLI dataset is adversarial in nature in terms of the training set (Raffel et al. 2020 Devlin al. 2019). do not conduct experiments on WNLI. Since the original test set is not publicly available, we test on the GLUE validation set.
Baselines. Our experiments are based on the T5 model (Raffel et al. 2020 implemented Hug- gingFace (Wolf et al. 2020). Due to the lack of widely known and reproducible T5-based KD
29

---

Published as a conference paper at ICLR 2026
Table 9: Experiment settings on CIFAR-100
Parameter
Value
Optimizer
SGD
Epochs
240 (480 for MLKD)
Batch Size
64
Learning Rate
0.05
Learning Rate Decay
Shrinks by 0.1 150th, 180th, and 210th epochs
Momentum
0.9
Weight Decay
5e-4
CE Loss Weight
KD, CTKD: 0.1
DKD, MLKD: 1.0
Temperature (T)
(default)
KD Loss Weight (Λςδ)
(default)
DKD: 12
Table 10: Experiment settings on ImageNet
Parameter
Value
Optimizer
SGD
Epochs
100
Batch Size
512
Learning Rate
0.2
Divided by 10 at 30th, 60th, and 90th epochs
Momentum
0.9
Weight Decay
le-4
CE Loss Weight
KD, CTKD: 0.1 DKD, MLKD: 0.5
Temperature (r)
2 (default)
KD Loss Weight (AKD)
(default)
DKD: 12
methods, we use the traditional KD method (Hinton et al. 2013), with a teacher-student setup of T5-Base/T5-SSmll with parameters 222M/60M.
Implementation Details. We use the AdamW optimizer (Loshchilov & Hutter 2017) with a training
leamning rate of 0.0003 and distillation learning rate of 0.0005, with weight decay set to 0.1. The number of training epochs is 10, the batch size is 32, and the distillation coefficient and temperature set to 0.3 and 4.0, respectively. The rank and number of expert heads of InherNet are 128 and 2, respectively, with a total of 128M parameters.
C.1.3 VISION + LANGUAGE: MULTI-MODAL EVALUATION
We use the AdamW W optimizer (Loshchilov & Hutter 2017) with a learning rate of 0.001 and a weight decay of 0.1. A cosine learning rate schedule is applied with a linear warm-up for 10K iterations over 30 epochs. The batch size is set to 1024, and the hyperparameters for CLIP-KD are consistent with those in the original paper (Yang et al. 2024). while other training settings follow the original CLIP (Radford et al.) 2021). It is worth noting that, due to limited computational resources, experiments were pre- pre-trained only on CC3M, with all other experimental settings aligned with those in the original CLIP-KD paper. Following standard practices, we use Recall @ K (R @ K) to evaluate
30

---

Published as a conference paper at ICLR 2026
Table 11: Configuration of paired visual and text encoders used in multimodal evaluation.
Visual encoder:
CNN
Text encoder:
Transformer
Model
Params
Layer
Width
Head
Params
ResNet-101
56.26M
12
512
8
37.83M
ResNet-18
11.43M
EfficientNet-B BB
4.66M
12
384
21.29M
MobileNetV3
2.04M
InherNet
12.74M
12
512
18.95M
Table 12: Comparison of top-1 and top-5 Accuracy (%) on the ImageNet validation set between In- herNet and previous state-of-the-arr knowledge distillation methods, using ResNet34 as the teacher and ResNet18 as the student.
Ausunary
Teachas
Stadent
CRD
OFD
ReviowDD
SimKD
CAT-KD
KD
KD<CTKD
DKD
MILD
MIKD4Lagi Skil
lihhr
Inlas
teg-1
11.26
21.20
1ug-3
89.07
90:45
90.05
90.73
retrieval performance within the top K nearest neighbors. Table visual and text encoders used in multimodal evaluation.
shows the configurations of the
C.2 RESULTS ON IMAGENET
Table 12 presents a comparison top-1 and top-5 accuracy between InherNet and various distilla- tion methods on the ImageNet dataset. It can be observed that InherNet Large continues demon- strate leading performance on the large-scale dataset, while InherNet Small achieves good balance between efficiency and effectiveness.
C.3 ADDITIONAL EXPERIMENTS ON LARGE TRANSFORMER LANGUAGE MODELS
The T5 experiments in the main text demonstrate that InherNet naturally extends to encoder- Transformers Raffel et al. (2020). To further assess the applicability InherNet to large-scale autoregressive LMs, we conduct additional experiments on LLaMA-2-7B Touvron et al. (0033) using its HuggingFace implementation.
Why conventional KD difficult for LLaMA-style LMs. Token-level KD for large decoder- only LMs faces several practical challenges:
High-dimensional logits. ILaMA-2 uses vocabulary of roughly 32K tokens. Token-level KD requires the teacher output the full vocabulary distribution at every generation step and the student match it via KL loss. This leads to large tensors of shape 32k x (sequence length) x (batch size), which makes KD memory- and bandwidth-intensing especially for long sequences.
Strict teacher forcing. For stable token-level KD, the teacher must read ground-truth tokens when producing logits. If the teacher instead conditions on become unstable. However, teacher forcing at each time step requires forwarding the teacher model for every position in the sequence, which increases both memory usage and computation.
tokens, the KL targets
Tokenizer mismatch. Standard KL-based KD assumes that the teacher and student share the same vocabulary and output dimension. When the tokenizers differ, the teacher's probability vector over its vocabulary cannot be directly aligned with the student's vocabulary, and the KL loss no longer well-defined.
practice, KD is thus only straightforward when both models share identical tokenizer and output layer.
Empirical instability. Recent studies on KD for autoregressive LMs report that traditional KD can degrade performance fail to converge in some regimes Gu at. 2023) Zhong al. (024). further limiting its applicability to large L.L.s.
token-level KL loss between teacher and student outputs. Instead, operates directly at the level of network weights: we factorize and inherit the linear layers of the teacher using the proposed SVD-based initialization and asymmetric MoE
In contrast, our proposed InherNet does not rely
31

---

Published as a conference paper at ICLR 2026
Table 13: Additional results on LLaMA-2-7B. "Math" reports accuracy on GSM8K; "Generality" reports accuracy on MtLU-style evaluation. The InherNet achieves comparable or better per- formance than the 7B teacher in math reasoning while using substantially fewer parameters.
Method
Parameters
Math
Generality
Teacher
7B
76.34
50.31
Student
7B
76.88
51.06
InherNet
4.2B
77.19
49.82
structure. As result, InherNet is agnostic to the exact vocabulary and can be applied even when the teacher and student have different tokenizers.
We implement InherNet on LLaMA-2-7B, setting the number of experts to H = 4 and the rank of each inherited linear mapping to 256. We consider domains:
Math reasoning. We fine-tune on MetaMathQA Yu et (2023) and evaluate on GSM8K Cobbe et al. (202T).
General instruction following, We fine-tune on databricks dollossddolllSs5l 15k Conover et al. (2023) and evaluate on an MMLU-style benchmark Hendryykk al. (2020).
Conventional KD baselines are not applicable due the aforementioned constraints. Instead, we include a self-distillation baseline where both teacher and student share the same LLaMA-2-7B weights. This setting is equivalent to Zhang et al. (2019), and thus offers no parameter compression. For InherNet, we compress the 7B-parameter teacher into -.2--parreeter InherNet model. The results are summarized in Table
Discussion. The --B-amammtterr InherNet model slightly surpasses the original 7B teacher on GSM8K and maintains competitive performance the general instruction-follooinn benchmark. This indicates that InherNet is also applicable to larger-scale Transformer LMs, achieving strong performance with nearly half the parameter count. We attribute this behavior to the extended SVD initialization, which preserves most of the teacher's s forward computation, and the asymmetric MoE structure, which supports stable generalization during fine-tuning.
Summary of scalability to larger models The above experimental results already evaluate Inher- Net on substantial-scale models and datasets. To further demonstrate scalability large decoder- only Transformer LMs, we instantiate InherNet I3). this experiment, the teacher and the self elf-distilled baseline both use the full 7B-parameter LLaMA-2 model, whereas the inherited model compresses the teacher into 4.2B B-aaameete Inher- Net. The inherited model matches or slightly surpasses the teacher.
LLaMA-2-7B, following the above setup (Ta-
These results, together with the architecture-agnosii nature of theory (Theorem 2.4 Theo- rem B.5, and Corollary [2.9). indicate that InherNet is not limited to small networks: the same SVD-based inheritance and asymmetric MoE structure can applied to larger convolutional, Trans- former, and multimodal architectures, and remains effective in large-model regimes.
We have conducted additional ex- periments the VQA task using VQA v2.0 Goyal al. (201)) With the teacher/student models VLMileVLM V2 3 1.7B" Chu et al. 20024), we follow Chu et al. (0244) by freezing the vision en- coder and tokenizer, and splitting training into pre-training stage and multi-task fine-tuning stage. For top-K selection in text-focused vision-token knowledge distillation, we take the top 16 tokens with the highest attention scores. In the multi-task stage, only VQA data is used. All other settings follow Chu et al. (2024). Given the dataset's large scale and limited computational resources, which make training excessively time-consuming, we train the model on only 10% of the training set and evaluate cross cross-modal alignment on the full test set. We compare the proposed InherNet with vanilla KD Hinton et al. 2015). setting the number of experts 4 and rank 256. Results are shown
InherNet's performance on more complex multimodal tasks
below:
https://hugginngaacecc https:/wwggnnnrrr.e...ttttyyylls
32

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of VQA performance between vanilla KD and InherNet.
Method
Size
overall
other
number
Vanilla
1.7B
56.12
38.45
34.28
InherNet
1.5B
57.03
41.63
37.22
InherNet
InherNet KD
77
76
75.6
75.6
75.8
Accuracy (%)
75
74.7
74.5
74.4
74.1
74
73.7
72.7
72
71
70
8
16
32
64
128
Rank r
Figure 3: The impact distillation on InherNet different scales with varying ranks.
C.4 EXPERIMENTAL DETAILS OF INSIGHTS ON INHERNET
Insight 1. Distillation benefits small-scale InherNet, but harms the performance of large- scale InherNet.
We construct InherNets of different scales based on various ranks (r = 8, 16, 32, 64, 128), using them student networks, and adopt ResNet-56 as the teacher network for knowledge distilla- tion (Hinton et al. (015). Figure shows the performance changes of InherNets of different scales before and after distillation. The results indicate that small-scale InherNets (r I 32) benefit sig- nificantly from distillation, with smaller models showing greater improvement. In contrast, large- scale InherNets experience a performance drop after distillation. We speculate that this is because large-scale InherNets already inherit most of the teacher's knowledge and may outperform the teacher through independent training, InherNet Large consistently outperforms the teacher network). Therefore, further distillation offers little benefit such networks and may even have negative effect.
Insight 2. Rank significantly affects model performance; using multiple expert heads performs better than single head.
We investigate the impact of rank size and the number of expert heads H on the performance of InherNet. shown Figure rank exerts a more significant influence on InherNet. We attribute this to the fact that rank directly determines how well InherNet inherits core knowledge from the teacher network, which is crucial for subsequent learning-an advantage that cannot compensated by simply increasing model parameters such as the number of expert heads. Moreover, also observe that single-head structures perform worse than multi-head ones. However, increasing the number of heads beyond (H > 1) does necessarily lead to further gains. We speculate that while multi-head mechanisms combined with gating improve generalization and diversity, too many expert heads may cause overfitting.
Insight 3. Compared to traditional distillation methods, InherNet converges significantly faster.
We select the commonly used KD method with epoch=240 as the subject for studying model con- vergence speed. Figure 5 shows the change in training and testing loss during the training process for different methods icantly faster than traditional KD methods. However, InherNet without SVD initialization shows highly unstable generalization performance during training. This phenomenon strongly supports the effectiveness of the proposed extended SVD method in accelerating model convergence.
single experiment. The results indicate that InherNet converges signif-
33

---

Published as a conference paper at ICLR 2026
Rank
8
16
32
64
128
71.13
71.32
73.24
75.87
75.93
75
2
72.91
73.48
73.58
75.67
75.81
Head II
74
3
75.59
73
4
72.53
73.79
74.73
75.48
75.70
72
5
71.86
73.56
74.78
75.87
75.72
Figure 4: The impact of rank size r and the number of expert heads H on the performance of the proposed InherNet.
CID
Restew
Itlcc Lep
Inte Meloonn N.D.3RD
Melure
Training Losa
00
Teaking Lowa
00
24
10.5
352
Epach
Spoch
Figure 5: The training and testing loss curves of InherNet and various KD methods during training.
D RELATED WORK
D.1 KNOWLEDGE DISTILLATION
Knowledge distillation (KD) (Hinton al. 2015) is an efficient model compression technique that helps smaller student network learn better during training by transferring the knowledge of larger teacher network. Traditional knowledge distillation methods typically focus on minimizing the prediction differences between the teacher and student networks, typically using loss functions like KL divergence calculated from the teacher's 's logit outputs. ecent research has focused on two main directions: logit distillation (Hinton et al., 2015 Chen al.) 2020 Li al., [0200] 2023b Zhao et al. 2022 Jin et al. [2023 Zhang et al. [2023a] Sun et al. (2024) and feature distillation (Yim et al.
2021 Chen al. 2022 Lin et al. 2022 2022 Guo et al. (023). In logit distillation, the goal is to optimize the similarity of the outputs by matching the prediction probabilities of the teacher and the student, while feature distillation focuses on transferring the intermediate feature representation within the model, providing more fine-grained learning information. However, due to the limited capacity of student network, especially the performance ceiling imposed by the teacher network, the student's performance often struggles to match that of the teacher (Gou et al. 2021 Hu et al. 2023 Xu et al. 2024b).
2017 Peng al. [019 Heo et al. 2019 Chen al. 202
Li al.
D.2 LOw-RANK DECOMPOSITION
In recent years, low-rank decomposition has become key focus in model compression, aiming reduce storage requirements and improve inference efficiency by approximating weight matrices through the product of low-rank matrices. Initially applied to the weight matrices in convolutional neural networks (CNNs), particularly the 4D tensors of convolutional kernels, early studies (Denton)
al. 2014 Lebedev et al. 2014 Tai et al.) 2015 Xu et al.) 2019: Yang al. 2020) focused on effective matrix and tensor decomposition methods, such as using CP decomposition to decompose convolutional kernels into multiple low-rank layers (Lebedev et al. 2014). While this approach reduces model complexity, can lead to issues like vanishing or exploding gradients when scal- ing to large, deep networks, making training and fine-tuning more challenging ('Tai et al. 2015).
34

---

Published as a conference paper at ICLR 2026
To address this, subsequent research has reshaped nigh-dimensional tensors into 2D matrices and applied techniques like singular value decomposition (SVD) for compression, effectively reducing computational load. For example, channel decomposition (Zhang et al. 2015) using SVD reduces convolution layers to two, with one layer being a 1 convolution, and further eliminates small sin- gular value channels reduce computation. However, these low-rank decomposition methods typically applicable to specific CNN structures and vision tasks (Wang et al. (0026). Moreover, ex- isting methods have not explored the potential link with student networks in KD, while our proposed InherNet aims to establish an effective bridge between the two.
Low-precision low-rank compression. Recent work has explored randomized low-precision low- rank factorization for compressing individual matrices and large language models. [Saha et al. (2023) propose LPLR, general matrix comtiii-cooppsssiin algorithm that approximates a single matrix as A R LR using randomized sketching and joint low-rank and low-precision factors, with approximation- error guarantees. [Saha et al. (2024) builds on this idea and applies low-rank and low- sewpprccision decompositions all weight matrices in LLM, yielding post-training compression scheme that focuses on bit- and rank-efficient storage while keeping the network architecture unchanged.
In contrast, work is not generic matrix-compressio routine but SVD-driven network inher- itance framework: we reparameterize the teacher model into low-rank, gated one-down-many-un module and train the resulting InherNet end-to-end. Our analysis centers on inheritance-awant chitecture and optimization, including convergence guarantees, parameter-efficiency bounds, and expert specialization, which is theoretically instructive and complementary Saha al. (2023) and Saha et al. )2024).
E THE USE OF LARGE LANGUAGE MODELS
A Large Language Model (LLM) provided assistance in the preparation of this manuscript. Its role was strictly limited
Proofreading: Checking the text for spelling errors and grammatical accuracy.
Suggesting alternative phrasing to enhance readability and flow, without changing the scientific meaning.
Improving Clarity:
All changes suggested by the tool were manually reviewed and implemented by the authors to main- tain the integrity of the work.
35


Reviews:
[{"review_id": "gYTBpv321W", "reviewer": "Reviewer_FZTJ", "summary": "This paper introduces a new method called InherNet, which is designed to improve how smaller neural networks (like student models) can \"inherit\" knowledge from larger ones (teacher models). Instead of just mimicking the outputs like traditional knowledge distillation, InherNet directly reuses and compresses the teacher’s internal structure using low-rank decomposition and SVD. It builds a kind of lightweight but expressive network that can perform well even with fewer parameters. The authors test InherNet across a range of tasks—vision, language, and multimodal—and show that it generally outperforms existing distillation methods, both in speed and accuracy. They also include a lot of theoretical analysis to explain why this approach works.", "questions": "N.A. See weaknesses", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authos also provide a rigorous mathematical analysis of the InherNet architecture, such as (1) convergence guarantees under standard assumptions, (2) proofs of parameter efficiency and knowledge preservation based on singular value spectrum, (3) formal definitions of Parameter Efficiency, Expressivity-to-Parameter Ratio, and Approximation Error.\n\n2. The design of a fixed asymmetric expert-head structure combined with SVD-based decomposition is novel. The paper provides a clear architectural comparison with LoRA and other PEFT approaches. Their proposed method offers a new paradigm in model inheritance, distinct from traditional KD or PEFT methods.", "weaknesses": "1. I noticed is that a couple of the baselines, like MLKD and Logit Std., were trained for more epochs than other methods. That makes it tricky to know if InherNet is really better, or if the training schedule just favors it. For a fair comparison, I think all methods should be evaluated under the same settings. Otherwise, the results lose a bit of their strength.\n\n2. In the analysis section, the authors mention that distillation can hurt performance for large InherNet models, which is super interesting, but they kind of gloss over it. That feels like a missed opportunity. If there’s a sweet spot where inheritance works better than distillation, we need more detail on that. Right now, it leaves a bit of a gap in understanding how to apply the method in different cases.\n\n3. While the paper does include T5 for some NLP benchmarks, the whole method seems mostly built around CNNs. If this method is going to be truly general, we need to know how it applies to modern Transformer models, especially for large-scale language tasks. At the very least, I’d like to see a bit more discussion or even a small-scale experiment showing that InherNet can handle Transformer-style architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILMbaL47l4", "reviewer": "Reviewer_Gz98", "summary": "The paper introduces InherNet, a novel neural network inheritance method designed to overcome the capacity gap that limits traditional student models in Knowledge Distillation (KD). Instead of training a student to mimic the teacher, InherNet directly inherits the teacher’s structure and knowledge through an asymmetric low-rank decomposition of the teacher’s weights. Using Singular Value Decomposition (SVD) for initialization, InherNet reconstructs a lightweight yet expressive network that retains the teacher’s principal components while maintaining architectural consistency. This approach effectively balances model depth, width, and compression efficiency. Experiments on both unimodal and multimodal tasks show that InherNet outperforms conventional student networks with similar parameter counts, demonstrating a new direction for efficient model compression beyond standard distillation frameworks.", "questions": "1. Since the proposed algorithm performs layer-wise compression, does it result in suboptimal results for compressed models?\n\n2. The proposed algorithm leverages multi-expert structure inheritance to perform compression. Does the number of expert heads result in quantitative results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed SVD-Driven NNI algorithm directly compresses the teacher models instead of training a separate student model during the knowledge distillation process. Therefore, the proposed algorithm can build more complex but lightweight network architectures without harming the number of model parameters and inference time.\n\n2. The proposed algorithm is supported by a comprehensive theory analysis. Thus, readers can fully understand the advantages of the proposed algorithm.\n\n3. The paper demonstrates the effectiveness and efficiency of the proposed SVD-Driven NNI through comprehensive experiments across different models and applications.", "weaknesses": "1. The proposed algorithm performs layer-wise compression during the knowledge distillation process. It does not account for the correlation between layers when performing layer-wise compression. Due to this, the proposed algorithm might have suboptimal results after compression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksleGbHo7A", "reviewer": "Reviewer_pnLL", "summary": "This paper proposes InherNet, a Neural Network Inheritance (NNI) framework that directly inherits the structure and knowledge of a teacher model by performing asymmetric low-rank decomposition and SVD-based initialization on the teacher’s weights, rather than relying on the indirect matching strategies commonly used in traditional knowledge distillation.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper systematically introduces the \"Network Inheritance (NI)\" paradigm for the first time, breaking away from the conventional knowledge distillation (KD) approach that transfers knowledge solely through soft labels. Comprehensive evaluations on multiple datasets, including CIFAR-100, GLUE, and CC3M, demonstrate the generality and robustness of the proposed method.", "weaknesses": "1. The related work “Matrix Compression via Randomized Low Rank and Low Precision Factorization” also employs LoRA for model compression. The authors need to clarify the distinction between their approach and that work, beyond merely stating that they come from different research domains.\n\n2. The authors should provide an analysis of model performance under varying compression ratios, showing how performance changes as the compression rate increases.\n\n3. The role of the MoE (Mixture of Experts) module is unclear. In traditional MoE architectures, different expert models are typically responsible for distinct tasks. However, in this work, each expert is fine-tuned on the same dataset without explicit specialization, making the proposed MoE appear more like a parameter expansion trick rather than a true mixture-of-experts mechanism.\n\n4. It would strengthen the paper if the authors could evaluate the proposed method on larger-scale models to verify its scalability and generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KijsaZy0mi", "reviewer": "Reviewer_mSWQ", "summary": "This paper introduces InherNet, a novel neural network compression method that extends beyond traditional knowledge distillation by directly inheriting both the structure and knowledge of teacher networks. InherNet leverages asymmetric low-rank decomposition and SVD-based initialization to construct lightweight but expressive networks while preserving the teacher’s representational capacity. The proposed method applies a unified framework that balances depth, width, and compression efficiency, enabling it to achieve superior performance compared to traditional student networks of similar size. Theoretical analyses provide rigorous guarantees on convergence, parameter efficiency, and representational power preservation, making InherNet a robust and impactful contribution to the field of efficient model compression.", "questions": "1. Explore InherNet’s performance on more complex multimodal tasks, such as Visual Question Answering (VQA) or video-text retrieval, and analyze how well it handles cross-modal alignment?\n2. Could the authors provide a thorough empirical and theoretical comparison with LoRA, highlighting the differences in design, performance, and scalability?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of asymmetric low-rank decomposition combined with an expert-style structure is innovative. The proposed approach is distinct from existing KD methods and parameter-efficient techniques like LoRA, offering a fresh perspective on model compression.\n2. The paper addresses an important problem in neural network compression and proposes a novel method that goes beyond the limitations of traditional KD. By directly inheriting the teacher's structure and knowledge, InherNet opens new possibilities for designing lightweight networks with high performance.\n3. The use of SVD-based initialization is a key technical contribution, ensuring stable optimization and effective knowledge transfer. The theoretical analysis of convergence guarantees, parameter efficiency, and representational power preservation is comprehensive and well-supported.", "weaknesses": "1. The paper briefly mentions LoRA but does not include a detailed comparison. Since LoRA also uses low-rank decomposition, it would be valuable to analyze InherNet’s advantages (or trade-offs) compared to LoRA in terms of performance, scalability, and parameter efficiency.\n2. The asymmetric design and expert-style structure may add implementation complexity. The paper could benefit from pseudocode or a detailed algorithmic overview to make the method more accessible to practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)