Geometry-aware training of factorized layers in tensor Tucker format
Authors: Emanuele Zangrando
Steffen Schotthöfer
Gianluca Ceruti
Jonas Kusch
Francesco Tudisco
Venue: NeurIPS 2024 poster (2024)
Keywords: Tucker Tensors
Tensor Neural Networks
Low-Rank Compression
Optimization
Low-Rank Optimization
Dynamical Low-Rank Approximation
Primary area: optimization_for_deep_networks

Abstract: Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during train and inference of large networks. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format and parameterizes it as the product of two rank-r matrices. However, this family of approaches often requires an initial full-model warm-up phase, prior knowledge of a feasible rank, and it is sensitive to parameter initialization.
In this work, we introduce a novel approach to train the factors of a Tucker decomposition of the weight tensors. Our training proposal proves to be optimal in locally approximating the original unfactorized dynamics and stable for the initialization. Furthermore, the rank of each mode is dynamically updated during training.
We provide a theoretical analysis of the algorithm, showing convergence, approximation and local descent guarantees. The method's performance is further illustrated through a variety of experiments, showing remarkable training compression rates and comparable or even better performance than the full baseline and alternative layer factorization strategies.

Geometry-aware training of factorized layers in tensor Tucker format
Emanuele Zangrando" School of Mathematics, Gran Sasso Science Institute,
Steffen Schotthöfer"
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA schotthoefers@onn..c gov
I." I'Aqual Italy emanuele.. wanggandoggng..ssii......
Gianluca Ceruti Department of Mathematics, University of Innsbruck, Innsbruck, Austria gianluca, www.tteeuuiriae ac. at
Jonas Kusch
Department of Data Science, Norwegian University of Life Sciences,
As, Norway jonas www.hhumuucuuuooooooo
Francesco Tudisco
School of Mathematics and Maxwell Institute,
University of Edinburgh, Edinburgh, UK;
School of Mathematics, Gran Sasso Science Institute, L'Aquila, Italy f tudisco@ed. ac.uk
Abstract
Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach layer factorization, which reshapes weight tensors into matrix format and parameterizes them as the product of two small rank matrices. However, this approach typically requires an initial of a feasible rank, and it is sensitive to parameter initialization. In this work, we introduce a novel approach train the factors of Tucker decomposition of the weight tensors. Our training proposal proves to be optimal in locally approximating the original unfactorized dynamics independently of the initialization. Furthermore, the rank of each mode is dynamically updated during training. We provide a theoretical analysis of the algorithm, showing convergence, approximation and local descent guarantees. The method's performance is further illustrated through a variety of experiments, showing remarkable training compression rates and comparable or even better performance than the full baseline and alternative layer factorization strategies.
warm-up phase, prior knowledge
Introduction
The memory footprint and computational cost for inference and training are major limitations of modern deep learning architectures. A variety of techniques have been developed to address this issue, aiming reduce model size and computational complexity. Popular approaches include
Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, pruning via sparsification struggles to take advantage of GPU hardware designed for dense matrices, and it is difficult to provide error estimates on model performance when using quantization (24). Moreover, while able to reduce resource requirements for inference, these methods struggle to achieve memory reduction during training without affecting performance. As pointed out in (20, 24), training accurate sparse or quantized neural networks from the start is particularly challenging. At the same time, the training phase of modern architectures can require several days on several hundreds of GPUs 77, requiring huge memory storage and energy consumption overheads. Thus, being able to reduce the resource demand of both inference and training while maintaining model performance is of critical importance.
weight sparsification [26, 559 30] and quantization 83 151.
Along with sparsification and quantization, layer factorization is another popular and successful model compression approach. Representing layer weights using different types of matrix and tensor factorizations can yield huge memory reduction while retaining model performance and robustness 67 13. A wealth of recent work has shown theoretical and experimental evidence suggesting that layer weights of over-parametrivar
networks tend to be low-rank 3 5 22 and that removing small
singular values may even lead to increased model performance while dramatically reducing model size [70 68). One significant advantage of low-rank factorizations is that a low-parametric factorized model can be used throughout the entire training (79
36 68) and fine-tuning phase [3] 80). The
most direct way of doing this is by representing each layer's weight tensor W as the product of small factors and then directly training each factor independently in block-coordinate descent. In the matrix case, this boils down to imposing the rank-r parametrization W = UVT for each layer W. with U, V rectangular matrices with only columns. The network is then trained on the set of rank-r matrices MM = (UVT: : U. V have r columns), interpreting the loss as a function of U, V alone. This is the approach taken also by popular fine-tuning strategies such as LoRA [3 86 50
When W is a higher-dimensional tensor, as in the case of convolutional kernels for example, the same approach can be implemented using different types of tensor low-rank factorizations, including canonical polyadic (CP) and Tucker formats [43 49 63 7 42 21. While direct training of layer's
factors is widely used in deep learning, this approach has two major disadvantages:
1. The rank r of the factorization needs to be chosen a-priori, and the performance of the compressed model can highly depend on it [70,79)]
2. The training flow is highly sensitive with respect to the choice of the initialization, which may result in high-oscillatory slow-converging loss and sub-optimal performance and may require warm-up phase during which the full model is trained prior to the rank reduction [79 36 (68.
Point in particular, is directly related to the geometry of the constraints set. In the matrix case, is well-known that M, is a Riemannian manifold with points of very high curvature near small singular values [19]. These points give rise to stiffness and result in ill-conditioning. This intrinsic poor conditioning can be overcome by projecting the gradient flow on the tangent bundle of M, as presented in (68).
In the dimensioall tensor case, we face the same problems. However, trying to adapt the approach for matrices to other tensor factorizations is not trivial, as it may lead to prohibitive computational costs and memory requirements scaling with the order of the tensor. Moreover, not all the tensor factorizations have the required Riemannian structure to design projections and tangent planes. This paper introduces a rank-adaptive geometry-aware training algorithm that trains factorized tensor layers in Tucker format taking advantage of the underlying Riemannian structure, yielding strictly better performance than direct factorizations and overcoming both Points [ 2 above. Our main contributions are:
We design an algorithm for training tensor layers in Tucker format that rank-adaptive, as the ranks of the layers are dynamically updated during training to match a desired compression rate;
We provide theoretical guarantees of loss descent, convergence to stationary points in expectation, and approximation to the ideal full model;
We provide extensive experimental evaluation showing that the proposed method yields remark- able training compression rates (e.g., more than 95% for VGG16 on CIFAR10), while achieving comparable or even better performance than the full baseline and alternative baselines.
2

---

1.1 Related work
Related work on network compression methods differs structurally by the mathematical object of consideration, i.e, matrix- tensor-valued parameter structures, as well as the type of parameter reduction. Weight pruning [28] 63 59 [1] enables parameter reduction by enforcing sparsity, i.c.,
zero-valued weights, whereas low-ra rank compression imposes parameter reduction by factorization
of weight matrices [3 49
79
66
84 and tensors
38
42
72
35
On top different
of approaches that transform tensor layers into compressed matrices 68
34
49
79
tensor decompositions have been used to compress convolutional layers. Such approaches include
a combination of these (21). Other works focus on performing efficient optimization on Stiefel manifolds to preserve cheap parametrizations Other methods consider only
CP decomposition 44
63]. Tucker 442 38 tensor trains 44 72
orthonormality, with methods based on regularization landing [57 8
of orthogonal groups [46] 48) and Riemannian schemes [58
67
the floating point representation of the weights, c.g. 775
25
27
14 76 or combination of the
above [SI. From the algorithmii point of view, related work can be categorized into methods that
compress networks entirely a postprocessing step after full-scale training [60] [66]
38 21 4
iterative methods where networks are pre-trained and subsequently compressed and fine-tuned (2
34
79. and methods that directly compress networks during training [68 61]. As no full-scale training is needed, the latter approach offers a better potential reduction of the overall computational footprint. Only a few of these methods propose strategies for dynamically choosing the compression format during training fine-tuning, e.g.. by finding the ranks via alternating, constraint optimization in discrete [47] and discrete-continuous fashions [4] However, both these approaches require knowledge of the full weights during training and overall are more computationally demanding than standard training. In [68]. a rank- adaptive evolution of the gradient flow on a low-rank manifold was proposed to train and compress networks without using the full-weight representation; however, only for matrix-valus valued layers. While a direct extension of this method to Tucker tensors is possible, the resulting algorithm exhibits prohibitive memory footprint and computational complexity. The development of rank-adaptive training methods for tensor- tensor-valued layers poses non-trivial challenges that may prevent loss descent and performance of the compressed net. For example, numerical instabilities arising from the CP decomposition during training have been observed in [44] and [63)
2 Geometry-aware training in Tucker format
For tensor W, we write Mat. (W) denote the matrix obtained by unfolding W along its i-th mode. The tuple = (r1, r2....,ra ra) is called Tucker rank of W if r; = rank(Mat, (W)). Every d-order tensor W with Tucker rank p = (ri,..., ra) can be written in Tucker form (or Tucker decomposition)
W = C x Us entry-wis defined as
Phoend 1d) = Σ
W 1
(...Ua(a,
OO ag=
where C' E R Xrr is a core tensor of full Tucker rank p = (r1,..... ra) and the U, E R matrices with orthonormal columns. From this representation, we immediately represented in Tucker format, then the cost of storing W and of performing linear operations with W (e.g. matvees convolutions) is O Ta + 71 +... naTa), as opposed to the O(n na) cost required by the standard full representation. When n, >> ri, e.g., n4 > 1.5r,, the latter is much larger than the former.
that if W is
In the following, we develop a rank- cank-adaptive algorithm that trains layers in Tucker form in a robust and efficient manner. Our derivation follows five points:
1. We introduce the dynamical low-rank approximation framework in Section 2.I which provides gradient flows for layers in Tucker format. However, the direct use of these evolution equations to train the network will require prohibitively small learning rates due to the high curvature of the manifold of Tucker tensors.
We introduce a reparameterization in Theorem 2.1 that allows us to formulate robust dynamics for the reparametrized factors.
By integrating numerically the resulting gradient system (with e.g. SGD as explicit Fuler) along with basis augmentation step, we propose geometry aawre rank-adaptive training strategy for
3

---

the network in Tucker format. This approach, however, requires d + 1 forward and backward evaluations of the network, resulting in significantly increased computational costs.
4. We show in Corollary 2.2 that computational costs can be substantially reduced by noting that the computation of the augmented basis can largely be simplified. This leads to our proposed training scheme in Algorithm| The algorithm is equivalent to the integration of the gradient system but requires only two instead of dl + gradient evaluations.
5. Due to its equivalence to the approach constructed in point 3. can show that Algorithm D iindi rectly updates weights along straight lines on the manifold, thus leading to three main theoretical properties: loss descent (Theorem 3.1 convergence in expectation (Theorem 3.2). and robust bound showing approximation of the full model (Theorem 3.3
2.1 Dynamical low-rank approximation
For p (ri,.... rd)), the set
M, = ( rank(Mat((W)) = ri, ,,....d]
is manifold with the following tangent space at any point W C xx Us Mn
40
Tw f0c
x U + Σ C Cx ÖU, X Up 5C E R oUg Tu
(I)
==1
where Sj is the Stiefel manifold of real ny r, matrices with orthonormal columns. To design strategy that computes layer weights within Mp using only the low-rank Tucker factors C' and (Uih. formulate the training problem as continuous-time gradient flow projected onto the tangent space T. As shown in Section 3 the continuous formulation will allow us to derive a modified backpropagation pass which uses only the individual small factors C, {U:} and that does not suffer from a slow convergence rate due to potential ill il-conditioned tensor modes (see also Section 4.2
Let be neural network and let W be weight tensor within f. Consider the problem of minimizing the loss function L with respect to just W while keeping the other parameters fixed. This problem can be equivalently formulated the gradient flow
(tt) V (W)((W())
(2)
where, for simplicity, we write the loss as a function of only W and where "dot" denotes the time derivative. When t - co, the solution of (2) approaches the desired minimizer. Now, suppose we parametrize each tensor layer time-dependent Tucker form (t) = C(t)) _ U.(() E Mp
Then ((t) E Tw. the tangent space of Mp at W()). Thus, ② boils down to
W(I) = P(W (O) W (L)
where P(W) denotes the orthogonal projection onto Tw-Mp- Using standard derivations from dynamical model order reduction literature (4), the projected gradient flow in (3) leads to the following evolution equations for the individual factors C(t) and (t)
Ü, = -T U,U.T) Mat, ( (Vw L(W) Xzyi U,)) Ma),(C))
C=-VwC(W) x X
where i denotes the pseudoinverse and where we omitted the dependence on t for brevity. Even though 4 describes the dynamics of the individual factors, the equations for each factor are not fully decoupled. A direct integration of (4) would still require taping the gradients w L with respect to the full convolutional kernel W. Moreover, the pseudoinverse of the matrices Mat, (C)) adds stiffness term to the differential equation, making its numerical integration unstable. The presence of this stiff term is actually due the intrinsic high-curvature of the manifold M, and is well understood in the 10 9. As observed in [6]] an analogous term arises when looking at low-rank matrix parameterizations, and it is responsible for the issue of slow convergence of low-rank matrix training methods, which is observed in [79, 36 668.
dynamic model order reduction community (39 53 37 54
To overcome these issues, we use the following change of variables. Let (C) = Q.ST be the QR decomposition of MC).(C)T. T. Note that S; is small square invertible matrix of size r, ri. Then, the matrix K; = U.S has the same size as U, and spans the same vector space. However, the following key result holds for K,.

---

Theorem 2.1. Ler W = C Cx U, E Mp be such that (3) holds. Let Mat,(C)T = be the QR decomposition of Mat,(C) (C) and let K; = U. Then,
K; - V ( Ten, (Q x x U3 x; K.).K.).
(5)
C = cCCCC x xز_ Uz)
where Ten; denotes "tensorization along mode i", i.e. the inverse reshaping operation of Mat..
The supplementary material provides the proof in Appendix D The theorem above allows us to simplify (4). obtaining a gradient flow that only depends on the small matrices K; and the small core tensor C. Moreover, it eliminates a stiffness term: this added regularity appears reasonable as no inversion is now involved in the differential equations. A rigorous regularity statement can be found in Theorem 3.3 We would like underline the importance of the careful construction of K; to arrive at this theorem. A naive extension of [68] to Tucker tensors can be constructed by reshaping of W into matrices MW) V, with S; = MC),(C) and V X j Uy. Then, which directly inherits the robustness
K; U.S; can used to update U, into all directions properties presented in [68] However, this construction of K yields a prohibitive memory footprint of (n, and computational costs of (n rendering the resulting method impractical.
Based on the numerical integration of (5) we propose a robust, memory-efficient, and rank-adaptive method to update the network parameters by using only the core tensor C and the basis matrices K, The proposed approach is as follows: Fix an approximation tolerance - > 0, then, first update the basis matrices performing for all i ,,...,d:
1. Form K; = U.S,, where S, is the square ry X T3 matrix from QR decomposition of Mat,(C)*
2. Compute KMTeeW with step integration of (5) starting from K;
3. Form the new augmented matrix Unew by orthonormalizing the columns of [U; Kneww
Second, update the core tensor and truncate:
4. Lift the core tensor Č = C x (UTEE) UÜ using the new augmented basis matrices
5. Compute Cnew with step integration of (5) starting from C using fixed basis matrices Unew
6. Perform rank adjustment step to the prescribed tolerance by computing the best Tucker approxi- mation of Crew, i.e. solving the following optimization (rounding) task:
Find C' E .M of smallest rank p' = (r,...., r'd such that ||C - ClI S C [|, (6)
where p (f1...., F'a) and Mgzp denotes the set of tensors with component-wise Tucker rank lower than 2p.
In practice, the final rank adaptive step is done by performing a high-order SVD (HOSVD) [6] on Cnew The parameter is responsible for the compression rate of the method, as larger values of yield smaller Tucker ranks and thus higher parameter reduction. The computed tensor C E MM has the form C C' x U/ Mpp and the computed U/ R2r,xr with rf S 2r; are then pulled back the initial dimension by the change of basis U, = U?OUU' E R and the new core tensor C' is then assigned C". This implementation, however, comes at the expense of evaluating the network and gradient tapes
times for an order tensor.
The next key result will overcome this issue and will allow us to reduce the necessary network and gradient tape evaluations two.
Corollary 2.2. Let W C x U, E Mp be such that (6) holds. Let Mata(C) =QST be the QR decomposition of (at,(C)) and let K; U.S. Then,
span ([U,, K. = span (LU,, ((,)(
The supplementary material provides the proof in Appendix E Using Corollary 2.2, one can replace the individual forward evaluation and descend steps for K, by a single backpropagation. All available new information is given by the gradients Vu, L, which can be evaluated from the same tape. Combining the above strategy with Corollary 2.2 we obtain Algorithm| I an efficient ann-tadapivve geometry-aware training method for tensor in Tucker format. Note that without the explicit computa- tion of K; = U,S, we compute all basis gradients VU; L. single network evaluation and we use Vu; C to augment the basis. Note that stochastic gradient evaluations can be done in practice and that
5

---

Algorithm 1: TDLRT: Efficient Tensor Dynamical Low-Rank Training in Tucker format.
Input: Initial low-rank factors C ~ ri X... Vaa U, n, X ri;
adaptive: Boolean flag that decides whether or not to dynamically update the ranks;
T: singular value threshold for the adaptive procedure. G2 t Vu, C(C x U:)
/* Single-sweep grad evaluation */
2 for each mode do
3 UPeW + QR(U,, G.I)
/* Augmentation and orthonormalizationn */
C C X (UPeN)TU TU,
/* Projection Tucker core onto new basis */
5 C' t descent step with direction VcL(C x UTew)
(C,UI..... Ua) < Tucker decomposition of up to relative error 7 as 66)
7 t U-UTUUU U.. for i = -1 1....,d
/* Rank ad adjustment */
momentum methods are applicable for the descent step on line of Algorithm T In case the rank decreases after the retraction step, we only use the corresponding subset of the old basis functions to form the momentum term, In case of rank increase, the momentum term of the new basis vectors is initialized as zero. As side note, the rank truncation proposed in Algorithm T allows to maintain the nice theoretical guarantees of the algorithm, but in practice any completionnfaactoizatinn method such as 23 could be used for this step.
2.2 Computational Complexity
The computational costs for the full network training come from back and forward passes through each layer. For layer with weight tensor W R x--, they require O II, n.) operations, where b is the batch size. When using TDLRT, these computational costs reduce to O ra + +b n,r;) operations, yielding significant reduction in computational costs to determine the gradient. However, performing low-rank updates also adds computational costs due to several factorizations. Here, the QR and SVD Mat, (C',, which are needed in the updates of U, and the truncation step, require OEE, Ts II, ry), and the QR on K; requires OEE, n.r?)) operations. Hence, in total, we have for every layer a cost of Ob II: + b b. neY; + Ei= (mar? + r; I[;=1))) ry)) operations for TDLRT, vs. the Obb II; nı)) required by the full baseline. Thus, TDLRT scales linearly with the dimensions T4j and for r, < n4, which is typically the case, see Appendix C.2 it has advantageous computational cost.
3 Convergence and approximation analysis
In this section, we present our main theoretical results. First, we show Algorithm D guarantees descent of the training loss, provided the compression tolerance is not too large. Second, we show that when Algorithm is implemented with SGD with a decaying learning rate, the method converges to stationary point in expectation. Third, we prove that the compressed network computed via the rank-adaptive TDLRT scheme in Algorithm -pprooaapsos the full model that one would obtain by standard training, provided the gradient flow of the loss is, at each step, approximately low-rank. The latter result shows that if a high- performing subnetwork of low Tucker rank exists, then the proposed TLDRT will probably approximate it. For brevity, some statements here are formulated informally, and all proofs and details are deferred to Appendix [ in the S.M. Suppose that for each convolution W. the gradient V W. Lipschitz, i.e., V C(Y) [| & L, and VVV,) C(2))l S L2l|Vi - Yzll around W. Then,
function of W, is locally bounded and
Theorem 3.1 (Descent). Let ((A) C x U, be the Tucker low-rank tensor obtained after one training iteration using Algorithm and let (O)) be the previous point. Assuming the one-step integration from 0 to X is done exactly, it holds Lw (W)) < LW (O() at + Br, where a, 3 > are constants independent of and T, and where Lw denotes the loss as a function of only W.
We now prove that the rank adaptive training method in Algorithm I converges to a stationary point in expectation if implemented with SGD and decaying learning rate.

---

94-
2 96
basdkes
TOURT
TELRT
TCLAT Tucorr
254
ratelis
(a) Alexnet Cifar10
VGGI6 Cifar10
(c) ResNet18 Cifar10
ResNet18
Tiny-
Imagenet
Figure/ 1: Comparison of compression performance for different models against the full baseline for the Cifar 10 (a-c) and Tin--Imaeene (d) benchmark. The mean accuracy of 20 weight initializations displayed. TDLRT achieves higher compression rates at higher accuracy with lower variance between initializations.
Theorem 3.2 (Convergence). Denote with (t) is the weight tensor after t passes of Algorithm before the rank truncation step, and ((1) the one obtained after the rank trancation. Assume Algorithm is implemented using SGD as descent method with learning rale sequence (X²) satisfying the Robbins-Monro conditions:
Σ.
<
Suppose also that the spectral distribution stabilizes fast enough over time, i.e,
E120ElIW (I) - ((II < +oo
and that the projected stochastic gradient has a controlled drift, namely
E
V 1) x, Pi 1 s + D/V 1) x, (ū,(t-1)11
for some μ. v > O, where Py is the orthogonal projection onto the range U. Then, the following convergence condition holds
Inf E[ X5 P) t-000
Details of the proof are contained the appendix.
Theorem 3.3. For integer k. let L = kλ, and let WL)) be the full convolutional kernel, solution of ② at time Let C(I), {U;(1)} be the Tucker core and factors computed after k training steps with Algorithm 2 where the one-step integration from
is done exactly. Finally, assume that for any Y in a neighborhood of W (t), the gradient flow -V Lw (Y) is "e-close" to Ty Mg. Then,
[W(() C(1) X/-1 U,(0I S 6g 02 +2/+c0///
where the constants C1, C22 C3 depend only on L and L2.
In particular, both bounds in the above theorems do not depend the higher-order singular values of the exact nor the approximate solution, which shows that the method does not suffer instability and slow convergence rate due to potential ill-conditioning (small higher-order singular values). Note that this result is crucial for efficient training on the low-rank manifold and is not shared by direct gradient descent training approaches, section. Moreover, we emphasize that (7) provides a sufficient condition for the computation of high-performing subnetwork. In fact, for smooth enough network models f, condition (7) implies that (() = f(C(t) x (t), i.c. the computed subnetwork approximates the full model.
we will numerically demonstrate in the following
4 Experiments
In the following, we conduct a series experiments to evaluate the performance the proposed method as compared to the full model and to standard layer factorization and model pruning baselines. The full baseline the network trained via standard implementation. tensor layers, we consider here convolutional networks and apply the decomposition to the
order to test the method

---

convolutional kernel W. In terms of layer factorization, we compare against different baseline approaches: direct training of the factors in the low-rank matrix factorization format [34 49 79 36), the low-rank tensor Canonic-Polyadee
the low-rank tensor Tucker format 42 338, and the low-rank Tensor-Train format 662 The convolution operation can then be written
format 44
71
4
63
completely in terms of the small factors using the factorization ansatz. While the above papers propose different initialization and rank selection strategies, all the referenced literature trains the factors in the chosen layer factorization format by implementing forward and backward propagations simultaneously and independently on each factor in block-coordinatt fashion. This way of training on the low-rank manifold ignores the geometry of the manifold, whereas TDLRT directly exploits the underlying geometry to avoid points of high curvature. We compare the training strategy alone. Thus, we ignore any model-specific initialization and regularization addition. We also compare with Riemannian gradient descent (RGD) for tensors in Tucker format implemented using the HOSVD retraction 77 17 and with the matrix dynamical training algorithm [68], where the standard forward and backward passes are endowed with a rank-adaptive QR projection step, similar to the proposed Algorithm In terms of pruning techniques based on sparsification, we compare with methods from two of the most popular strategies: iterative magnitude pruning (IMP) (20), and single-shot pruning initialization, single-shot network pruning (SNIP) 455 and Gradient Signal Preservation (GraSP) 778. The experiments are performed on an Nvidia RTX3090, Nvidia RTX3070 and one Nvidia A 100 80GB. The code is available in the supplementary material.
[
4.1 Compression Performance
The compression performance of TDLRT is evaluated on CIFARIO and tiny-imagenet. The typical data augmentation procedure is employed for this dataset: a composition standardization, random cropping, and random horizontal flip. All methods are trained using batch size of 128 for 70 epochs each, as done in 77 36.. All the baseline methods are trained with the SGD optimizer; the starting learning rate of 0.05 is reduced by factor of 10 on plateaus, and momentum is chosen as
0.1 for all layers. The rank of each tensor mode for the fixed-rank baseline methods is determined by parameter k. i.e., we set where SGD is used for the descent steps at lines 4 and 9, with momentum and learning rate as above. Dynamic compression during training is governed by the singular value threshold т, see Equation (
KT max. The proposed TDLRT method employs Algorithm 2
inference time 177
Alecten- TDU VSG16 TOURT
vSGfe
aaseline
0.8
Method
tine 00% sox.
6.6-
case) nilk)
2,35
14 rar,, 4 floxmet 12.0
0.4-
30
0.2
TO Tacker
TDLRT
CP
10 convolution paramters
20'
6.0
202
d0 6ic
800
optimization step
Figure 2: Left panel: Computational footprint of low-rank convolutions. TDLRT surpasses the baseline performance for meaningful compression rates. Middle panel: Convergence behavior of Lenet5 on MNIST dataset in the case of an initial overestimation of the rank, with exponentially decaying singular values. Mean and standard deviation (shaded area) over 10 random initializations. Right panel: Compared to standard Tucker decomposition, with and without rank adaption, wall training time to reach 60% accuracy for TDLRT.
Figure I (a-c) shows the mean accuracy of TDLRT as compared to competing factorization baselines. TDLRT achieves higher compression rates at higher accuracy with lower variance between weight initializations than the competing methods. In the case of the VGG16 benchmark, TDLRT is able to maintain baseline accuracy for compression rates over 90% and exceeds the baseline on average for 0.03, i.e., 95.3% compression. Alexnet has 16.8% of the parameters of VGG16. Thus, compres- sion is naturally more challenging to achieve. Nevertheless, TDLRT outperforms the baselines and remains close to the full network performance. Similar behavior is observed on ResNet18.
Table [ shows a comparison of the best- pest-peeformnn compression between all the factorization-bant and pruning-basse based baseline methods as well as TDLRT in the CIFAR10 benchmark for Alexnet, ResNet18, 88, and VGG16. In the proposed evaluation, TDLRT is on par or outperforms all the alternatives, including pruning based on sparsity (implemented without warmup for the sake of fair
8

---

Table Comparison of the performei compression rates for different methods on the CIFARIO benchmark with Alexnet, VGGI6, and ResNet18. For each column, we report first and second best results.
VGGI6
Alexnct
ResNet18
test acc. [%)]
c.r. [%]]
test. acc. [%]]
c.r. [%]]
test acc, [%]
C.r. [%]
Baseline
92.01
0.0
85,46
0.0
94.33
0.0
TDLRT (ours)
90.23
94.40
82.39
83.12
92.72
78.73
Factorization
Matrix DLRT
168
89.13
83. 23.22
73.57
71.57
80.98
56.85
Tuccker-fctorizeed
R8
86.71
91.4
70.30
69.74
91.11
74.19
Matrix-factorized
79
84.54
94.34
77.07
68.20
92.07
77.49
CP-factorized
44
82.53
89.98
76.14
71.46
91.87
69.95
Tucker RGD
74
81.48
84.26
73.88
74.01
92.76
74.18
TT- TT-factorized
62
87.27
90.30
78.13
88.14
87.13
81.24
Pruning
SNIP 45
89.58
56.23
89.50
78.50
IMP 20
87.21
58.54
90.50
82.50
GraSP
78
88.50
77.30
89.40
77.90
comparison), Tensor- Train (TT) and Tucker factorization, Riemmanian gradient descend (RGD) for Tucker decompositions, well as the matrix-valued DLRT, due to the higher flexibility of the Tucker format where compression along each tensor mode individually is possible. The compression rate (c.r.) computed as 1 c/ f, where e is the number of convolutional parameters in the compressed model after training and f the number of convolutional parameters of the full model. While this the compression rate after training, we emphasize that methods based on factorizations yield an analogous compression rate during the entire training process. We also remark that no DLRT version of CP decomposition is shown as CP is not suited for dynamical low-rank training due its lack of manifold structure. Similar results are obtained for the tiny-imagene benchmark, see Fig. (d) and
Table The rank evolution of the networks during training discussed in Appendix C.2
4.2 Robustness of the Optimization
To further highlight the advantages of Algorithm@an as compared to standard simultaneous gradient descent on the factors of the decomposition, we show in Figure the accuracy history of LeNet5 on MNIST using TDLRT as compared standard training on Tucker and CP decompositions. In the case of TDLRT, an optimization step denotes the evaluation of Algorithm for all convolutional layers for one batch of training data, while for the other methods, we refer to a standard SGD batch update for all factors of the tensor decompositions of all layers. All linear layers of the network are trained with a traditional gradient descent update and are not compressed. In this experiment, we initialize the network weights simulate scenario where the rank is overestimated. To this end, we employ spectral initialization with singular values decaying exponentially with powers of ten. Integrating the low-rank gradient flow with the TDLRT Algorithm 2 leads to faster and more robust convergence rates the network training process.
4.3 Computational Performance
The computational performance in inference and training of convolutional layers in Tucker decom- position depends on their current tensor ranks, see Section 2 We evaluate the inference time of 120K RGB images and memory footprint of VGG and AlexNet in Tucker factorization as used in Algorithm and compare them the non-factorized baseline models in Figure 2 As a result, for realistic compression rates, see also Figure I] the computational footprint of TDLRT is significantly lower than the corresponding baseline model.
Rank adaptive TDLRT training comes at the additional expense of QR and SVD operations per optimization step compared to standard fixed rank training without orthonormalization. However, the increased robustness of the optimization and faster convergence reduces the computational overhead of TDLRT as demonstrated in Figure 2
In order to provide a fair comparison between the methods, report the time to achieve 60% accuracy target at compression rate of 90% for all methods,

---

LeNet5 MNIST with the setting as in Section 4.2 We refrain from measuring time to convergence since the standard Tucker decomposition is not able to reach similar accuracy levels the same compression ratio as TDLRT.
4.4 Fine-tuning with LoRA-like low-rank adapters
In this section, we presenting another application of our method, namely fine-tuning pre- pre-trained models through the use of low-rank adapters. In particular, our approach Algorithm Tis completely agnostic between model compression and adaptation, the approach is the same. More precisely, given a pre-trained model fw with tensor pre-trained weights W', it is possible to add a low-rank Tucker correction W. Then, given task loss L(f), by defining L(W) L(fw-+w)
report ourselves to the original formulation of the problem. Moreover, we would like to stress that this approach can be applied also to matrices, which are particular case of tensors with d 2 modes. To showcase these two settings, we present two different settings in which we test our method. In [2 we show the fine-tuning of Deberta V3 [29 on the GLUE benchmark [77]. In this test case here, the low-rank adapters have been applied to all matrices in attention layers, and the final performance against LoRA [33] fine-tuning is reported. Apart from our additional hyperparameter т, all the other hyperparameters had been kept as reported in [85].
In Z we report the results for fine-tuning stable diffiusonn 644 with Dreambooth (65]. To adapt the model, we applied Tucker tensor corrections to each convolution of the Unet, and matrix adapter to each attention layer of the text encoder network. We applied our method on all these correctors, while keeping the same hyperparameters setting as in [55]. We would like to observe that the kind of adapters they propose for convolutions consist matrix factorization of a reshaping of the convolutional tensor. This results in potentially bigger number of parameters needed, as matrix factorization would be O(Cdd, d2 + F)r) against O(Cri + Fr2 + dirs + d2r's + (222(3)4) for plain Tucker factorization, where F is the number of output features, C' is the number of input channels, dj and d2 are the spatial dimensions of the convolutional kernel. This observation is in fact reflected in the numbers in Table 2 in which we can observe higher compression potential for tensor decompositions compared to matrix ones.
Table 2: Fine-tuning performance metrics on Deberta V3 Glue benchmark (left) and on Stable diffusion Dreambooth (right).
GLUE
LoRA
TDRLT(Ours)
method
loss
# params
# params
1.33M (rank 8)
0.9M (7 = 0.15)
LoRÁ (r = 8)
0.260
5 M
CoLa (Corr.)
0.6759
0.7065
LoRA (r = 5)
0.269
MRPC (Acc.)
0.8971
0.9052
LoRA (r =3)
0.274
1.8 M
QQP (Acc.)
0.9131
0.9215
Ours (T = 0.02)
0.2635
1.8 M
RTE (Acc.)
0.8535
0.8713
Ours (T -0.1)
0.272
1.5 M
SST2 (Acc.)
0.9484
0.9594
5 Discussion and limitations
This work leverages the geometry of the Tucker tensor factorization manifolds construct a robust and efficient training algorithm for neural networks in compressed Tucker format. The proposed method has theoretical backbone of approximation error bounds the full model and guarantees of loss descent and convergence to stationary points in expectation. The method is superior to standard factorization approaches with alternating or simultancous gradient descent, as demonstrated in the compression- to- to-accuracy ratio for various benchmarks and models. The method provides a significant reduction of hyperparameters to a single parameter т. This parameter has clear interpretation as compression rate per layer depending on the layer's 's importance on the overall network training dynamics. We, however, note that further strategies to pick an adequate are possible and remain to be investigated. Further, we note that an efficient implementation on GPUs requires an efficient tensor rounding algorithm Algorithm [ Finally, the proposed method assumes well-performing low-rank Tucker sub-nets exist in the reference network. While we observe this empirically, further investigations are required to provide theoretical evidence supporting this assumption, similar to the case of cully-connetted linear layers, see 6).
10

---

Acknowledgments and Disclosure of Funding
This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-ACO5-000R2270 with the U.S. Department of Energy. The United States Government retains and the publisher,by accepting the article for publication, acknowledges that the United States Government retains non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan.
The work of E. Zangrando was funded by the MUR-PNRR project Llw-paaraetrri machine learning". Francesco Tudisco is partially funded by the project FIN4GEO within the European Union's 's Next Generation EU framework, Mission 4, Component 2, CUP P2022BNB97.
References
[1] Ablin and G. Peyré. Fast and accurate optimization the orthogonal manifold without retraction. In G. Camps- Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 5636-560 5657. PMLR, 28-30 Mar 2022.
[2] P.-. Absil and J. Malick. Projection-like like retractions on matrix manifolds. SIAM Journal on Optimization, 22(1): 1358 2012.
[3] S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[4] M. Astrid and S.-I. Lee. neural networks compression. In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). pages 115-118, 2017.
Cp-decomposition with tensor power method for convolutional
[5] B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers. Inf. Inference, 11(1):307-333 2022.
[6] B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg. Learning deep linear neural networks: Riemannian gradient flows and convergence global minimizers. Information and Inference: A Journal of the IMA, 11(1):307-353, 2022.
[7] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang. A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-244 24654,, 2022.
[8] N. Bansal, X. Chen, and Z. Wang. Can we gain more from orthogonality regularizations in training deep networks? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[9] G. Ceruti, J. Kusch, and C. Lubich. A rank-adaptive robust integrator for dynamical low-rank approximation. BIT Numerical Mathematics, pages 1-26, 2022.
[10] G. Ceruti and C. Lubich. An unconventional robust integrator for dynamical low-rank approxi- mation. BIT Numerical Mathematics, 62(1):23-44, 2022.
[11] G. Ceruti, C. Lubich, and D. Sulz. Rank-adaptive time integration of tree tensor networks. SIAM Journal on Numerical Analysis, 6142): 2023.
[12] G. Ceruti, C. Lubich, and H. Walach. Time integration of tree tensor networks. SIAM Journal on Numerical Analysis, 59(1):289-313 1112899311,,222 2021.
[13] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. International Conference on Learning Representations (ICLR), 2017.

---

[14] M. Courbariaux, Y. Bengio, and J.-P. David. Training deep neural networks with low pre- cision multiplications. In Workshop contribution at International Conference on Learning Representations, 2015.
[15] M. Courbariaux, Hubara, D. Soudry, El- El-vaniv, and Y. Bengio. Binarized neural net- works: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv:1602.0288 02000000..220........ 2016.
[16] L. De Lathauwer, B. De Moor, and J. Vandewalle. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4):1253-12755 2000.
[17] I.. De Lathauwer, B. De Moor, and J. Vandewalle. On the best rank-1 and rank-(r r 2,.... m) approximation of higher-order tensors. SIAM journal on Matrix Analysis and Applications, 31(4):1324-1344 2000.
[18] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas, Predicting parameters in deep learning, 2014.
[19] F. Feppon and P. F. Lermusiaux. A geometric approach to dynamical model order reduction. SIAM Journal on Matrix Analysis and Applications, 39(1):510- 1-100053,,000 2018.
[20] J. Frankle and M. Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019.
[21] M. Gabor and R. Zdunek. Compressing convolutional neural networks with hierarchical tucker-2 decomposition. Applied Soft Computing, 132: 109:56, 2023.
[22] T. Galanti, Z. S. Siegel, A. Gupte, and T. Poggio. Sgd and weight decay provably induce low-rank bias in neural networks. 2022.
[23] M. Ghadiri, M. Fahrbach, G. Fu, and V. Mirrokni. Approximately optimal core shapes for tensor decompositions. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1237-11225, PMLR, 23-29 Jul 2023.
[24] A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference, 2021.
[25] Y. Gong, L. Liu, M. Yang, and Bourdev. Compressing deep convolutional networks using vector quantization. In International Conference on Learning Representations (ICLR), 2015.
[26] Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
[27] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In International conference machine learning, pages 1737-1746. PMLR, 2015.
[28] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[29] He, J. Gao, and W. Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangee embedding sharing, 2023.
[30] Y. He. X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[31] A. Hnatiuk, J. Kusch, L. Kusch, N. R. Gauger, and A. Walther. Stochastic aspects of dynamical low-rank approximation in the context of machine learning. Optimization Online, 2024.
[32] E. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021.
12

---

[33] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[34] Y. Idelbayev and M. A. Carreira-Perpinan Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[35] Y. loannou, D. Robertson, J. Shotton, R. Cipolla, and A. Criminisi. Training enns with low-rank filters for efficient image classification, 2016.
[36] M. Khodak, N. Tenenholtz, L. Mackey, and N. Fusi. Initialization and regularization of factorized neural layers. In International Conference on Leurning Representations, 2021.
[37] E. Kieri, C. Lubich, and H. Walach, Discretized dynamical low-rank approximation in the presence of small singular values. SIAM Journal on Numerical Analysis, 54(2):1020-1038 2016.
[38] Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. In International Conference on Learning Representations (ICLR), 2016.
[39] O. Koch and C. Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434 454, 2007.
[40] O. Koch and C. Lubich. Dynamical tensor approximation. SIAM, 31, 2010.
[41] T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM review, 43)) 500, 2009.
[42] J. Kossaifi, A. Bulat, G. Tzimiropoulos, and M. Pantic. T-net: Parametrizing fully convolutional nets with a single high-order tensor. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7822-7831, 2019.
[43] V. Lebedev, Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky. Speeding-up convolutional neural networks using fine-tuned CP-ddecopositiion In International Conference on Learning Representations, 2015.
[44] V. Lebedev, Y. Ganin, M. Rakhuba. I. Oseledets, and V. Lempitsky. Speeding-up convolutional neural networks using fine-tuned CP-decomposition. In International Conference on Learning Representations (ICLR), 2015.
[45] N. Lee, T. Ajanthan, and P. H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity, 2019.
[46] M. -araado and D. Martinez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization the orthogonal and unitary group. In K. Chaudhuri and R. Salakhutdi- nov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3794-3803. PMLR, 09-15 Jun 2019.
[47] C. Li and C. J. R. Shi. Constrained optimization based low-rank approximation of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[48] Li, F. Li, and S. Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. In International Conference on Learning Representations, 2020.
[49] Y. Li. S. Gu, L. V. Gool, and R. Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEF/CVE International Conference on Computer Vision (ICCV), October 2019.
[50] V. Lialin, Shivagunde, Muckatira, and A. Rumshisky. Relora: High-rank training through low-rank updates, 2023.
13

---

[51] B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[52] C. Lubich. Time integration in the multiconfiguration time-dependent hartree method of molecular quantum dynamics. Applied Mathematics Research eXpress, (015(2))31--325 2015.
[53] C. Lubich and I. V. Oseledets. A projector-spliting integrator for dynamical low-rank approxi- mation. BIT Numerical Mathematics, 54(1):171-188, 2014.
[54] C. Lubich, B. Vandereycken, and H. Walach. Time integration of rank-constrained tucker tensors. SIAM Journal on Numerical Analysis, 56(3):1273-12200 2018.
[55] S. Mangrulkar, S. Gugger, I., Debut, Y. Belkada, S. Paul, and B. Bossan, Peft: State-of-the-art parameter-efficient fine-tuning methods.
2022.
[56] Z. Mariet and S. Sra. Diversity networks: Neural network compression using determinantal point processes, International Conference on Learning Representations (ICLR), 2016.
[57] E. Massart. Orthogonal regularizers in deep learning: how to handle rectangular matrices? In 2022 26th International Conference on Puttern Recognition (ICPR), pages 1294-1299, 2022.
[58) B. Mishra, G. Meyer, S. Bonnabel, and R. Sepulchre. Fixed-rank matrix factorizations and riemannian low-rank optimization, 2013.
[59] P. Molchanov, S. Tyree, T. Karras, Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
[60] M. Nagel, M. Baalen, T. Blankevoort, and M. Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334 2019.
[61] S. Narang, G. Diamos, S. Sengupta, and E. Elsen. Exploring sparsity in recurrent neural networks. International Conference on Learning Representations (ICLR), 2017.
[62] A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov. Tensorizing neural networks, 2015.
[63] A. Phan, K. Sobolev, K. Sozykin, Ermilov, Gusak, Tichavsky, V. Glukhov, I. Oseledets, and A. Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In European Conference on Computer Vision, 2020.
[64] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[65] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning cext-to-image diffusion models for subject-driven generation, 2023.
[66] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran. Low-rank matrix factorization for deep neural network training with high- sigh--imensional output targets. In 2013 IEEE International Conference Acoustics, Speech and Signal Processing, pages 6655-6659, 2013.
[67] D. Savostianova, E. Zangrando, G. Ceruti, and F. Tudisco. Robust low-rank training via approximate orthonormal constraints. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
[68] S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems, 2022.
[69] D. Scieur, Roulet, F. Bach, and A. d' Aspremont. Integration methods and accelerated optimization algorithms. In Advances In Neural Information Processing Systems, 2017.
14

---

[70] P. Sharma, J. T. Ash, and D. Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In International Conference on Learning Representations (ICLR), 2024.
[71] D. Song, P. Zhang, and F. Li. Speeding up deep convolutional neural networks based on tucker-cp decomposition. In Proceedings of the 2020 Sth International Conference on Machine Learning Technologies, ICMLT 2020, page 56-61, New York, NY, USA, 2020. Association for Computing Machinery.
[72] E. Stoudenmire and D. J. Schwab. Supervised learning with tensor networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[73] K. Ullrich, E. Meeds, and M. Welling. Soft weight-sharing for neural network compression. In International Conference on Learning Representations (ICLR), 2017.
[74] B. Vandereycken. Low-rank matrix completion by riemannian optimization. SIAM Journal on Optimization, 23(2): 2013.
[75] V. Vanhoucke, A. Senior, and M. Z. Mao. Improving the speed of neural networks cpus, In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 2011.
[76] G. Venkatesh, E. Nurvitadhi, and D. Marr. Accelerating deep convolutional networks using low-precision and sparsity. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2861-2865. IEEF, 2017.
[77] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[78] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representations, 2020.
[79] H. Wang. S. Agarwal, and D. Papailiopoulos. Pufferfish: Communication-fficciii models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
[80] Y. Wang, Y. Lin, X. Zeng, and G. Zhang. Multilora: Democratizing lora for better multi-task learning, 2023.
[81] Z. Wang, C. Li, and Wang. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14913-14922, 2021.
[82] G. Wanner and E. Hairer. Solving ordinary differential equations II, volume 375. Springer Berlin Heidelberg New York, 1996.
[83] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4820 2016.
[84] H. Yang, M. Tang, W. Wen, F. Yan. D. Hu, H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 678-679 2020.
[85] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023.
[86] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection, 2024.
15

---

A Impact statement
This paper presents work whose main goal is to reduce the training costs of tensor-based architecture, while maintaining mathematically sound guarantees of performance in terms of convergence and approximation. As in the majority deep learning research, there are potential societal consequences of our work, none of which we feel must be specifically highlighted here. A positive societal impact the reduces carbon emissions by more efficient neural networkt training and inference. Regarding ethical aspects, feel nothing has to be added.
B Algorithm for tensor taping based on Theorem 2.1
For the sake of completeness, gradients of the Tucker factors as obtained by Theorem 2.1
provide the algorithmic description of the method to generate
The training algorithm for tensor-valued neural network layers in Tucker format is presented in Algorithm 2
Through the lens of computational efficiency, the main difference to Algorithm iis the
following:
Each time we back- prokaaatp through a convolutional layer W C x Us we form the new variable K; = U.S; as in Theorem 2.1 we integrate the ODE in (5) from (; = K; to (A(A), 0, and then update the factors U; by forming orthonormal basis of the range of (A)). This
strategy directly follows from Theorem 2.1
Similar to Algorithm T we implement the orthonormalization step via the QR factorization while we perform the integration of the gradient flow via stochastic gradient descent with momentum and learning rate X, which coincides with stable two-step linear multistep integration method [69]. Once all the factors U2 are updated, we back pack-propagate the core term by integrating the equation for C' in (5), using the same approach.
Similar to Algorithm [ the Tucker rank of the new kernel can be adaptively learned with a key basis-augmentation trick. The implementation for Algorithm 2 works as follows: Each time we backpropagate K; E "T,, we form augmented basis K; by appending the previous U; the new (A), K, [K]] We compute an orthonomal basis UTew e Rm.x2r; for K, and we form the augmented 2r, X ... x 2ra core C = C x (U?ee) U. We then backpropagate the core C integrating 5 starting from C(0)= C. Finally, we perform rank adjustment step by computing the best Tucker approximation of C' to a relative tolerance 0. This step corresponds solving the following optimization (rounding) task:
TU;
Find Ò E MMg2 of smallest rank p' (ri...., r'd such that -
= +||²||
where p (ris (ris..., ra) and Ms2p denotes the set of tensors with component-wise Tucker rank lower than 2p. In practice, this is done by unfolding the tensor along each mode and computing a truncated SVD of the resulting matrix. The tensor Ć E MM is then further decomposed in its Tucker decomposition yielding a factorization C = C' x U MMp' The parameter
responsible for the compression rate of the method, as larger values of 7 yield smaller Tucker ranks and thus higher parameter reduction. To conclude, the computed U/ E R2axxer with r S 2r; are then pulled back to the initial dimension of the filter by setting U. U.!e"U Ruixr!, and the new core tensor C' is then assigned C".
This implementation shares the robust error bound of Algorithm However it comes at an increased computational cost due to d necessary evaluations of the network and gradient tape, where first d are due to the basis updates K, and the last for the coefficient update S.
C Additional experiments
C.1 Additional experiments for ResNet18 on Tmag-em
Table displays the compression to test accuracy results for ResNet18 on Tiny ImageNet as supplement to the results in section 4.1
16

---

Algorithm 2: TDLRT: Standard Dynamical Low-Rank Training of convolutions in Tucker format.
Input: Initial low-rank factors C X Vaa U; n, ri;
adaptive: Boolean flag that decides whether or not to dynamically update the ranks; T: singular value threshold for the adaptive procedure.
for each mode do
Q f- QR decomposition of MC),
K, - U,S;
K; descent step; direction V (x,[(Tee:(Q.)) Xjzi Uj X K2); starting point K; if addatiiee then
/* Basis augmentation step */
K; [K; U:]
U orthonormal basis for the range of K;
C - C x (U!T*)
C descent step; direction VeL(C x U starting point Ò
10 if adaptive then
/* Rank adjustment step */
(C, UI..... Ua) +- Tucker decomposition of up to relative error r
Uj + U.ee Uj for .,...,d
13 else
14
U, + U. for i 1, I....,d
Table 3: Tiny-imagenet benchmark with ResNet18, TDLRT outperforms standard Tucker factoriza- tion terms of the compresioon-O--cccurayy ratio.
test acc [%]
[%]
Baseline
51.1
0.0
TDLRT, 0.02
50.90
83.95
TDLRT, 0.06
49.32
92.12
Tucker-factorized
49.66
83.66
C.2 Additional experiments VGG16 on Cifar10
The rank evolution over the optimization steps of VGG16 on Cifar10 are displayed in Figure The color gradients indicate the position of the respective tensor basis in the network, where lighter green denotes bases near the input and darker green denotes bases near the output layer of VGG16. Higher singular value cutoff tolerance results faster rank decay; however, across different choices of tau, a monotonous decrease in the ranks to a steady state is observable.
The proposed method Algorithm T allows we initialize the network layers with full rank. The decay the layers' ranks over time is typical for Alg1 and, indeed observed for other architectures well. This is aligned with theoretical [5] and empirical [88] findings stating that neural networks trained with SGD exhibit low-rank structure. The results of Fig 3 indicate that Alg. can identify this low-rank manifold during training.
choose any Tucker ranks at initialization In Figure 3
C.3 Additional experiments for AlexNet on Cifar 10
Tab contains the Tucker ranks of Alexnet compressed with TDLRT supplement to the results presented in section4.1 All the test cases were run with the rank adaptive version of the integrator.
For Cifar10, standard random crop and random flip are used for data augmentation at training time. methods are trained for 70 epochs using a batch size of 128. All methods are trained using the SGD optimizer, with starting learning rate of 5 x 10-2 with scheduler that reduces it by factor of whenever validation loss reaches plateau. Polyak momentum was for all layers but batch normalizations, which was set to 0.9.
17

---

500
500
500
basis ranks
basis ranks
333
333
333
166
166
100
200
300
200 optimization iter
100
300
100
200
300
optimization
opamization iter
0.1
0.08
0.05
Figure 3: Rank evolution of the Tucker bases over optimization steps of all rank adaptive layers of VGG16 for Cifarl0 using Algorithm!1 [ The lighter color indicates ranks of bases of deeper layers of the network. A higher singular value cutoff threshold results in faster rank decay and smaller steady-state ranks, leading to a potentially higher compression rate.
Table Reproduction of the results of Alexnet on Cifar10. The ranks reported refer the Tucker ranks of each convolutional layer.
test acc. [%]
layers' ranks
test compression rate [%]
Baseline
79.63
[64, 3, 3,
0.0
[192, 64, 3,3
384, 192,
256, 384, 3,3
256,, 256,
TDLRT = 0.6
76.26
[25, 3, 3,
74
[76, 25,
[53, 3,3
153,3,3,
102, 102,
TDLRT T = 0.7
73.08
[19, 3, 3.3
83.5
19,3.3..
115, 57,
76, 115,
176, 76,,7,3,3
For more complicated datasets and architectures like Cifar10 on VGG16 and Alexnet, results in Tab[] Tig[[show that our proposal consistently gets better results than the methods under comparison parity of compression.
D Proof of Theorem 2.1
Theorem. Let W = C x U, E MM be such that (3) holds. Let (C), = =Q.ST be the QR decomposition of Mat,(C) and let K; = U.S. Then,
K, - V K. C TTen; j U, X K;)
and č=-VoL(C x U,)
(8)
where Ten; denotes "tensorization along mode i", i.c. the inverse reshaping operation of Mat,.
Proof. The proof follows the path first suggested in |52 $4], i.e., the quantity V, defined next is
frozen in time: 1/2 0. A detailed derivation of the matrix tensor equations for K; and C' is beyond the scope of the present work and is provided 40 53
12
We begin recalling the evolution equations for the factors of the projected gradient flow 3
f Ü = =- -U,,,) Mat. (wW((W) xsts UJ) Mat,(C)t, 1.....d
è =-Vw<((W) x==1U; U,.
18

---

where i denotes the pseudoinverse. We assume that the tensor W admits a differentiable Tucker tensor representation, i.c., (t) C(t) x U:(t) Mp for t [0, X] satisfying the associated
gradient-flow tensor differential equation
W(1) V (()
For sake brevity, the parameter going to be omitted. Next, we perform a QR-factorization
Mat, (C) = QiS!
Thus, observe that the matrix (W) admits SVD-like decomposition follows
Mat; (W) = U.S.V, with V, = Qi
where the matrix Qi possess othornormal columns, i.e., Q.Q: = We introduce the quantity
K₁ U.S, where S = Ma=M)t((C)Q...
By construction, we observe that
S, = U. MW),
The tiny matrix S satisfies the following differential equation
$ = Ü.T Mat, (W),, + U.T Mat. (W), + U.T (W),
= U,TU.) SVV, V. + UUT Mat, (WVV, + U. (a),
UU Mat
The first null identity follows from the gauge condition on U,. The second null identity follows by the initial assumption V; = 0. Therefore, the matrix K; satisfies the differential equation
K. (UiS.) Ú,S, + U.S, =
[T- =-(T U,U,) (Vo j U, Mat, - U.U. (Vw
(U,U, - 1) Mat, Vw jztt U/ ) Q0 U.U, VVt
(9)
(U.U, 1) Mat. (W (8 U,)Q: - U.U, (V
= -U.UT - Mat, (V C)V, - U.U,T (V C)V
= MVw
To conclude, we set Z K2V, and let W Ten: (KVVT) Ten: (Z). We obtain
V L(W) V C(enn (2) = V Z(Tnn (Z)VV,
where remind that V, V; I. Hence
Mal.(Vw ((W)) = V (Teene (2)) = VK, C(W)
(10)
The first diffffrrrenttal equation is then obtained combining and
k; = Mat; w V)V; -v (Ten,
(11)
The right-hand side can be further reduced using standard tensorization formulas [4
Ten, (K Ten, (QT) i7i
U, x K..
The second differential equations follows by observing that VWL = v,Lx, Us + Σ C Cx Xiti U..
19

---

The tensor C' satisfies the differential equation
C' VwL(W) x, U,U
VCC(W) x; Us UUUxXUU; U.
= -VoC(W) x, U,T.U
= -VecC(C x; U:)
where the extra terms disappear due to the imposed gauge conditions U, Ü, 0.
E Proof of Corollary 2.2
Corollary E.1. Let CC xx Uj M, be such that (6) holds. Let Mati(C)) = QiST be the QR decomposition of Ma), and K; = U.S.. Then, span ([u,, Ki - span ([U,, Vu, ((W)))
(12)
Proof. From Eq. 99) of the proof in Theorem 2.1 it is apparent, that
K; U.S, + U.S; Mat, - [Vw[(Ten;(W)))
(13)
Moreover, we observe that using the chain rule,
Vu, C = Mat, cut(VV
(V
Then, with 13) we have
VE,C Vat = KKS.T
Using full-rankness of S; concludes the proof.
F Proofs of descent and approximation theorems
Theorem. Let (X) C' xd j=1 U3 be the Tucker low-rank tensor obtained after one training iteration using Algorithm 2 and let (0)) be the previous point. Then, for small enough learning rate X, it holds Lw (W(() & Cw (0) - aA + Br, where a, , are constants independent of X and т, and where Lw denotes the loss as function of only W.
Proof. Let (t) C(t) x, Ü,J Here, W(t) and C(t) denote the augmented solutions for E [o arising from the intermediate steps of the TDLRT Algorithm 2 We observe that
(() = ((00)))(C<(((())))) (V)
(VC(W())) ((t) x; 011
=(VC(W(() x, D.T.T.C ,²(1))
= (VVC(((I) x, U.!.T, LV)(W()) x, 0 = = -VVC((( x &
we define a == minogrsi (WW((()))) x; 0 ITP. it follows that for E Ao d
L(W (1)) - a
(14)
dt
Integrating 14 from / -= 0 until / = X, we obtain
((W())) S ((W()) aA.
Because the augmented subspaces Ú; contain by construction the range and co-range of the initial value, we have that (0))= = W()). Furthermore, the truncation is such that |[W(A) W(X) s Therefore
C(W())) & ((W())) + Br
where 8 = maxosrsi [V | + (1 (1)
20

---

Lemma F.1. The following estimate holds
[W (()) Xj U,(A)U(()) 1
where the hidden constants depend only on L1 and L2.
Proof. It has been shown in
68 Appendix] that there exists constant x O(h(h + <)) such that
|U (at - (Wat((W()))I S vj=1.....
where the value has been shown to depend only on the constants L1, L2 and X. The proof of the Lemma follows a recursive constructive argument
(W(A) - ((A) ((),
< (A) - W(AA) X00)), (I(()))))))))))))))))))))) UUAUU,A W(A) XAAA,, ([W(A) W(A) x ()(N),,))T (( Xa (a(A)U - -W(A)) U2QU0(N)
& [[W(A) W(A) (AA),) (W(())))))))))))))))))))))) xa Ua(A)ua(A) - -W(III
[[W(A) ((A) X5-1 (A)))
The conclusion is obtained iterating the provided argument.
Theorem F.2. For an integer k, let = kλ, and let (t) be the full convolutional kernel, solution of
② at time Let Ctt), {U,(t)s be the Tucker core and factors computed after k training steps with where the mee integration from O to À is done exactly. Finally, assume that for any in a neighborhood of W (t), the gradient flow Lw (Y) "e-close" to Mp. Then.
Algorithm 2
[W(11) C(I) U,(OI S 6ι + 02t + C3T/A
where the constants C12 02 and e3 depend only on L1 and L2.
Proof. First, provide bound for the local error, i.e., the error obtained after training epoch. we apply Lemma E. we obtain that
[W(A) C(A) (A(IIII
[|[(()) - W(A) X5- 1 ( BONUN CCA)) U,()II
6 (W x - -CA) Us(1l
< + (I(()) - U,(A) -(AI)..
suffices to study the latter term. For E [0, A]. we define the quantity
E(t) == ((t) =
satisfics the differential initial value problem
Č - CWW x U,(A), -()) X5-1 T,(A)T.
The term W() can be written as a perturbation of Č
w(I) Ō x²==1 U₁(A) R(),,
where
R(t) = ((1) ((t) x U,(A)U(()TT.
Then, we observe that
[W((t) - W(AIl < I (I) / / S CGA.
The remainder can be estimated follows
[[(()]] < [(t)) R(J]++)[() S 2L 2 20
21

---

Furthermore, the full gradient can be re-written as
Vw = Vw ((((t) x U2(A) + R(t) Vw-(C(() x5_ (2(A)) D(t),
where the defect (tt is defined
D(t) = V C(C(t x U2(A) + R(t)) Vw U3(1)
Because of the Lipschitiz assumption, have that
| - L2]| & 2L2(L1)+0)
Next, we compare the two differential equations
Č(() = - ((t) - VwC(((( X5= U,) x1)x)==11 U
Vw C(Ō()) x U, x UJ + D(t),
where C(0) = (()). by construction. The solution (?(A) of the second censor-differential equation coincides with the solution of the last training step of the TuckerDLRT algorithm. The first differential equation has been constructed such that solution is C(A) - ((A) x, UJ. Therefore, the study of the local error follows by direct application of the Gronwall inequality
[C(A)) - Č(A) S exp(C2A)2L2(11> O)A
To conclude, the global error in the training epochs follows by using the Lipschitz continuity of the gradient flow: We move from the local error time the global error in time by a standard ODEs argument of Lady Windermere's fan $1I.3].
G Proof of stochastic convergence
In this section, we provide the details the proof of convergence stationary points in the stochastic setting, Thcorem 3.2 The proof extends the approach of [1] to the tensor case and relaxes some of the assumptions there made on the matrix case, while following the same overall structure.
Theorem G.1 (Convergence). Let (t) be the weight tensor after E N iterations of Algorithm 1 before the rank truncation step, and (t) as the one obtained after the rank truncation. Assuming that
Algorithm
iis implemented using SGD as the descent method.
The loss function assumed to positive, locally bounded, and differentiable with a Lipschitz gradient.
The learning rule sequence d satisfies the Robbins-Monru conditions, i.e.
Σ~=+0 Σ x +00
The spectral distribution stabilizes fast enough over time, i.e.
Σ E (i(()) (()I]] +00
(15)
220
The projected stochastic gradient has a controlled drift, namely
E [IVLCW(( 1)) x, Pi i] () for some 14, 22.220 0,
(16)
where Py = UUT the orthogonal projection onto the range of U, and [+] (t), (U,(1) denotes the conditional expectation. Then the following convergence condi- tion holds
lim inf E [[VCC(W( (-000
-1)) X3 Puş(t- =0
22

---

The convergence proof of Theorem 3.2 is based on the following technical lemmas.
Lemma G.2. Let L be a differentiable loss function, and assume that its gradient V (WW is one- sided Lipschitz continuous with constant L2. Then, for any W and W the following inequality holds
L(W) - L(W") + (VL(W"), W W") La WIW - W 2
Proof. By using Bauchy-SShwaane inequality, we have that
L(W) L(W") + (-' d dt c( W t(W
t(W- - ((W-")) de
(((W-w)). = L(W") - (VL(W"), W - -W'> - L.' (VC(W") - VC((W + +t(W- +((W-()) w)). .W. -W) dt
s <(W") - (VL(W"), ) W - -W" T 15--wwp -w'l? ': ['a
= C(W") - (VC(W"), W - w") L I - w/12.
2
Lemma G.3. Let ú,,1 [ be given basis set with orthonormal columns, C denote the loss function computed on the whole dataset, and LB denote the loss calculated on a batch B. Then, for any j (1....,dl dd and W, holds that
E (VL(W), VLB(W) x143" PUço X5- Poye,) 20
Proof. We introduce first the function
xú) = (0):=(VV(N)) (VC(W) X5æj* Pú,o X3- Pe. VLB(W) Xztj" Puzo X3 Pi
Let m be the the infimum on the set U as defined below
m
inf (())
with u = UU E x rran(()) S }g....
Deu
Because the term VLW) can be decomposed into sum of VLB(W) Xj$j* Púç+,o Xj+ PU and its orthogonal component, we observe that the infimum satisfies
m S (U (VC(W), VLB(W) x x5t5* Pu,, o X, Pus.)
Hence, by taking the expectation on both sides, we can conclude that
[[ (VL(W), (L)(W) Xjt5" PUço x5 Pu.x)
inf E [o(()]]
2 inf |[VC(W) Xit1" Puço Xj+ 1
Beu
Deu
The conclusion follows by observing that the most right term in the inequality positive.
With the technical lemmas established above,
now in the position to prove the theorem 3.2
Proof. (Thereom 3.2 To simplify the notation, we will denote by E.[-] the conditional expectation E.[. E[- ( ]. We first remind the reader of properties of the conditional expectation. Specifically, for any deterministic function vi and any random variable X, we have that
Eg (W (t)) = (W 1)),
T E = [E
We will begin by examining an upper bound for the one-step drift of 2 We denote by W(t) = Č(t) xj (t) the weight tensor before truncation at step E N. As per the assumption, the optimization the C-step is defined using an SGD update. Therefore, we have
= (10-:://0DDD-NNN-------00-0-----
23

---

Hence
W(t) C(t) 0t)
- -
1))) x, Po,00
where Pu = UU. is the orthogonal projector onto the range of arbitrary matrix U. Applying G.2 we have that
E. ((W())) - L(W(t- 1>)]
+ 2EL2 k (W 1)) X3 Posce 2
-xx,E (VC(W((- 1)), VL 1)) x, P (0)
We notice that for the augmented basis (t) When we expand L3 (W 1)) x, Pi)()) using its sum representation and apply Lemma G.3 to the mixed terms, we obtain
it holds Pūç(0 (t) + Po,
Es ((W()))))((
S -, (VC(W( - 1)), (t x, Po,(e-1)) + X2 E, ( 1)) x, Po,co
2
12L22 EE [IVL 1)) x, Po,,c0117)
--N(VC(W(( 1)), VC(W(( - 1)) Xj (t)((-1)) +
= (1 1)) X3 PU 12+ 23L2 Es ||VLB(W - 1)) x, 12
2
(17)
The locally bounded loss computed on the truncated approximation ((t) is bounded via
(t) s (Z
((())-(()) - C(W (WW (t)l
(18)
By combining equations 17 and 18). we arrive at the following bound
E. ((W())) - C( - 17)]
22 E
$ -e VL - 1)) xj Poça-nnl + 2
[V 1)) x, P2 + +CE (Ie((()))))))))))){>) - W(0]]
(19)
Following assumption 16 we have
E. [(W(()) C(W(()) 1n}]
S <- 1)) X3 PU;(0-1) 12+ X2L2 (u (w+W]V(((((t-1))
1)) x, Po,(a-)11P)
+ CE, - [(OI
= =-A((1 1 { 1)) Xj P(),(t-))1
+ CE. ( w
2
S < 1)) xj (Uç(t-)) 12 La + CE2 [[I - W(el].
where we assume that d S 2/V. Finally, by taking the expectation on both sides, we obtain
A2 +CE ((())
E [[(W())) < -E (I)
2
By summing the last equation over 1,....? we get
L(W(O)) < E ((W()) - ((W(()) <
T
X (V - 1)) x, Po,(e-11113]
+
Σ
x +
+ > E [I -((e))
2
t=1
24

---

By rearranging the terms of the latter equation and by sending T - to0, we finally obtain
+
X
(L
- 1)) xj Poç(o-u11)
()) Lajf
+000
R [Iw(1) W(N)] < +00
The conclusion follows by the Bobbins-Mome conditions, i.c.
lim inf EE [VV - 1)) x, Po,ce-1111 = =0
7-000
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See statements in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the analysis is discussed after the corresponding theorem statements and in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
All assumptions are stated in the corresponding theorems, and their proofs are linked and stated in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The numerical examples are fully described in Section 4
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
The code is given in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/p.ps..ccppullc//guui /
wwt.Suubmisiioooiiioooooooio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The numerical examples, the neural network training parameters for all computer vision tests, and the auttaaaummetttionn for the used datasets are described in are described in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification; The test case statistics are explained in the corresponding sections in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide an overview of the used compute resources Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://p////.prrc..ooo/oo//////a
Answer: [Yes]
Justification: The research is fundamental mathematical research. We do not see how it violates any aspect of the code ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impact is discussed Appendix A Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We use open datasets and very fundamental benchmarks, so we think our work does not has risk for missuses.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We neither crowdsource experiments nor perform researchw ith human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "hzV4JHJNqk", "reviewer": "Reviewer_TTwz", "summary": "Reducing the size of neural networks is an important problem for reducing the cost, memory usage, and even inference time. Many works focus on reducing the size after the training phase and use techniques such as sparsification and quantization. This paper, on the other hand, focuses on reducing the size by representing the tensor corresponding to the layers of the network as Tucker decompositions. Perhaps the more important aspect is that the paper considers dynamically changing the Tucker rank while training the model. In other words, the size reduction is simultaneous with training and not after the training is performed.\n\nIt is argued that due to instability in the gradients, it is required to adopt a geometry-aware training strategy. Essentially, the strategy is to do HOOI based on the gradient of factor matrices. Then, compute a new tensor based on the new factor matrices and update the core tensor based on the gradient of this new tensor. Therefore, for each iteration, two passes are required to update the components: one pass to compute the gradients of factor matrices and one pass to compute the gradient of the core tensor.\n\nThe paper presents theoretical results about the convergence and reduction of loss. In addition, it presents a sizable empirical study that shows favorable results for the proposed approach. The approach outperforms a variety of factorization and pruning methods in terms test accuracy and compression rate.\n\nThe paper is generally well-written and provides an appropriate method for a very important problem.\n\nI think a relevant paper to discuss is [1], which gives an algorithm to compute the approximately optimal Tucker rank when a size constraint on the size of the Tucker decomposition is given. This could replace the approach based on the tolerance parameter $\\tau$ when a hard size constraint is given. It would be interesting to investigate the interaction between that algorithm and the approach presented in this paper.\n\n[1] Ghadiri, Mehrdad, Matthew Fahrbach, Gang Fu, and Vahab Mirrokni. \"Approximately optimal core shapes for tensor decompositions.\" In International Conference on Machine Learning, pp. 11237-11254. PMLR, 2023.", "questions": "The results show that Tucker decomposition works better than CP and tensor-train. Is there any intuition for why this is the case? Do you expect Tucker to be better than any other tensor network, or could decompositions like hierarchical Tucker do better? Hierarchical Tucker could be preferable because of the smaller number of required parameters.", "limitations": "I don't see any direct potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well-written and provides an appropriate method for a very important problem.", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and appreciation of the work. Below we provide our response to the raised questions.\n\n\n**Q1.** We appreciate this insightful question. Selecting the appropriate tensor structure a-priori is indeed challenging, as it depends heavily on the specific tensor structures in the neural network and the data used. As shown in Table 1 of our main paper, when using standard direct training of the factors, there is no definitive \"winner\" among Tucker-factorized, Matrix-factorized, TT-factorized, and CP-factorized approaches across the three test problems. However, when employing our proposed geometry-aware TDRLT scheme for training, we observe notably improved performance.\n\nTherefore, we believe that Tucker's superior performance is primarily due to the training method rather than the decomposition choice alone. While extending this training strategy to CP is challenging due to the absence of Riemannian geometry, we believe that further research into extending the proposed geometry-aware training to TT, HT, and other tree tensor network structures would be highly relevant.\n\n\nFor what concerns the comment in the \"Summary\" section, we thank the reviewer for bringing this paper to our attention. We agree that the algorithm presented in [1] offers an interesting alternative approach to consider. In particular, their method is well-suited for scenarios where there is a fixed memory budget, such as training on resource-constrained edge devices. In such cases, the objective shifts from \"finding the best accuracy-compression trade-off\", which is the focus of our method, to \"finding the best-performing Tucker decomposition given memory constraints\", where the approach from [1] could be highly beneficial.\n\nTo explore this aspect further, it would be useful to extend the method in [1], which currently addresses a single Tucker tensor, to an approach that optimizes the shapes of all tensor-valued layers within the neural network given a global network-wise memory constraint. This is an intriguing research direction that could significantly enhance the applicability of tensor decompositions in constrained environments.\n\nWe will certainly include a discussion of this paper in our revised manuscript, highlighting it as a potential alternative approach to our proposed method. This addition will provide readers with a broader perspective on the available techniques and their respective advantages and limitations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vS8CIy4y9R", "reviewer": "Reviewer_SSBR", "summary": "The authors study the training of layer factorization models to reduce the number of parameters in deep neural networks. They propose a geometric-aware rank-adaptive training strategy to avoid requiring prior knowledge of ranks and the sensitivity to the weight initializations. Their theoretical results show convergence and approximation error guarantees for the method.", "questions": "Is it possible to conduct experiments on larger models or different architectures (like transformers, even a small one can help) to strengthen the empirical evaluation?", "limitations": "Yes, the authors discuss about the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method is quite sensible and is accompanied by good theoretical guarantees.", "weaknesses": "The empirical evaluation is slightly weak. While the results could convince me it is better than existing methods (as the proposed method is also quite sensible), from the scale of the model used, it is hard to judge whether it could provide good enough performances on larger models, when model compression during training is more needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback, below we provide our response to the raised weaknesses and questions.\n\n**Q1**\nThank you for your feedback regarding the empirical evaluation. In response to your suggestion, we have conducted several new experiments to strengthen our empirical evaluation.\n\nSpecifically, we have focused on parameter-efficient fine-tuning using low-rank adapters (LoRA), as suggested by reviewer vGH1. This approach allows us to present additional experimental evidence that complements our previous results. Fine-tuning, rather than compressing a model from scratch during training, enables us to experiment with larger networks while remaining within our time and resource constraints.\n\nIn particular, we have tested the proposed training model on larger architectures such as DebertaV3 on Glue benchmark and Stable Diffusion (v1.4) dreambooth. In DebertaV3, we applied our method to matrix layers by interpreting matrices as order-2 tensors. For Stable Diffusion we applied our method to tensor layers in the Unet, while standard LoRA convolution works essentially by reshaping the tensor into a matrix.\n\nThese new experiments provide further validation of our method’s performance and demonstrate its applicability to larger models and diverse architectures. We believe these results strengthen the empirical evaluation and could address your concerns about scalability.\n\n| GLUE test case| LoRA - rank 8, 1.33M params | Ours - 0.9 M params |\n|-----------------------|---------------|-----------------------------|\n| CoLa (Matt. Corr)     | $0.6759$      | $0.7065$                    |\n| MRPC (acc)            | $0.8971$      | $0.9052$                    |\n| QQP (acc)             | $0.9131$      | $ 0.9215$                   |\n| RTE (acc)             | $0.8535$      | $0.8713$                    |\n| SST2 (acc)            | $0.9484$      | $0.9594$                    |\n\n\n Stable Diffusion  | loss | \\# trainable parameters \n----------------------|---------------|----------------------------------\n LoRA ($r = 8$)       | $0.260$       | $5$ M                            \n LoRA ($r = 5$)       | $0.269$       | $3$ M                            \n LoRA ($r = 3$)       | $0.274$       | $1.8$ M                          \n ours ($\\tau = 0.02$) | $0.2635$    | $1.8$ M                          \n ours ($\\tau = 0.1$)  | $0.272$       | $1.5$ M", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3PvJmvyAg", "reviewer": "Reviewer_jjrQ", "summary": "The authors present a novel algorithm for training neural network layers using Tucker tensor decomposition. The approach addresses common issues with layer factorisation, including the need for an initial warm-up phase and sensitivity to parameter initialisation. Th method dynamically updates the ranks during training. The authors provide theoretical guarantees on loss descent, convergence, and approximation to the full model, supported by experimental results showing high compression rates and performance comparable to or better than baseline and alternative strategies.", "questions": "- The proposed method appears to outperform the full representation at 96% compression (Figure 1). Can the authors provide an in-depth discussion and analysis of this finding? What factors contribute directly to this performance, and does it align with theoretical expectations?\n- Can the authors include data points for the proposed method within the full compression range for a direct comparison with other methods?", "limitations": "The paper could benefit from a more detailed discussion on the limitations of the proposed method in different training scenarios and potential strategies to mitigate these limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- A strong motivation as there is a clear need for further efficiency improvements\n- Introduces a novel rank-adaptive geometry-aware training method that dynamically updates ranks during training\n- Proposed to overcome the sensitivity to parameter initialisation and the need for a full-model warm-up phase\n- Thorough theoretical analysis, including guarantees on loss descent, convergence, and approximation", "weaknesses": "- Claims do not match the results. The abstract says \"our training proposal proves to be optimal in locally approximating the original unfactorized dynamics\" and while there are guarantees, they are not proven to be optimal.\n- Not very clear in many parts. In particular, Section 2.1 is difficult to follow.\n- Results seem incomplete. For example, figure 1 does not show compression below 60% and for the proposed method is only shown for 93+% in Figure 1C while the other methods are shown for 60-93%. I could not find any reason for this lack of direct comparison and missing data points. \n- According to the plots, the proposed method outperforms the full representation at 96% compression. This is a very surprising finding that requires an in-depth discussion, which is lacking.\n- Table 1: The authors say that \"TDLRT outperforms the factorization-based and the pruning-based baselines\" and bold their method for Alexnet c.r. and Resnet test acc. but according to the same table, baselines actually outperform the proposed method in those metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for their feedback. Overall, the reviewer highlights key positive aspects of our work, such as the importance of the topic, the novelty of the approach, and the thorough theoretical analysis. The main weaknesses noted are centered on the need for additional runs and a more detailed discussion of results. We are, however, surprised by the perceived impact of these weaknesses, especially in light of the significant strengths identified. The reviewer mentions that our work is \"unclear in many parts\" and that \"Section 2.1 is hard to follow\". However, without further details, it is challenging for us to address these points, particularly since clarity has not been a concern for the other reviewers.\nIn the following, we address all the identified weaknesses (Ws), as we believe several points merit further clarification. We also answer the raised questions (Qs).\n\n**W1** Thank you for pointing this out. In that sentence of the abstract, our intention was to refer to the optimality of the Dirac-Frenkel variational principle to approximate the dynamics along with the supporting theoretical results of approximation and convergence. However, this statement is not crucial and can be misinterpreted, therefore we will remove it from the abstract.\n\n**W2**\nWe recognize that certain sections of the paper, particularly Section 2.1, contain technical details that may require a background in Riemannian optimization and dynamical model-order reduction theory. We have made significant efforts to make this section as clear and accessible as possible, but we understand that some concepts may still be difficult to follow. However, a conference paper is not the ideal format for providing an extensive introduction to these topics.\n\nWe would appreciate further clarification on the specific parts of Section 2.1 or other sections that you found unclear. This will help us address your concerns more effectively and improve the paper's readability.\nWe aim to clarify the main ideas of Section 2.1:\n\n- Our goal is to derive a low-rank gradient flow that closely approximates the standard full gradient flow. To achieve this, we project the gradient flow onto the tangent bundle as described in Equation (3), which represents the standard gradient flow of Riemannian optimization. \n- However, this equation forms a stiff system that is challenging to solve directly with a discretized scheme, necessitating a small learning rate.\n- By introducing a reparametrization of the weights in Theorem 2.1, we obtain new evolution equations that are well-posed and can be solved with larger learning rates. Despite this improvement, the resulting method remains inefficient as it requires d+1 gradient evaluations.\n- In Corollary 2.2, we demonstrate that the basis computation can be significantly simplified, leading to Algorithm 1, which requires only 2 gradient evaluations instead of d+1.\n\nWe point out that we have already attempted to convey this structure and our reasoning at the beginning of Section 2. Please let us know if this explanation clarifies your questions, and if not, which specific parts remain unclear. \n\n\n**W3** and **Q2** The reason our method is sometimes only shown for large compression values is that one key feature of our method is that the compression rate is somewhat determined by the method itself, rather than being a user-defined input, as with the fixed-rank baselines. Nonetheless, we have adjusted the truncation tolerance to obtain smaller compression rates and have included additional data points in the attached rebuttal PDF file. We emphasize that methods at compression rates below 60\\% are generally less relevant, so we have focused on the range of large compression rates to maintain the focus on the relevant range of compressions and highlight the strengths of our method. However, if the reviewer considers it valuable, we can include additional runs for smaller compression rates in the appendix.\n\n**W4** and **Q1**\nThank you for your remark. We emphasize that this phenomenon, where networks with a smaller number of parameters outperform their full baselines, is well-documented in the literature, see e.g. [1,2]. One explanation for this behavior is that neural networks are often overparameterized, and parameter reduction or compression can have a regularizing effect enhancing generalization.\n\nNote that in our comparisons to the baseline, we use the same hyperparameters for compression methods and the full-rank baseline. With this approach on VGG16 we indeed obtain better accuracy than the baseline. However, given your comments, we decided to see if we could find better hyperparameters. This gives us results where the baseline outperforms compression methods. We decided to add these results since the overall accuracy is improved this way for the baseline and for all the compression methods.\n\n[1] The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2018\n[2] Snip: Single-shot network pruning based on connection sensitivity, 2018\n\n**W5**\nThank you for pointing out this oversight. We apologize for the confusion.\n\nUpon reviewing Table 1, we realized that an error occurred when bolding the results for AlexNet's compression rate (c.r.) and ResNet's test accuracy. We were updating the table with new competitor results shortly before submission and inadvertently left the bold formatting incorrect. We will correct this in our revised manuscript to accurately reflect the performance of each method.\n\nRegarding the specific metrics, it is true that while the TT-factorized method achieves a higher compression rate for AlexNet, its accuracy is lower compared to TDLRT, although it still outperforms other methods significantly. For ResNet18, while the Tucker RGD method achieves a slightly higher accuracy (0.04\\%), it does so at the cost of a reduced compression rate.\n\nWe will ensure that the table accurately represents these findings and provide a more detailed discussion in the text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZjzw7qUsC", "reviewer": "Reviewer_vGH1", "summary": "This paper extends the dynamic low-rank neural network training (DLRT) method to the rank-adaptive Tucker tensor format (TDLRT). The proposed reparameterization method greatly reduces the computational complexity and numerical instability of the projected gradient descent. Under certain conditions, TDLRT with SGD converges to a stationary point in expectation, where the tensor found by TDLRT provably approximates the full mode. The experimental results show that TDLRT converges faster with less variance and better accuracy-compression rate tradeoff than other factorization-based and pruning-based methods.", "questions": "- How does the one-step integration work? Does it not add significant computational overhead?\n- Are the conditions and assumptions in Theorem 3.2 and 3.3 reflect the actual DNN training? Is there any assumption that does not hold in practice?\n- How sensitive is TLDRT to the choice of hyperparameters like $\\tau$, learning rate, weight decay or initialization schemes? Do they require extensive hyperparameter search to find the right ones?\n- Can TDLRT be used for the low-rank adaptation (LoRA) [1] setting? E.g., fine-tuning the convolution kernels of the U-Nets for the diffusion models by TDLRT with very low tensor ranks.\n\n[1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).", "limitations": "- The utilization of QR decomposition might hinder using the proposed method to train a large model on a large dataset, e.g., diffusion model training.\n- The number of parameters is determined after training and indirectly adjusted by a hyperparameter $\\tau$. Since a good strategy of choosing $\\tau$ is not yet proposed, one might need to train the DNN multiple times to obtain a desired accuracy and model size.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written. Most parts are supported by sufficient technical details.\n- The proposed method is an extended version of DLRT. Yet, some unique challenges appear in the Tucker tensor format have addressed with interesting approaches, e.g., the reparameterization method and Corollary 2.2.", "weaknesses": "I have some minor concerns on writing and the experimental result\n\n- More detailed information about one-step integration methods could be provided for the readers who are not familiar with the concept.\n- Some sections assume a certain degree of background knowledge on DLRT, e.g., gauge conditions in Line 612.\n- The evaluation of training time for Tucker Decomposition and TDLRT was conducted on a toy-sized model (LeNet5) and dataset (MNIST). Evaluating training time on a dataset and a model of more practical sizes would provide better understandings of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like the thank the reviewer for the insightful feedback. Below, we provide our responses to the main points raised in the review. \n\n**Q1+W1** Regarding \"How does the one-step integration work? Does it not add significant computational overhead?\". With \"one-step integration'' we mean a single step of a gradient-based optimization method or, equivalently, any classical time-integration method. In this work, we use stochastic gradient descent (SGD) which in the time integration view is the explicit Euler method. However, the use of other methods is certainly possible. This step does not add any overhead compared to the full-rank training and, in fact, is one of the parts in our method that allows us to have a significant reduction of computational costs and memory footprint. Note that for the full-rank training, the one-step integrate function is called repeatedly using the full gradient, whereas we only need to call this method with the gradient for a low-rank factor. We will add a sentence to our manuscript to clarify what we mean by the one-step integration method.\n\n**W2** Thank you for the suggestion which we believe makes a lot of sense. We will add a short description to our manuscript.\n\n**Q2** Regarding \"Are the conditions and assumptions in Theorem 3.2 and 3.3 reflect the actual DNN training?...\". Regarding Theorem 3.3, boundedness and Lipschitz continuity of the gradient are very common assumptions one must make to show most analytic results. Both assumptions are reasonable since one would assume the gradient remains bounded during the optimization and the gradient does not completely change when slightly changing the input. The latter can be achieved using most common activation functions, since it requires a condition on the boundedness of the derivatives. The assumption that the gradient flow remains close to the low-rank manifold is observed empirically, but it is extremely difficult to understand analytically. Numerous experiments have shown that neural networks are usually well-represented by low-rank weights, and some analytic investigations exist for heavily simplified architectures. Regarding Theorem 3.2, the Robbins-Monro conditions are very standard assumptions and can be ensured by the user when picking the learning rate. The stabilization of the spectral distribution over time is something that we observe empirically, though it is hard to show anything on an analytic level. Here, we see that after sufficient iterations of TDLRT, the basis does not change, and most of the change is seen in the core tensor. The drift assumption is meant to play the role of the more standard (but more restrictive) assumption of finite variance, which is common when studying convergence of stochastic methods. \n\n**Q3** This is a good question we should have discussed in our manuscript. First, regarding the tolerance parameter for truncating the core tensor, this is, of course, a hyperparameter that needs to be chosen. We commonly use similar values here in different test cases and we never needed an in-depth hyperparameter search to observe good performance. However, it is hard to say if this holds for all architectures and datasets, and most likely, one must adapt here. However, we wish to point out that a single parameter determines individual ranks and compressions for different modes in each tensor in each layer. Here, other approaches, like LoRA, which you mention later, require the user to choose good ranks for each layer, which is certainly a harder task and requires an intensive parameter search if one wants minimal memory overhead. Regarding parameters like learning rate, weight decay etc., we found that our method is relatively robust with respect to their choice, similar to the full baseline. This is aligned with the theoretical findings of e.g. Thm 2.1 and Thm 3.3. \n\n**Q4 + W3** Thank you for this excellent question. Certainly, our approach fits LoRA excellently, and **we have now provided some results on how TDLRT can be used in such a setting** (we refer to the main rebuttal pdf file). The main question is always if these architectures allow for low-rank weights (or in terms of the assumption of Theorem 3.3, if $\\varepsilon$ small). If the answer is yes, then our approach should yield good results. Moreover, all the theory presented in the manuscript applies to this case.\n\n**L1** Regarding \"The utilization of QR decomposition might hinder using the proposed method to train a large model....'' Please note that the QR decomposition only needs to be computed on a small matrix of dimension $n_i\\times r_i$, leading to computational costs of $O(n_i\\times r_i^2)$. Thus, if the rank is small, the QR decomposition is usually not a limiting factor (and these small ranks are in fact often the case for LoRA style fine-tuning).\n\n**L2** Regarding \"The number of parameters is determined after training and indirectly adjusted by a hyperparameter. Since a good strategy of choosing \nis not yet proposed, one might need to train...\". We agree, development of a method that takes a parameter budget and determines $\\tau$ to obtain the best model for the given budget is a relevant research question. We have been pointed to  [1] by reviewer TTwz for an approximative Tucker decomposition (of the core tensor) under a parameter budget. \nHere the research question is: Given a parameter budget, how to determine the core shapes of the core tensors of **all** low-rank convolutions in the network to best approximate the full-rank model, if the low-rank dynamics, implicitly given by the data, are representable within this budget. \nThis question is as relevant as it is non-trivial, and we would like to refer to future research to properly address it.\n\n[1] Ghadiri, Mehrdad, Matthew Fahrbach, Gang Fu, and Vahab Mirrokni. \"Approximately optimal core shapes for tensor decompositions.\" In International Conference on Machine Learning, pp. 11237-11254. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AHLfDrdW2h", "reviewer": "Reviewer_Rp73", "summary": "This paper proposes a method for neural network model parameter compression. It parameterizes weight tensors in the format of Tucker decomposition, and trains the factors instead of the origin weight tensors afterwards. The method is able to adaptively modify the tensor rank. Authors also provide detailed theoretical analysis including the computational steps, and convergence, approximation and the gradient descent guarantees of the method. Experiments show good performance of the method in terms of image classification accuracies, model compressive rates, and the high running efficiency.", "questions": "What is a standard Tucker decomposition like? Could you provide more details for the comparison of the proposed method and the vanilla standard Tucker decomposition, especially over the space/time complexity comparison, and their convergency analysis?\n\nWhat are the compressed parameters in the neural networks? Are the parameters merely the convolutional kernel? Can the method be applied on the weights of linear layers?\n\nWhat does ``geometry-aware’’ indicates? What kind of geometry is the method aware of? Could you give more explanation?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Writing is easy to follow. Theoretical analysis is self-contained and solid.  Experiments are sufficient to demonstrate the claimed good properties of the method.", "weaknesses": "I did not observe obvious weakness. However, I found that the studied neural networks seemed to be a bit of out-of-date. As a reviewer, I hope to see applying the method in up-to-date models e.g. transformers, in other fields. I am not sure whether this point would make negative influence on the significance of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to thank the reviewer for their feedback.\n\n**W1**: We agree that our paper can be improved by showcasing stronger empirical evaluations for larger models. We have therefore conducted several new experiments on the popular parameter-efficient fine-tuning \nusing low-rank adapters (LoRA), as suggested by reviewer vGH1. \nHere, we have tested the proposed training model on larger \narchitectures such as DebertaV3 on Glue benchmark and Stable Diffusion \n(v1.4) dreambooth. In DebertaV3, we applied our method to matrix layers \nby interpreting matrices as order-2 tensors. For Stable Diffusion we \napplied our method to tensor layers in the Unet, while standard LoRA \nconvolution works essentially by reshaping the tensor into a matrix.\nWe believe that these results strengthen the empirical evaluation and we hope these could address your concerns.\n\n| GLUE test case| LoRA - rank 8, 1.33M params | Ours - 0.9 M params |\n|-----------------------|---------------|-----------------------------|\n| CoLa (Matt. Corr)     | $0.6759$      | $0.7065$                    |\n| MRPC (acc)            | $0.8971$      | $0.9052$                    |\n| QQP (acc)             | $0.9131$      | $ 0.9215$                   |\n| RTE (acc)             | $0.8535$      | $0.8713$                    |\n| SST2 (acc)            | $0.9484$      | $0.9594$                    |\n\n\n Stable Diffusion  | loss | \\# trainable parameters \n----------------------|---------------|----------------------------------\n LoRA ($r = 8$)       | $0.260$       | $5$ M                            \n LoRA ($r = 5$)       | $0.269$       | $3$ M                            \n LoRA ($r = 3$)       | $0.274$       | $1.8$ M                          \n ours ($\\tau = 0.02$) | $0.2635$    | $1.8$ M                          \n ours ($\\tau = 0.1$)  | $0.272$       | $1.5$ M\n\n**Q1**. The standard Tucker decomposition is the choice of the parameterization $W(i_1,\\dots,i_d) = \\sum_{j_1,\\dots,j_d} C(j_1,\\dots,j_d) U^{(d)}(i_d,j_d)$ to represent the d-mode tensor $W$. This choice is one possible extension of the classical singular value decomposition for tensors. For what concerns the vanilla Tucker decomposition and its training, we compare with the prototype method used often in training low-rank decompositions (such as LoRA), for which the update is simply given by a step of stochastic gradient descent on each factor of the decomposition, i.e. $C(t+1) = C(t)- \\lambda_t \\nabla_C \\mathcal L(t),\\, U^{(i)}(t+1) = U^{(i)}(t)-\\lambda_t \\nabla_{U^{(i)}}\\mathcal L(t)$.\nApart from the instability with respect to small singular values of vanilla Tucker, we would like to underline that this method introduces additional invariances in the parameter space (since orthonormality of the basis $U^{(i)}$ is not preserved over time). In terms of space complexity, Tucker vanilla and our decomposition are the same. The time complexity of one optimization iteration for the vanilla Tucker is of order $O(b(\\prod_{i=1}^d r_i+ \\sum_{i=1}^d n_i r_i))$, where $n_i$ are the dimensions of the tensor weight, $r_i$ are the Tucker ranks and $b$ is the batch size. The training of vanilla methods is essentially SGD, so the convergence properties of it are the same of stochastic gradient descent.\n\n**Q2** This is a good question and we will add a small discussion of this point in the manuscript. The presented theory and the algorithm is developed in general for Tucker tensors with $d$ modes, that for $d = 2$ covers also the matrix case. In particular then it is possible to apply the method both to matrices and tensors with more than two modes. In the matrix case, the Tucker decomposition would degenerate into a singular value like decomposition of the weight matrix.\n\n**Q3** The choice of the name \"geometry-aware\" refers to the fact that the proposed method is \"aware\" of the geometric structure underlying the problem. In section 2.1 we formulate the training of tensor-weighted neural networks as a gradient flow $\\dot W = -\\nabla_W \\mathcal L(W)$. When we want to superimpose a low-rank structure, vanilla methods follow a gradient flow $\\dot C = -\\nabla_C \\mathcal L, \\, \\dot U^{(i)} = \\nabla_{U^{(i)}} \\mathcal L$. This last gradient flow lies in the (Riemannian) manifold $\\mathcal M_r$ of Tucker tensors of rank $\\mathbf{r}$, but it is not aware in some sense of the original problem related to the geometry of the constraint. Our proposed method follows the global dynamics $\\dot W = -P(W) \\nabla_W \\mathcal L(W)$, where $P(W)$ is the **orthogonal** projection on the tangent space $T_W \\mathcal M_r$. Notice that the right hand side of this last projected differential equation is a solution of the minimization problem $ \\underset{\\delta W \\in T_W\\mathcal M_r}{\\arg \\min}||\\delta W+ \\nabla \\mathcal{L}(W) ||_{F}^2$, basically by definition of orthogonal projection. This variational principle is essentially telling us that locally we're taking the closest direction to following the original unfactorized dynamics. This property is not satisfied by the vanilla gradient flow system, that does not follow locally the original unfactorized dynamics.\nMoreover, our proposal is aware of certain properties of $\\mathcal M_r$, such as high curvature around tensors with small singular values along some mode. As shown in theorems 3.1 and 3.3, the provided theoretical bounds regarding approximation and descent do not depend on the singular values, making the method stable in their presence. This property is also highlighted numerically in figure 2, where we show that vanilla methods seem to suffer from this ill-conditioning.\n\nWe will add some details about the questions in the manuscript, and we hope the explaination has clarified any possible doubt. In any case we remain available if any further clarification is needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their insightful feedback. We have considered each comment and have made several improvements as a result. Below, we address and review the key points raised by the reviewers:

1. **Empirical evaluation**: 
Most reviewers commented that while our work presents a promising method with strong theoretical guarantees, additional benchmarks would strengthen the empirical evaluation. Reviewer SSBR and Reviewer Rp73 requested experiments with transformer architectures, while Reviewer vGH1 suggested results using LoRA and U-Net.  
In response, we have conducted new experiments that include fine-tuning results via LoRA adapters on DebertaV3 (up to $\sim$ 1.5M parameters) and Stable Diffusion (up to $\sim$ 5M parameters). The results can be found in tables 1 and 2 in the rebuttal pdf file. These new empirical evaluations show that our proposed approach outperforms the LoRA baseline, demonstrating the effectiveness of our method on a broader range of architectures. We thank the reviewers for their suggestions, which have significantly enhanced the quality and robustness of our work.

2. **Clarity and presentation**: 
Reviewers were divided on the clarity of our presentation. While Reviewers TTwz, vGH1, and Rp73 found the paper clearly written and easy to follow, Reviewer jjrQ expressed that the paper was difficult to follow and unclear in parts.  
To address this, we have done our best to provide explanations addressing specific questions and concerns, aiming to make our presentation more clear without sacrificing the necessary technical rigor. We will do our best to review the manuscript to improve clarity and readability in light of this feedback.

Meta-review: As models continue to grow in size, it is crucial to explore more parameter-efficient alternatives. This paper examines a "parameter saving" technique, namely imposing low-rank constraints. Low-rank constraints have proven effective even for modern transformers. What distinguishes this paper is its combination of theoretical insights and practical aspects. On the theoretical side, the paper analyzes the local convergence of gradient-based optimization under specific low-rank constraints. On the practical side, it proposes adapting these constraints to address optimization challenges, particularly focusing on the geometric aspects of the decomposition. The paper targets an important practical topic and addresses it both theoretically and empirically, I therefore recommend accepting the paper.

There is a substantial body of literature on applying constraints to neural network weight matrices to enhance optimization performance, including constraints based on the Stiefel manifold. The paper could benefit from reviewing this existing research.
Decision: Accept (poster)