MCNC: Manifold-Constrained Reparameterization for Neural Compression
Authors: Chayne Thrash
Reed Andreas
Ali Abbasi
Parsa Nooralinejad
Soroush Abbasi Koohpayegani
Hamed Pirsiavash
Soheil Kolouri
Venue: ICLR 2025 Poster (2025)
Keywords: Model Compression
LoRA
PEFT
Transformers
ViT
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: A method that compresses large models by limiting parameters to low-dimensional manifolds. It achieves superior compression of CNNs, ViTs, and LLMs across tasks compared to current techniques.

Abstract: The outstanding performance of large foundational models across diverse tasks,
from computer vision to speech and natural language processing, has significantly
increased their demand. However, storing and transmitting these models poses
significant challenges due to their massive size (e.g., 750GB for Llama 3.1 405B).
Recent literature has focused on compressing the original weights or reducing the
number of parameters required for fine-tuning these models. These compression
methods generally constrain the parameter space, for example, through low-rank
reparametrization (e.g., LoRA), pruning, or quantization (e.g., QLoRA) during
or after the model training. In this paper, we present a novel model compres-
sion method, which we term Manifold-Constrained Neural Compression (MCNC).
This method constrains the parameter space to low-dimensional pre-defined and
frozen nonlinear manifolds, which effectively cover this space. Given the preva-
lence of good solutions in over-parameterized deep neural networks, we show that
by constraining the parameter space to our proposed manifold, we can identify
high-quality solutions while achieving unprecedented compression rates across
a wide variety of tasks and architectures. Through extensive experiments in
computer vision and natural language processing tasks, we demonstrate that our
method significantly outperforms state-of-the-art baselines in terms of compres-
sion, accuracy, and/or model reconstruction time. Our code is publicly available at
https://github.com/mint-vu/MCNC.

Published as a conference paper at ICLR 2025
MCNC: MANIFOLD-CONSTRAINED REPARAMETERIZATION FOR NEURAL COMPRESSION
Chayne Thrash' Ali Abbasi', Reed Andreas', Parsa Nooralinejad', Soroush Abbasi Koohpayegani", Hamed Pirsiavash", Soheil Kolouri
DDparrmeett of Computer Science, Vanderbilt University, Nashville, TN, 37235 *Department of Computer Science, University of Californa, Davis, CA, 95616
chayne. thrash, ali.abbast: reed.w. andreas, soheil .oolour weandirrdddddb..eeee... pncorali, i, soroush, hpi av@ucavvis..dde
ABSTRACT
The outstanding performance of large foundational models across diverse tasks, from computer vision to speech and natural language processing, has significantly increased their demand. However, storing and transmitting these models poses significant challenges due to their massive size (e.g., 750GB for Llama 3.1 405B). Recent literature has focused on compressing the original weights or reducing the number of parameters required for fine-tuning these models. These compression methods generally constrain the parameter space, for example, through low-rank reparametrization (c.g.. LoRA), pruning, or quantization (e.g.. QLoRA) during or after the model training. In this paper, we present novel model compres- sion method, which we term Cumifold-Constraiiee This method constrains the parameter space to low-dimensional pre-defined and frozen nonlinear manifolds, which effectively cover this space. Given the preva- Ience of good solutions in pee-parametteriee deep neural networks, we show that by constraining the parameter space to our proposed manifold, we can identify high-quality solutions while achieving unprecedented compression rates across a wide variety of tasks and architectures. Through extensive experiments computer vision and natural language processing tasks, demonstrate that method significantly outperforms state-of-the-ar baselines in terms of compres- sion, accuracy, and/or model reconstruction time. Our code is publicly available https:/github..com/mmmm-wwwWww
Neural Compression (MCNC).
INTRODUCTION
Recent state-of-the-as art performance computer vision (Dosovitskiy et al., 2021; Zhai et al., 2022), natural language processing (Brown et al., 2020; Touvron al., 2023), speech (Zhang et al., 2023b; Baevski et al., 2020), and other fields (Grechishnikova, 2021; Jumper et al., 2021) is often achieved through large transformer networks (Vaswani et al., 2017) with hundreds of millions to tens of billions of parameters, trained on vast amounts of data. For example, the Llama 3.1 model (Dubey et al., 2024) with its 405B parameters requires around 750GB of memory. With the growing size of models and the shift towards large foundational models, effective compression methods are increasingly vital reduce size without significant performance loss, enabling efficient storage, communication, and deployment on edge devices.
Fine-tuning foundation models for custom tasks and datasets is now the gold standard for producing nigh-performing. task-specific models. However, this approach raises concerns about the storage of these customized models and creates a computational bottleneck when loading task-specific models and transferring their weights from CPU to GPU. To address these issues, Parameter Efficient Fine- Tuning (PEFT) methods have gained popularity in recent years. These methods focus on efficiently compressing the residual weights parameters needed to adapt foundation models to specific tasks. For example, LoRA (Hu et al., 2022) imposes low-rank constraints on residual weights, reducing the number of trainable parameters required for fine-tuning, making it widely adopted approach.

---

Published as a conference paper at ICLR 2025
Random feed-Forward Network
M: k-dimennsoonl manifold on
Compression rate:
k+1
Target Model
gd-1
(4(aa
c R
S4-1
((r)
Sinusoidal Activations
[],
Reparemeterization: 20x (b(a)
Lower Overhead for CPU to GPU Transfer
Figure I: Illustration of our proposed reparameterization technique, Manifold Constrained Network Compression (MCNC). The parameters 0 E IR are decomposed as 0 = Op + 20, where 0p is fixed and A0 Bu represents the learnable residuals. Here, 3 is the amplitude, and S is umit vector on the d-dimensional hypersphere. The unit vector u is generated by mapping lower-dimensional vector a F through a nonlinear generator Φ I2 Sd allowing the learnable perturbation to lie within k-dimensional subspace wrapped around the hypersphere. The right panel depicts the model partitioning strategy, where the weights are divided into d'dimensional chunks, and the corresponding (a, 3) pairs are learned for each chunk.
A wide variety of methods have been proposed to compress large models (Tang et al., 2024). These approaches generally be grouped into five main techniques: weight sharing, quantization, pruning, knowledge distillation, and reparameterization approaches (Hu al., 2022; Nooralinejad al., 2023; Girish et al., 2023; Koohpayegani al., 2024) aim to reduce the number of parameters or simplify the computations by restructuring the model weights. This often involves parameterizing the weights through more efficient forms, such as low- rank matrices (Hu et al., 2022), which can maintain the model's expressiveness while significantly reducing computational and memory overhead. this paper, we introduce a novel non-linear reparameterization technique for model compression, which achieves unprecedented performance at extreme compression rates while reducing the CPU-to-GPP GPU transfer time when loading networks.
Relevant to our method, the reparameterization
Our work in this paper draws inspiration from recent reparameterization methods such PRANC (Nooralinejad et al., 2023) and NOLA (Koohpaycgani et al., 2024). These methods take advantage of the fact that overparameterized neural networks with d parameters often have large number of good solutions (Liu et al., 2022). Hence, by restricting the search for optimal solutions to random k-dimensional subspaces within the parameter space, where k < d, they provide significant compression, i.e., E. without sacrificing performance. In this paper, we take a different approach and ask: Could better solutions be found by expanding the complexity of the search space from random subspace (Nooralinejad et al., 2023; Koohpayegani et al., 2024) to a k-dimensional nonlinear manifold that more efficiently captures the structure of the parameter space? Below, we introduce the core idea of our method through thought experiment.
d
Winding a string around sphere. Consider optimizing a model with parameters E Rdd where 0 = fu + AA, with 6, fixed. Using polar decomposition. we express AA Bu, where / E IR the amplitude and u 23
A6 gd-1 represents the direction on the hypersphere gd-1, Now,
consider segment of the real line [-L, L], representing a string of length 2L, wrapped around gd-1 parameterizing a one-dimensional manifold with [- Instead of optimizing in d-dimensional space, we optimize over the amplitude and the manifold parameter Figure I (left panel) illustrates this for d = Extending this, we replace the segment with a k-dimensional space [- L, L)*, wrapping it around g to increase coverage. This reduces the parameter space from to & + 1, reparameterizing 0 by a and B. To wind such a k-dimensional subspace around the d-dimensional hypersphere $d-1, propose to use random feedforward neural network with sinusoidal activations, which we refer to as the 'random generator." Sinusoidal activations are essential because they introduce periodicity in parameterization, facilitating smoother and more uniform coverage of the hypersphere and enhancing the differentiability of the generator.
Contributions. Our specific contributions in this work are:
1. Introducing novel non-lineat reparameterization technique for model compression, which we term
Neural Compression (MCNC). This method restricts the
2

---

Published as a conference paper at ICLR 2025
optimization of network parameters to k-dimensional manifold within the original d- dimensional parameter space, enabling more efficient compression.
2. Demonstrating the effectiveness of our proposed method compared to recent network compression methods in the literature across vision and natural language processing tasks and across diverse architectures.
RELATED WORK
Our goal in this paper is to train deep model with minimal number of parameters, making it more efficient to communicate models between agents/entities or store on devices with limited memory. Our compact representation makes no assumptions about the model size, weight distribution, the number of non-zero parameters, or computational precision, making it orthogonal to methods like weight-sharing, quantization, pruning, and knowledge distillation. Hybrid approaches could integrate Manifold Constrained Neural Compression (MCNC) with these techniques. In the following, we briefly review these methods, which aim reduce the model size and improve the inference time.
Weight sharing can be enforced through the model architecture, such as with convolutional kernels (Krizhevsky al., 2012), recurrent networks (Hochreiter & Schmidhuber, 1997), or other hybrid architectures (Dehghani et al., 2019). It can also be achieved post-training via clustering mechanisms (Nowlan & Hinton, 1992; Ullrich et al., 2017) or hashing (Chen et al., 2015; Eban et al., 2020; Reagan et al., 2018) to bundle the weights of a model into a few shared components. Other notable weight sharing methods proposed in recent years include (Gao et al., 2019; Plummer et al., 2022; Shakerinava et al., 2024). Similar to weight sharing methods, MCNC uses a trains weights in higher dimensional space using a low dimensional representation. However, MCNC is a general reparameterization that can be combined with weight-sharing techniques, e.g., layers in the network can share oow-dimeniiona parameters to enhance compression.
Quantization is widely used model compression technique that reduces the bit- bit-width needed to rep- resent model weights and activations. For instance, Rastegari et al. (2016) proposed KNOR-Networks, where binary operations
used in network with XNOR gates to approximate convolutions, leading to a reported 58x speedup and 32x memory savings. The core challenge with these techniques is that naive quantization of models often results in a significant performance drop. On the other is often formulated as constrained optimization problem that
hand, quantized training/fine-tume requires specialized optimization techniques (Lee et al., 2021). A large body of recent work focuses on effective quantization methods for large models (Xiao et al., 2023; shkolnik et al., 2020; Yao et al., 2023; Dettmers et al., 2024). In the extreme case, quantization reduces float32 to binary, achieving 32x compression. In this paper, we target higher compression rates.
Pruning a primary technique for network compression, as highlighted in several studies (Hassibi et al., 1993; Kim et al., 2020; Lin et al., 2020; Siems et al., 2021; Tiwari et al., 2021; Hayou et al., 2020; Wang et al., 2020a; Li et al., 2021; Rastegari et al., 2016; Lee et al., 2021). High compression rates are achievable through methods like those in (Kusupati et al., 2020b; Isik et al., 2022; Zhang et al., 2022; Sun et al., 2024), making pruming one of the main methods for network compression. Unstructured pruning zeros many parameters, storing only the non-zero weights and their indices. Compression efficiency could further be improved with coding strategies like Huffman or Run-Length Encoding (RLE) (Han et al., 2015a). More recently, various works have considered structured/struccuuaal pruning removing entire modules from a network while preserving the performance (Anwar et al., 2017; Fang et al., 2023). We show that MCNC matches the performance of pruning methods at lower compression rates while outperforming them at higher rates.
Compression by reparameterization has recently emerged as powerful method for efficiently compressing large models. LoRA (Hu et al., 2022) introduced low-rank reparameterization for fine-tuning large models, inspiring many extensions (Valipour et al., 2023; Zhang et al., 2023a; Dettmers et al., 2024; Koobpayegani et al., 2024). PRANC (Nooralinejad et al., 2023), closely related to our work, constraints model weights to a randomly selected low-dimensional subspace. NOLA (Koohpayegani et al., 2024) combines ideas from PRANC and LoRA, pushing low-rank PEFT below the classic rank-one limit. NOLA and PRANC assume that good solutions lie on low-dimensional nonlinear manifolds within the overparameterized model's parameter space (Liu et al., 2022; Li et al., 2018), making it likely that a random subspace will intersect this manifold, allowing for viable solutions within the subspace.
3

---

Published as a conference paper at ICLR 2025
This paper stems from our curiosity to explore whether random k-dimensional manifold in d-dimensional parameter space (k < d) can be constructed to maximize coverage, thereby increasing the likelihood of intersecting the manifold of good solutions. Hence, our method can be considered as a generalization of PRANC (Nooralinejad et al., 2023) and NOLA (Koohpayegani et al., 2024).
Manifold constrained optimization has been employed in various works constrain model pa- rameters to a Riemannian manifold in order to induce various geometric characteristics (Fei et al., 2023). For instance, some methods require that the model parameters reside on or be close to the Stiefel manifold, i.e., the set of k orthonormal vectors in Rd, to reduce redundant representations (Ozay & Okatani, 2016; Huang al., 2018: Wang et al., 2020b). (Absil et al., 2008; Boumal, 2023) in neural networks is gaining significant attention, as evidenced by recent developments in geometric optimization libraries for deep learning (Meghwanshi et al., 2018; Kochurov et al., 2020; Luchnikov et al., (021). In this work, we use random feed-forward network with sinusoidal activations to parameterize
optimization
oow-dimensional manifold within a higher-dimensional hypersphere. We constrain optimization to this manifold, relying exclusively on standard (stochastic) gradient descent, avoiding the need for Riemannian optimization.
METHOD
At its core, Manifold Constrained Neural Compression (MCNC) relies on two key components: 1) an explicit nonlinear mapping that wraps k-dimensional space around a sphere in a dd-dimensional space, where k < d. and 2) partitioning the model's parameters into d-dimensional segments and optimizing them in the k-dimensional input space of the nonlinear map. This process would result in a compression rate of almost d/k.
Notations and Preliminaries. Let 0 E Rdd denote partition of the model parameters, which we aim to optimize. Through polar decomposition, we express 0 by its amplitude = ||]| and its direction 0/|10|12. Additionally, we denote the d-dimensional unit sphere with g Our goal to reparameterize u E gd-1 using significantly fewer parameters than To achieve this, we will employ 'generator' model to explicitly parameterize a k-dimensional manifold that best represents the d-dimensional hypersphere.
3.1 TRAVERSING A HIGH-DIMENSIONA SPHERE BY LOW-DIMENSIONA. MANIFOLD
Rationale. We aim to model a d-dimensional hypersphere using a k-dimensional manifold. For clarity, let's examine this concept in a simpler scenario: consider d = corresponding to a sphere, and k = 1, akin to string. This familiar setting helps illustrate our objective. Given string of specific length, how can we most effectively cover the surface of the sphere? The answer intuitive: simply wrap the string around the sphere. This wrapping acts as a nonlinear operator that takes straight line and deforms around the sphere. Similarly, a k-dimensional input space can be wrapped via nonlinear function, i.e., generator, around hypersphere in the d-dimension. Next, we discuss how to formalize this problem.
Formalizing the problem. To traverse hypersphere using low-dimensional manifold, we aim to maximize its coverage. We formalize this problem as follows: Let (L(-L, L1h) be the uniform distribution over the k-dimensional hypereube [ [-,,, r.)*, and (S()) the uniform distribution over the d-dimensional hypersphere gd-1. Our goal is to develop a nonlinear mapping that wraps the k-dimensional hypercube around the d-dimensional hypersphere, maximizing coverage. This problem can be formalized as finding a nonlinear function Φ R* + Rd that transforms ((-L,119) into ((dd-11), ). mapping samples ~ U[[LL, L}*) to the hypersphere such that s(a) U( Hence, we need to have a way to measure the uniformity of the ő(a)s. To do that, we measure the Wasserstein distance (Peyré et al., 2019) between the output probability distribution of & and the uniform distribution on the hypersphere.
Modeling the generator. We model the generator, o R Rd via a feed-forward network. We consider various activation functions, including the Sigmoid, Rectified Linear Unit (ReLU), and Sine activation (Sitzmann et al., 2020). First, we ask whether a randomly initialized network can provide the desired characteristics. Second, we ask whether we can optimize such a generator to provide maximal space traversal. We note that optimizing such generator model mirrors the challenges of deep generative modeling, subject thoroughly investigated in literature (Goodfellow et al., 2014; Arjovsky et al., 2017; Deshpande et al., 2018). We emphasize that our decision to use the uniform distribution on gd-1 target distribution stems from an assumption of prior knowledge about
4

---

Published as a conference paper at ICLR 2025
Randomly Initialized
Optimized
0.18% (M2=6.79)
1.35% IN, -0.88)
3.12% I0 -0.90
3,43% (W.50.58)
3.43% (Wg=0.58)
1.38% (46m2.65)
$2.85% (W) (y.00099)
Sigmold
(M2+573)
-0.60)
2.8%% (W,-0.00)
2.85% (W,00.80
2.85% (Wy-0 63)
0.660
WM
ReLU
9.6%% (M)=0.311
M24% (8211
86.90% (95m0.12)
SRE (85-1109)
67.4% (M))0.005)
98.2%% (W.40.04
59.09% (W:
Sine
L-10
L- 50
L-100 100
L- 1000
L- 100
L- 1000
Figure 2: Traversal sphere using 1-dimensional manifold, where Φ 32 is multi-layer perceptron with architecture + 1024 - 1024 + 3 and activation functions Sigmoid, ReLU, or Sine. The input bound L is absorbed into the first layer's weights. The left panel displays outputs of randomly initialized networks for different activations and L values, while the right panel shows outputs after optimization. Uniformity is quantified using exp(- (á, 1)), with T = 10.0 and W2 representing the Wasserstein distance between the network's ' output and the uniform distribution v. the importance different directions for the downstream task of optimizing a network. Should such information become available, the target distribution could be adjusted reflect this knowledge, allowing for more precise wrapping around areas of greater importance. this paper, we used the SWGAN (Deshpande et al., 2018) framework train the generator due to its simplicity.
As guiding example for developing our generator, we consider the case where modeled feed-forward network with the architecture 1 1024 1024 + 3. Figure 2 shows the output of Φ for randomly initialized networks with various activation functions, along with the outputs after optimization. We evaluate uniformity by reporting exp(-7W2 (λ,1)), where the output distribution of á and represents the uniform distribution on gd-1 These results are presented in Figure Interestingly, we observe that for larger values of L, the randomly initialized network with Sine activations achieves strong coverage of the sphere, with optimization only marginally improving the coverage. Therefore, in our main results, we use a randomly initialized need-forward network with Sine activations as the generator while also conducting an ablation study the impact of training the generator. Notably, the random generator can be efficiently stored or communicated using scalar random seed, assuming access to shared pscudo- mandom number generator (PRNG).
1 and
3.2 REPARAMETERIZATTOO AND MANIFOLD CONSTRAINED OPTIMIZATION
Given random generator model, R* Sd-1 we reparameterize d-dimensional residual vector as AO = BΦ(a), where 3 E R and a E R* thereby reducing the number of parameters from d to + Figure illustrates this concept. Let L: RRd E denote the loss function for a specific task of interest, e.g., image classification, where L(0) is the associated loss with parameter 6, We then constrain the training of the parameter as:
0" =00 + 3' (a* arg min L(O, - -K(a))
a,3
Note that we use Og to emphasize that optimization can begin from any random initialization or pre-trained weights, such as those used in PEFT. This optimization process constrains the model parameters, 0, to lie on k-dimensional manifold within Sd-1 which is parameterized by ø.
3.3 LEARNING THE DEEP MODEL
We initialize the generator <(.) randomly and freeze it. Then, given deep model, reshape its parameters to a long vector and then divide that into chunks of size d, and reparameterize each chunk with parameters using the generator ((-). In the case the model size is not divisible by
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of our method with pruning techniques on ViT-Ti and VIT-S models trained on 1000.. We exclude position embeddings, CLS token, and Layer Normalization parameters when computing the percentage of model size.
ViT-Ti
Method
Percentage Model Size
Acc.
VIT-S
Baseline
100%
Magmitude 2015b 66.7%)
Method
Percentage of Model Size
Acc.
PLATON 2022 (Pr. 66.7%)
50%
81.6
MCNC (Ours)
80.0
Baseline
100%
83.9
Magnitude 20155 (Pr. 86.7%)
73.0
Magnitude 2015b (Pr. 83.3%)
79.8
PLATON 20220222222 (Pr. 86.7%)
20%
76.1
PLATON 2022 (Pr. 83.3%)
25%
82.2
MCNC (Ours)
77.1
MCNC (Ouns)
82.7
Magniude 2015b (Pr. 93.3%)
60.9
Magnitude 2015h (Pr. 93.3%)
72.0
PLATON 2022 (Pr. 93.3%)
10%
PLATON 2022 (Pr. 93.3%)
10%
77.4
MCNC (Ours)
72.9
MCNC (Ours)
80.2
Magnitude 2015b (Pr. 96.7%)
45.8
Magnitude 2015b (Pr. 96.7%)
66.0
PLATTON 2022 (Pr. 96.7%)
5%
55.0
PLATON 2022 (Pr. 96.7%)
5%
71.1
MCNC (Ours)
69.1
MCNC (Ours)
76.3
Magnitude 2015b (Pr. 98.7%)
29.4
Magnitude 2015b (Pr. 98,7%)
41.5
PLATON 2022 (Pr. 98.7%)
2%
39.0
PLATON 2022 (Pr. 98.7%)
2%
57.6
MCNC (Ours)
61.3
MCNC (Ours)
66.7
Magnitude 2015b (Pr. 99.3%)
PLATON 2022 (Pr. 99.3%)
1%
30.6
MCNC (Ours)
52.9
d, the last chunk will have some extra parameters that will be ignored. Finally, we train the model by optimizing (a, (3) for all chunks using Eq minimize the loss of the deep model the task of interest, e.g., image classification. The backpropagation is as simple as calculating the gradient for model parameters and then using the chain rule backpropagate through the generator d(.). Hence, auto-differentiation can be directly used to optimize as and |3 without the need for geometric optimization techniques.
EXPERIMENTS
To demonstrate the performance of our method, evaluate MCNC under two different settings:
1. Training from scratch for image classification. In this setting, we let 6, be a randomly initialized network and optimize using MCNC. Note that since randomly initialized network can be communicated using only the random seed, this does not increase the cost of compression. We show our effectiveness at compressing both Vision Transformer (ViT) (Dosovitskiy et al., 2021) and ResNet (He et al., 2016) architectures.
2. Parameter Efficient Fine-Turre Tuning of LLMs. In this setting, we instead begin with a pre- trained 0* and optimize the AO MCNC. We conduct this experiment for L.LMs where fine-tuning of large models has become the norm.
Given that our method is orthogonal to the low-rank parameterization used in LoRA (Hu et al., 2022), we reparameterize either the original networks or their cank-constrained versions our experiments.
4.1 TRAINING IMAGE CLASSIFIERS FROM SCRATCH
ImageNet- 100 on Vision Transformer Architectures. We begin by evaluating on training Vision Transformers from scratch on the ImageNet-100 dataset (Tian al., 2020). We evaluate both ViT-Ti and ViT-S (Touvron et al., 20.1). We compare MCNC against iterative unstructured pruning method PLATON (Zhang al., 2022) as well the classic Magnitude pruning (Han al., 2015b). We report the final compressed model size as percentage. As unstructured pruning also requires storing the indices of non-zero values, these must be accounted for when computing the compression rate. Although this requires storing two values per weight, the number of bits required for each index can be reduced by storing the distance between each non-zero value (Han et al., 2015a). Therefore, we assume half-precision for the indices and prune sparsity rates 50% higher than the desired compression. We exclude position embeddings, CLS token, and Layer Normalization (Ba et al., 2016) parameters from all compression methods. We provide hyperparameters used for method and baselines in section A.3. Table show our results on ViT-Ti and ViT-S. Our results show that, particularly for high compression rates, MCNC is more effective than pruning for compression.

---

Published as a conference paper at ICLR 2025
Table Comparison with PRANC and NOLA on ImageNet-100 using ResNet-18 with multiple compression rates.
Percentage of Model Size
Method
Ref
Acc.
Baseline
100%
82.1
Ours
80.7 = 0.4
50%
Ours w/ LoRA
79.1 =: 0.7
Ours
79.9 0.3
20%
Ours w/ LoRA
80.0 = 0.2
Ours
78.0 = 0.0
10%
Ours w/ LoRA
80.3 = 0.5
Ours
75.4 = 0.2
5%
Ours w/ LoRA
79.6 = 0.1
PRANC 2023
2023
67.3
Ours
2%
70.2 + 0.1
Ours w/ LoRA
76.9 + 0.2
PRANC 2023
2023
61.1
NOLA 2024
2024
64.7
1%
Ours
63. = 0.2
Ours w/ LoRA
71.7 + 0.3
For ViT-Ti, we outperform pruning by 8% at 10% of the original model size. On VÝT-S, although pruning methods are able to outperform at higher compression rates, we see that our method is able to maintain high performance as we decrease model size.
ImageNet- 100 on 18 Next, show that our method works across architectures by training from scratch on ImageNet- 100 using the ResNet- 18 (He et al., 2016) architecture. We compare with other recent network reparameterizztioo methods PRANC (Nooralinejad et al., 2023) and NOLA (Koohpayegani et al., 2024). Since it was shown (Koohpayegani et al., 2024) that reparameterizing boost performance,
LoRA rather than the original model architecture our method this setting. Results are shown Table 2. Our method demonstrates a 7% accuracy improvement over the baselines at 1% of the original model size. In addition, compression rate we see that we can recover up to 97% of the original model accuracy at 10% of the size.
also show results of
reduce the
CIFAR-10 and CIFAR-100. We compare MCNC on CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) with both pruning methods well PRANC (Nooralinejad al., 2023) and NOLA (Koohpayegani et al., 2024) under extreme compression rates. We show results on the two datasets using both ResNet-20 and ResNet-56. For the pruning methods and PRANC, we directly compare with the results reported in (Nooralinejad et al., 2023). We present results for these experiments in Table can be seen that MCNC able consistently outperform all other methods while using similar number of parameters. In addition, just as NOLA improves upon PRANC with the addition of LoRA, MCNC gains a boost as well.
4.2 MCNC FOR FINE-TUNING LARGE LANGUAGE MODELS
In this section, compared the performance of MCNC and NOLA in parameter-efficient fine-tuning, We conducted fine-tuning experiments on two variants of the LLaMA-2 (Touvron et al., 2023) language model, LLaMA 7B, and LLaMA 13B, using the Alpaca dataset our training data (Taori et al., 2023). We report both training loss and validation loss on the Alpaca dataset. Additionally, we report MMLU (Massively Multitask Language Understanding) (Hendrycks et al., 2021) 5-shot accuracy following the standard practice. This benchmark spans 57 tasks across diverse fields such as computer science, mathematics, law, and history.
Implementation Details: We use the training code from QLoRA (Dettmers et al., 2023) and NOLA (Koohpayegani et al., 2024) for our experiments. We quantize the original parameters of the language model to 4-bit and apply and fine-tune the adapter on all layers of the transformer. For NOLA, we followed the hyperparameters reported in (Koohpayegani et al., 2024). We set the rank to for both NOLA and method in LLaMA 7B, and 16 in LLaMA 13B. In method, with the following specifications: a 3-layer MLP with an input dimension of 5, an output dimension of 5000, and hidden dimension of 32. In NOLA,
generator
64 bases for A and B in LLaMA 7B
7

---

Published as a conference paper at ICLR 2025
Table
Comparison of our method with pruning methods as well as PRANC and NOLA. We
include results for our method with and without LoRA applied to each of the layers. As none of the methods compress BatchNorm parameters, we remove them from all parameter counts.
Method
Ref
Arch.
Dataset
Params
Acc.
Baseline
R20
C10
269, 722
88.9
STR 2020a
2023
R20
C10
12, 238
76.0
PRANC 2023
2023
R20
C10
10, 000
81.5
NOLA 2024
2024
R20
C10
11, 500
82.4
MCNC (Ours) w/o LoRA
R20
C10
10,
82.3 + 0.3
MCNC (Ours) w/ LoRA
R20
C10
9,640
83.9
Baseline
R56
C10
853, 018
91.6
DPF 2020
2023
R56
C10
13,414 414
47.7
SuRP 2022
2023
R56
C10
10,834 834
66.7
STR 2020a
2023
R56
C10
13,312
67.8
PRANC 2023
2024
R56
C10
5,000
76.9
NOLA 2024
R56
C10
5,000
78.5 + 0.1
MCNC (Ours) w/o LoRA
R56
C10
280
78.7 + 0.2
MCNC (Ours) w/ LoRA
R56
C10
5,590
81.5
Baseline
R20
C100
275,
60.8
DPF 2020
2023
R20
C100
10, 770
12.2
SuRP 2022
2023
R20
C100
G. 797
14.5
STR 2020a
2023
R20
C100
10, 673
13.2
PRANC 2023
2023
R20
C100
0,000
32.3
NOLA 2024
R20
C100
5, 180
35.6 + 0.1
MCNC (Ours) w/o LoRA
R20
C100
110
34.7 + 0.2
MCNC (Ours) w/ LoRA
R20
C100
0,070
36.7+0.2
Baseline
R56
C100
858, 868
64.3
DPF 2020
2023
R56
C100
19, 264
19.1
SuRP 2022
2023
R56
C100
10,919
14.6
STR 2020a
2023
R56
C100
18,88) 881
26.0
PRANC 2023
2023
R56
C100
5,000
33.0
NOLA 2024
R56
C100
5,0
36.2 + 0.6
MCNC (Ours) w/o LoRA
R56
C100
5,
35.2 + 0.1
MCNC (Ours) w/ LoRA
R56
C100
0,515
36.5 + 0.8
Table 4: Instruction finetuning for quantized LLaMA-2: We fine-tune LLaMA-2 with the Alpaca
dataset and compare MCNC to NOLA, maintaining the same number of parameters for both. "Gener- ation GFLOPs" refers to the FLOPPs needed to generate the original parameters the adapter on the fly. MCNC achieves comparable performance NOLA while requiring fewer FLOPs for on-the-fly parameter generation. This efficiency results in faster inference and training for MCNC compared to NOLA. We use a single RTX 3090 GPU to calculate the throughput. Note that the throughput here includes both the adapter's 's reconstruction and the forward pass through the base model and adapter.
LLaMA-2 7B (4-bit)
Method
Trainable
MMLU
Train
Val
Throughput (Samples/Sec)
Adapter Model Reconstruction GFLOPs
Parameters
Acc
Loss
Loss
LoRA (rank=1) 2022
2.5M
45.6
0.98
1.06
7.1
NOLA 2024
28k
45.5
1.08
1.01
3.1
2.56 1.37
MCNC
25k
45.9
1.03
1.01
6.2
LLaMA-2 13B (4-bit)
LoRA (rank=1) 2022
3.9M
54.8
0.94
0.97
4.1
NOLA 2024
78k
54,8
1.00
0.96
1.7
17.53
MCNC
77k
55.0
1.00
0.95
3.9
4.22
and 140 bases for LLaMA 13B. These numbers of bases result in the same number of optimized parameters as our method for each architecture. All other hyperparameters were identical for both methods except the learning rate; used Ir 0.001 for NOLA and 0.01 for ours.
Results: The results are presented in Table MCNC has comparable performance to NOLA with similar number of parameters. However, MCNC requires 46% fewer GFLOPs for generating
8

---

Published as a conference paper at ICLR 2025
parameters the fly compared to NOLA. This efficiency translates into faster inference and training for MCNC compared to NOLA. As shown in Table 4, MCNC achieves double the throughput for LLaMA 7B and 2.2 times higher throughput for LLaMA 13B. While it is feasible to generate parameters offline and merge them with the original weights of the language model, this approach limits the applications when processing multiple tasks and their corresponding adapters in batch. Therefore, in scenarios involving batch processing of tasks, MCNC holds an advantage over NOLA due to its faster throughput.
4.3 ABLATION STUDIES
In the previous experiments, we've shown the effectiveness of our method in training large models constrained to our low dimensional manifold. However, as the manifold is modeled as neural network, the design space of the manifold itself is large. We explore the effect of various design deci- sions by applying different generator variations to the task of MNIST classification. We intentionally choose a very small dataset so that we can afford running several experiments. Full details of our experimental setup found in section A.4.
Effect of choice of activation function: In section 3.1, show that for k = and d 3, randomly initial- ized MLP with sinusoid activations can effectively cover gd-1. For this reason, we selected Sine as activation function. It's worth asking, however, if this coverage is truly important and whether other activation functions may actually provide better performance. We explore this in Table 5 by comparing against ReLU, Leaky ReLU, ELU, Sigmoid, and no activation function. For the no activation experiment, we no longer train separate amplitude as the magnitude of the inputs directly control the output magnitudes. Instead, we add this additional parameter as an additional input to the generator. From the results, it can be seen that many activation function choices are actively harmful and perform worse than none. Sigmoid and Sine activations are the only that outperform none with Sine performing the best. Importantly, when no activation is used, our method recovers a variation of PRANC (Nooralinejad et al., 2023).
Table 5: The impact of MCNC activa- tion function on MNIST classification.
Activation Function
Acc.
None (linear)
81.6 + 0.5
ReLU
76.0 + 2,8
Leaky ReLU
76.9 1 1.6
ELU
81.3 + 0.2
Sigmoid
83.7 + 0.7
Sine
84.6 + 0.7
Effect of generator input frequency: As shown in Sec-
Table 6: The effect of frequency of first layer of Sine activations on the accu- racy of MNIST.
tion 3.1, the magnitude of the input to the generator, i.e., L, controls the number of twists and turns in the manifold. While this twisting is necessary to effectively cover a high dimensional space, is also likely to increase optimiza- tion difficulty. It's worth asking then how different settings of this value affect compression of downstream models. To test this, we multiply the inputs to the generator by constant associated with the frequency and show results in Table 6. It's immediately apparent that higher frequencies are necessary to achieve high performance as a frequency value of 1.0 performs similarly to linear generator. Small increases in frequency greatly increase performance until saturation at 4.0.
Input Frequency
Acc.
1.0
81.9 + 0.4
2.0
83.5 0.2
4.0
85.1 1 0.2
8.0
84.7 + 0.4
16.0
85.0 + 0.4
32.0
85.5 + 0.4
Varying model size with fixed number of compressed parameters: A unique aspect of MCNC is that the total number of trainable parameters can be fixed regardless of the number of model parameters by varying its compres- sion rate. This allows us study how performance scales as model complexity increases while the number of train- able parameters remains the same. As model complexity increases, the number of good solutions should increase. Thus, we anticipate that this over- parameteriiznnnn will facilitate finding high performing compressed models re- gardless of the number of trainable parameters. We varied the MLP hidden size while maintaining the number of trainable parameters from the previous experiments and present the results in Table 7. As expected, see consistent increase accuracy as model size
Table 7: Impact of increasing model
size with fixed number of compressed parameters on MNIST classification.
Hidden Size
Acc.
16
81.1 + 0.3
32
82.5 + 1.1
64
84.0 + 0.7
128
83.9 + 1.3
256
84.6 + 0.4
512
85.2 + 0.2

---

Published as a conference paper at ICLR 2025
increases. While smaller architectures can occasionally achieve higher accuracy, the high standard deviations indicate that their optimization often fails to find good solutions.
Speedup of CPU to GPU transfer: By decreas- ing the overall size of the model, MCNC can also be used to reduce the time taken to move large models from CPU to GPU. As long as the generator loaded into GPU memory, only the much smaller set of as must be transferred from CPU to GPU and then expanded into the full model using the generator. To show this can result in speedup, we compare the time to load ViT-S from CPU to GPU memory with the time to load and expand the as associated with a ViT-S compressed by 100x. We perform this experiment 100 times using an RTX A6000 and report the average timings in Table 8. These results show that, despite having to complete forward pass of the generator, we are still able to reduce transfer time by half.
Table 8: CPU to GPU transfer time of uncom- pressed/compressed ViT-S model (100x).
Uncompressed Compressed 17.8
Speedup
35.5 ms
2.0x
Effect of training on the generator: In Section 3.1, we demonstrated that training generator only slightly im- proves its coverage of the hypersphere. To further investigate the impact of using trained generators, we applied MCNC with both random and trained generators on CIFAR-IO/ 10/1000 The re- sults presented in Table 9 show consis- tent but only marginal improvements in accuracy when using trained gen-
Table 9: Impact of using a random trained generator on CIFAR10/100 classification with different models.
Arch.
Dataset
Acc. Random
Acc. Trained
R20
C10
82.3 + 0.3
82.4 + 04
R56
C10
78.7 + 0.2
79.2 + 02
R20
C100
34.7 + 0.2
36.1 + 0.6
R56
C100
35.2 + 0.1 36.7 : 0.5
erators. Given the advantages of random generators, including: 1) easy scalability to different compression rates, 2) storage and communication efficiency (only requiring a random seed), and 3) potential cryptographic benefits, we used random generators for our main experiments.
CONCLUSION AND LIMITATIONS
Leveraging the prevalence of viable solutions in training overparameterized deep models, we have developed nonlinear reparameterization method aimed at exploring local minima within low- dimensional manifold of the parameter space. This approach utilizes a generator model that maps a low-dimensional uniform distribution a high-dimensional uniform distribution, effectively wrapping the -in-dimesiional input subspace around the high- nigh-dimensional hypersphere. Through extensive experiments computer vision and NLP tasks, we demonstrate that our method, MCNC, significantly outperforms state-of-the offte-----e-arr baselines in terms of accuracy and/or model reconstruction time. We believe this technique can enhance the storage and deployment of large models, which are integral to the recent advancements in AI, thus facilitating more democratized access to AI tools. However, this may result in negative impacts by putting AI in the hands of many non-sophisticated adversaries.
Limitations: We have not fully explored the optimal generator model configuration, including its architecture, loss function, and types of regularizers, and their effects on the downstream optimization. Moreover, due to computational constraints, we have not shown the effect of MCNC in training large language models from scratch, which may reveal the most important impact of MCNC.
ACKNOWLEDGEMENT
This work was partially supported by DARPA under Contract No. HR001 and HR001 12290115 15 and NSF grants 1845216 and 2339898, and NSF CAREER Award 2339898.
REFERENCES
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2008. 4
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-18, 2017. 3
10

---

Published as a conference paper at ICLR 2025
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 213-223. PMLR, 2017. 4
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450. 2016. URL http: // /aaxivivvv... org/ abs/ / 1077 06450. 6
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 18:12449-12466 2020.
Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press,
2023. 4
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL htt.tttts///ppcceddnnnge nwwrips.. httpp//iissspppppppeeeeeeeeeeeee files/paper/ 1220//ile//// (55TT///oo.cbb9677111aabbaaaaaaa
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning. pp. 2285-2294.
PMLR, 2015. 3
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https:
//openrevies ew net/ / htt///ormmmiiie doy.middo...y
Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3483-3491, 2018. 5,5
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023. 7
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Advances in Neural Information Processing Systems, 36, 2024. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL httts:: net / htte//orummmi -- f cbF FNTTy. 1,
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Elad Eban, Yair Movshovitz-Atiaas, Hao Wu, Mark Sandler, Andrew Poon, Yerlan Idelbayev, and Miguel A Carreira-Peppinán Structured multi-hashing for model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11903-11912, 2020.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16091-16101, June 2023. 3
Yanhong Fei, Xian Wei, Yingjie Liu, Zhengyu Li, and Mingsong Chen. A survey of geometric optimization for deep learning: From euclidean space to riemannian manifold. arXiv preprint arXiv:2302.082. 10, 2023, 4
11

---

Published as a conference paper at ICLR 2025
Shangqian Gao, Cheng Deng, and Heng Huang. Cross domain model compression by structurally weight sharing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8973-8982, 2019.
Sharath Girish, Kamal Gupta, Saurabh Singh, and Abhinav Shrivastava. Lilnetx: Lightweight networks with EXtreme model compression and structured sparsification. In The Eleventh Inter- national Conference on Learning Representations, 2023. URL https:/ httttt:///pppeeeeeee// http://ppppp/eeeee///ee/e nett/
forum? www.iidd-iiiileee ZCLg.2 2
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 4
Daria Grechishnikova. Transformer neural network for protein-specific de novo drug generation as a machine translation problem. Scientific reports, 11(1):321, 2021.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 2015a. 6
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015b. URL https: / http://oocecdccooooooooooooooccc ngs neurips cc/paper wol.ffles/pppe ess/pppr/2201// httee/aape__flleepppeeeeeeeeeeee 499025fe6- Papeeaaappaae--eee pdf.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEEE, 1993. 3
Soufiane Hayou, Caan-Francois Ton, Arnaud Doucet, and Yee Whye Teh. Robust pruning at initializa- tion. arXiv preprint wxXiv2002.007997 2020. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 6,7
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. 7
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997. 3
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http////pnneeeeeeeeeeeeeeeeeeeee net/ forum?
id=nzevKeeFyf9, 2, 3, 6,
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 448-456. JMLR.org. 2015. 17
Berivan Isik, Tsachy Weissman, and Albert No. An informatiin-thhorrticc justification for model pruning. In International Conference on Artificial Intelligence and Statistics, pp. 3821-3846. PMLR. 2022. 8,8
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunkouu Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
12

---

Published as a conference paper at ICLR 2025
Woojeong Kim, Suhyun Kim, Mincheol Park, and Geunseok Jeon. Neuron merging: Compensating for pruned neurons. Advances in Neural Information Processing Systems, 33:585-595, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv. org/ abs 412 2. 6980. 17
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Gcoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005.02879, 2020. 4
Soroush Abbasi Koohpayegani, Navaneet K L, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. NOLA: Compressing LoRA using linear combination of random basis. In The Twelfth International Conference on Learning Representations, 2024, URL https: htttps://ppennteee /pppnrevii... htt./oorummidd10 J 2, 3, 7,
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012. 3
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pp. 3544-5555 PMLR, 2020a.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, July 2020b. 3
Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wise gradient scaling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6448-6457, 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. International Conference on Learning Representations, 2018. URL
https:://penseeeeeeeee /ttp:////peeeeeeeeeee//eeeeeeeee .eet forum?id-ryupe oet.eprrerreeeeeteeooooooooooooo
WCW. 3
Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 4438-6477, 2021. 3
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL https: htt::///ppeeeiiieeeee net/ forum?id-S Jem8 1SF wB. 3,
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59: 85-116, 2022. 2, 3
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019, URL https: http:///ppee/eeeeeee net wtt.m/orrmm///mmm did= Bkg6RiCqY7. 17
Ilia Luchnikov, Alexander Ryzhov, Sergey Filippov, and Henni Ouerdane. Qgopt: Riemannian optimization for quantum technologies. SciPost Physics, 10(3):079, 2021.
Mayank Meghwanshi, Pratik Jawanpuria, Anoop Kunchukuttan, Hiroyuki Kasai, and Bamdev Mishra. Metorch, a manifold optimization library for deep learning. arXiv preprint arXiv: 18100011,,1 2018. 4
13

---

Published as a conference paper at ICLR 2025
Parsa Nooralinejad, Ali Abbasi, Soroush Abbasi Koohpayegani, Kossar Pourahmadi Meibodi, Rana Muhammad Shahroz Khan, Soheil Kolouri, and Hamed Pirsiavash. PRANC: Pseudo random networks for compacting deep models. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pp. 17021-17033 2023. 2, 3, 4, 8, 9, 17
Steven J Nowlan and Geoffrey Hinton. Simplifying neural networks by soft weight-sharing. Neural Computation, 4(4):473-493, 1992. 3
Mete Ozay and Takayuki Okatani. Optimization on submanifolds of convolution kernels in cnns.
arXiv preprint arXiv: 1rXiv:1610.07000, 2016. 4
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data
science. Foundations and Trends@ in Machine Learming, 1165-6)355-607, 2019. 4
Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, and Kate Saenko. Neural parameter allocation search. International Conference on Learning Representations, 2022. URL https:
//openrevi ew net/ forum? www.ooooddssrrrroooooooooooooooe iid=st
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Brandon Reagan, Udit Gupta, Bob Adolf, Michael Mitzenmacher, Alexander Rush, Gu- Yeon Wei, and David Brooks. Weightless: Lossy weight encoding for deep neural network compression. In International Conference on Machine Learning, pp. 4324-4333, PMLR, 2018. 3
Mehran Shakerinava, Motahareh MS Sohrabi, Siamak Ravanbakhsh, and Simon Lacoste-Julies Weight-sharing regularization. In International Conference on Artificial Intelligence and Statistics,
pp. 4204-4212. PMLR, 2024. 3
Noam Shazeer. Glu variants improve transformer, 2020. URL httpp:://aaxiii.rrrrr
2002.05202 20
moran shkolnik, Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bron- stein, and Uri Weiser. Robust quantization: One model rule them all. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 5308-5317. Curran Associates, Inc.. 2020. URL https::/prrceeeeee (eeeeeee0033838 (at.dda3e0.992222eee383833333000 Paper paper.pdf...
neuri iirs cc/paper_fil fileiie/////e weilee/paapar haass//ppper/000///
file/39 Beadeee3699
Julien Niklas Siems, Aaron Klein, Cedric Archambeau, and Maren Mahsereci. Dynamic pruming of a
neural network via gradient signal-to-noise ratio. In 8th ICML Workshop on Automated Machine Learning (AutoMI.), 2021.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020. 4
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference Learning Representations, 2024. URL https: : /pwpneeeeeeeeeeeeooooeeeeeeeeeee net forum? iiddidd-ppi PxoFut 3dw.. 3
Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. survey transformer compression. arXiv preprint arXiv:2402.05940 2024. 2
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An ietructiio-folloviii llama model. 2023.
https:/ //gittuuls com/ coom/tate htt//tttsuuula 1ab/
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XI. pp. 776-794, Berlin, Heidelberg, 2020. Springer- Verlaa. ISBN 978-3-030- 58600-00050801..0000 doi: 10.1007/ 45. URL https: /oii. org/10//////oo 1007 978 58611--88821110 8 _5.
14

---

Published as a conference paper at ICLR 2025
Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, and Deepak K. Gupta. Chipnet: Budget-aware pruning with heaviside continuous approximations. arXiv preprint arXiv:2102.07156. 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficien image transformers amp; distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 18-2411 Jul 2021. URL htw.h::tp///tcm.. 6, 17
h httpss////ccceeeeeee igg. www.m wttrrass/v13//ttuuroo22222 html.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. 7,7
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. In International Conference on Learning Representations, 2017. URL httpp::/
net / Eorum? id- HJGwcKelx. 3
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2023. 3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.1
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. arXiv preprint arXiv:2012.09243 2020a. 3
Jayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11555-1155155 2020b. 4
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato. and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 38087-38099, PMLR, 23-29 Jul 2023. URL https: http:://prrceeeee http//ppooeeeggae//e/ee
ml .ww.ppresss.............. w20///222222222 xi ao2...............o html. 3
Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. comprehensive study on post-training quantization for large language models. arXiv preprint arXiv:2303..8833 2023. 3
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019. 17
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 12104-12113, 2022. 1
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018. URL https:
//opppreevew..nne net.eeewwwweeeeeeeee.e.o.......e www.:/ntt/fommmmmmmmm wwwpummmddrrlddppp RR. 17
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and
Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pp.
PMLR, 2022.
3, 6
15

---

Published as a conference paper at ICLR 2025
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fin-tuning In The Eleventh International Conference Learning Representations, 2023a. URL
i d= q62uwRJjiY.
Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, al. Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023b.
A
APPENDIX
A.1 EXAMPLE CODE FOR APPLYING MCNC
Below, we include PyTorch code which shows how to create linear layer which is reparameterized using MCNC.
import torch
import torch.nn as nn
import torch. nn. funct ional as from math import ceil
class MONC MONC_linnary (nn, :
def
.innt_
self, generator, in feature,, out out_featuress bias-True)
11
12
13
Roparameterizes linear layor with MCNC.
16
Args:
17
generator (nn_uee The frozen generator network. in_features (int): Number of input features. out_reatures (int): Number of output features.
18
20
bias (bool): True, includes a bias term.
21
.*.
22
super (MCNC_Linear, self) .iff)_niiit_.. i in_features, out_features, bias) self. rew...esehttrruuirssssss... wew.ireeuuuueeeeeeeeeeereeee grad False gclf.generator generater
25
26
27
29
Get input and output dimensionality of Generator self.k self. welff.gnerrator.ggm seteerattorreet_intt_ssiizlll self.d
30
31
Calculate number of Generator inputs to the weights total www.uuunssoeeeeeeeaeeaee n_gen_inputs inputs = int (ceil (total uncompressed parans / self.d))
34
nurel (self, weight))
35
36
Initialize learnable MCNC parameters
38
alpha_init torch. zeros (n_gen_inguts, self.k)
39
self.alpha Forch.nn.Paramete.. (alpha_init, requires_grad-True)
40
beta init torch. (ones(n_alll) (n_alpha, 1)
self.beta torch. www.a.ttt..eerrreeeeeeaaaaa (beta_ gequires_grad--Tuee)
43
44
def ferwarddself, x)
15
Forward pass repa rometeriii zing weights with MCNC.
16

---

Published as a conference paper at ICLR 2025
Args: x (torch, Tensor) : Input data.
49
50
51
Returns:
T
Output of the linear layer.
53
**"
54
# Reparameterize weight with MCNC
55
gen_out = self. (eeeea gen_out. view (self. weight shappe) + selt.
self.
56
return F, F.linear (x, w, self.bias)
A.2
RECOMMENDED SETTINGS FOR MCNC
Table 10: Recommended hyperparameter defaults for MCNC.
Setting
Recommendation
Input Dimension
9
of layers
3
Width
1000
Input Frequency
4.5
Weight Initialization
U #: 1
Optimizer
Adam
Learning Rate
5- 10x larger than uncompressed model
A.3 DETAILS ON COMPRESSING IMAGE CLASSIFIERS
Generator architecture When training from scratch with MCNC, we use the same Generator configuration listed in Table 10 and set d such that we achieve the desired compression. All linear layers in our generator do not include bias so that we guarantee a zero initialization by setting our inputs to zero.
VIT hyperparameters The baseline models used for pruning are trained for 800 epochs using the AdamW (Loshchilov & Hutter, 2019) optimizer. We set batch size to 1024 with initial learning rate of 0.001 and use a cosine learning rate scheduler. We use the same augmentations oulined in (Touvron et al., 2021) for ImageNet. For all pruning methods, we search for the best learning rate in 4
- 4, le - 4, Se 5} per compression rate. We train for 20 epochs and set the parameters of the cubic sparsity scheduler 1; 1000, , 12000. For PLATON, In addition, for each compression rate we run both with and without MixUp (Zhang et al., 2018) and CutMix (Yun al., 2019) augmentations and take the best results. We use the AdamW (Loshchilov & Hutter, 2019) optimizer and a cosine learning rate scheduler. For MCNC, we train for 800 epochs with cosine learning rate scheduler using the Adam (Kingma & Ba, 2015) optimizer. Learning rates and whether Mixup/Cutmix are used for each method are shown in Tables 11 and 12. For pruning methods, we set batch size to 128 and use 256 for MCNC on ViT-Ti and 512 for MCNC on VIT-S.
set 31 0.85 and [3 0.95.
ResNet hyperparameters For all experiments, we use the same set of data augmentations in PRANC (Nooralinejad et al., 2023). Likewise, we exclude BatchNorm (loffe & Szegedy, 2015) parameters from our compression and do not consider them when computing the compression rate. When applying MCNC to ResNet architectures, we train for 400 epochs with a learning rate of 0.01 using the Adam (Kingma & Ba, 2015) optimizer. We decay the learning rate by 0.5 if the loss has not improved in 4 epochs. When applying LoRA to a convolutional layer with kernel size & and input and output channels eı and c2, reshape it be of size kc, kc2 and create low rank matrices A E kc; xr B E Brxkez, When applying LoRA, we use rank 64 as in (Koohpayegani et al., 2024) and do not attempt find the optimal rank.
A.4 ABLATION EXPERIMENTAL SETUP
Unless stated otherwise, all ablation experiments follow the same experimental setup. In general, we compress an MLP with two hidden layers and hidden size 256 compressed to only 0.2% of the original parameters. We choose such extreme compression that the problem is difficult enough
17

---

Published as a conference paper at ICLR 2025
Table 11: Hyperparameters for compressing ViT-Ti
Method
Percentage of Model
Learning Rate
Mixup/Cutmix
Size
Augmentations
Magnitude
50%
0.0005
PLATON
50%
0.0005
MCNC
50%
0.001
Magnitude
20%
0.0005
PLATON
20%
0.0005
MCNC
20%
0.001
Magnitude
10%
0.0005
PLATON
10%
0.0005
MCNC
10%
0.01
Magnitude
5%
0.0005
PLATON
5%
0.0005
MCNC
5%
0,01
Magnitude
2%
0.0005
PLATON
2%
0.0005
MCNC
2%
0.01
Magnitude
1%
0.0005
PLATON
1%
0.0005
MCNC
1%
0.01
Table 12: Hyperparameters for compressing ViT-S
Method
Percentage of Model Size
Learning Rate
Mixup/Cutmix
Augmentations
Magnitude
25%
0.0005
PLATON
25%
0.0005
MCNC
25%
0.002
Magnitude
10%
0.0005
PLATON
10%
0.0005
X
MCNC
10%
0.002
Magnitude
5%
0.0005
PLATON
5%
0.0005
MCNC
5%
0.005
Magnitude
2%
0.0005
PLATON
2%
0.0005
MCNC
2%
0.005
to variations in design decisions. Note that while the reported numbers of ~ 85% accuracy low for MNIST classification, training a linear classifier with ε regularization and sparsifying to similar number parameters results 79% accuracy. The default architecture of our generator has an input size of 9, two hidden layers of size 1, 000, and an output size of 5, 000. Note that 5,000 0.002. For each experiment, we search for the best learning rate {0.1, 0.01, 0.001) and report results averaged across three trials.
18

---

Published as a conference paper at ICLR 2025
Table 15: The effect of generator hidden size on the accuracy of MNIST.
Table 16: The effect of generator depth the accu- racy of MNIST.
Generator Width
Acc.
Accuracy w/o Residual Conn.
Accuracy w/ Residual Conn.
# of Layers
64
83.5 + 0.8
128
84.3 0.4
83.7 = 0.9
N/A 83.9 + 0.5 84.0 + 0.3
256
84.3 0.2
84.9 0.8
512
8..7
85.2 0.1
1024
84.5 + 0.5
85.2 0.8
84.0 + 0.5
2048
85.0 0.7
A.5 ADDITIONAL ABLATION EXPERIMENTS
Impact of varying and d with fixed compression rate: The ratio of k to d determines its compression rate. Here, we multiply the input and output size of the random gen- erator by increasing constant factor keeping the com- pression rate fixed. The results are presented in Table 13. It is evident that value close leads poor performance. As each generator input has an associated amplitude, the amplitude uses larger percentage of the learnable parameters for smaller values of k. As h in- creases, the amplitude uses a much smaller percentage of the parameter budget, increasing the generator's ' ability to model complex functions.
Table 13: Impact of varying generator input and output size with fixed com- pression rate on MNIST classification.
k
d
Acc.
1
1000
69.3 = 1.8
3
2000
78.4 = 0.2
7
4000
83.2 + 0.7
15
8000
84.9 + 0.2
31
16000
85.8 + 0.2
Effect of Generator weight initialization By repre- senting our manifold as a randomly initialized neural network, it is able to easily be communicated using only a random seed. However, the space of possible random initialization schemes is large, and it is not clear what method of initialization would be most ef- fective. To study this, we initialize generators where the weights are generated from both normal and uni- form distributions with multiplying the variance of the distribution by factor of c. As this multiplica- tion also implicitly controls the input frequency, we always let 1 for the first layer. We present re- sults of this experiment in Table 14. It's clear from these results that drawing weights from uniform distribution provides better performance than a nor- mal distribution. In addition, results are higher when the variance is smaller. We note, however, that these results likely intertwined with the selection of frequency in the first layer.
Table 14: The effect of generator weight initialization on the accuracy of MNIST. We vary the scale of each distribution by multi- plying the variance by c.
Initialization
Acc.
Uniform Uniform
0.5
85.1 t 0.2
1.0
84.6 = 0.0
Uniform
2.0
83.2 + 0.9
Uniform
4.0
80.6 = 1.6
Uniform
8.0
80.1 = 0.8
Normal
0.5
81.8 + 0.5
Normal
1.0
81.6 = 0.2
Normal
2.0
82.1 = 0.9
Normal
4.0
81.9 + 0.2
Normal
8.0
82.0 + 1.0
Effect of generator width and depth It's unclear how architecture variations in the generator of MCNC affect compression performance. To see the effect of generator hidden size, construct generators with three layers and vary the hidden dimension and show results in Table 15. While accuracy initially improves, it quickly saturates suggesting there not benefit infinitely increasing width. Likewise, we study the effect of depth by constructing generators with hidden size of 1000 and varying the number of layers. In addition, we experiment with the effect of residual connections as it's possible that increased depth causes our optimization to experience vanishing or exploding gradients. We present the results in Table 16. Similarly, we see an improvement when extending beyond single hidden layer.
A.6 COMPUTATION OF RECONSTRUCTION FLOPS FOR LLAMA-2 7B AND 13B
For Llama-2-7 2-7b, note that the model is composed of 32 layers. Each transformer layer can then be decomposed into 7 linear layers which we apply method to:
19

---

Published as a conference paper at ICLR 2025
1. 4 linear layers in the self-attention block each of size 4096 X 4096
2. 3 linear layers in the MLP block. Each has size 4096 11008. Note that the third layer comes from the gate the SwiGLU activation function (Shazeer, 2020)
We use rank 8 for both NOLA and method which gives 11 matrices of size 4096 8 and matrices of size 11008 x 8. Assuming we can compute the flops to generate each of these matrices then the total flops would be:
32 (11 FLOPS 8) + 3 FLOPS x 8))
For NOLA, we use 64 basis so:
PLOPS(4090 x 8) = 2 64 * 4096 = 4.19MPLOPS
FLOPS 8) 2 64 * 11008 8 11.27MFLOPS
Giving total of: 32 * (11 4.19 + 3* 11.27) = 2. 2.5GGFLPPS
For MCNC, each run of the generator requires (5 32 - 32 32 32 5000) opera- tions. The number forward passes necessary for each matrix is:
4096 8
ceil
) = 7
5000
11008 8
ceil
5000
We also need to multiply each 5000 dimensional output by single scalar, so the total number of operations is:
FLOPSS x 8) = 7 * 2 * (5 * 32 + 32 32 + 32 5000) + 7 * 5000 = 2.20MSS
FLOPPS (1108 x 8) 18 2 32 32 32 32 5000) 18 5000 5.89MFLOPS
Giving total of: 32 * (11 2.29 + 3 5.89) = 1.37GFLOPS
For Llama-2-13b, the computation is the same except:
1. there are 40 layers
2. hidden dimension is 5120
3. intermediate size is 13824
4. LoRA rank 16
20


Reviews:
[{"review_id": "MeCUC9KXSl", "reviewer": "Reviewer_97Hi", "summary": "The paper proposed a new approach to compress deep neural networks. The basic idea is to enforce the DNN parameters to lie on a k-dimension manifold for every chunk of d parameters where k << d. The proposed approach uses a generator model that maps a low-dimensional uniform distribution (k-dimension) to a high-dimensional uniform distribution (d-dimension). Evaluations on computer vision and NLP tasks demonstrate that the proposed approaches can achieve extremely high compression rates.", "questions": "Can the author clarify how does the benefit of the proposed approach over baselines translate to practical computational performance gains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ The proposed algorithm outperforms baselines in the high compression rate regions\n+ The idea is interesting and shows that there exists a set of parameters in a DNN that lies in the lower-dimension manifold.  \n+ Evaluations are thorough", "weaknesses": "The paper presents an interesting idea of compressing neural network parameters into a low dimension manifold. Instead of training a generator to produce the neural network parameters, the proposed approach can use a randomly-initialized generator to project a trainable k-dimension vector into a d-dimension vector as parameters. I find the idea very interesting. The empirical results also demonstrate that the approach can achieve better accuracy under extreme compression rates. \n\nWith that said, here are a few concerns on the work\n- The benefits of achieving extreme compression rates under the high accuracy loss are not clear to me. Using table 1's result as an example, with 1% percentage of model size, the proposed approach can achieve an accuracy of 49% compared to the baselines which are 33%-39% on ViT-Ti model. However, compared to the baseline model accuracy 76.2%, it is a huge accuracy drop (more than 20%). Under this context, one might consider using a different model variant that is smaller but achieves a higher accuracy than compressing the ViT-Ti model.  Although the proposed method can achieve better accuracy compared to baselines under high compression rates, I am concerned about how likely such advantages would be useful in practice due to the significant accuracy drop compared to the non-compressed or lightly-compressed model. \n\n- Table 4 highlights the same concern. The trainable parameters from the LoRA model are already minimal compared to the number of parameters in the LLaMA-2 7B and 13B. With the quantization from NOLA and MCNC, both of them slow down the training throughput of the model. I agree with the advantages of MCNC compared to NOLA, but neither approaches are appealing compared to running the original LoRA method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOUiY6pqOe", "reviewer": "Reviewer_5SSn", "summary": "The paper presents a new approach to parametrize the network parameters: via another generator model. Specifically, weights are split into blocks(vectors) of size $d$ (i.e., $\\mathbb{R}^d$) and parametrized to be coming from a $k$-dimensional space ($\\mathbb{R}^k$) via mapping $\\phi: \\mathbb{R}^k \\to \\mathbb{R}^d $. The mapping $\\phi$ is defined as a compact feed-forward neural network with sine activation with number of layers and hidden dimensions being hyper-parameters of choice; though for most experiments there are not more 3 layers with hidden dimensions of not more 1024. The mapping $\\phi$ is randomly initialized, and fixed. The only trainable parameters are the actual inputs to the $\\phi$, i.e., the inverted image of parameters in this lower-dimensional space. The authors demonstrate that such a simple setup allows to train from scratch and finetune various models in parameter efficient way.", "questions": "Please address weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The paper presents very simple yet powerful idea of essentially hashing high-dimensional weights into lower-dimensional manifold. Instead of hashing directly from higher to lower dimension, the proposed approach fixes the inverts of the hashing function (i.e., generator), and learns the \"hashed\" outputs via sgd. This idea has deep roots in random projections/random hashing literature, and as such I am very glad that similar approach can be exploited for the parameter efficient training of the neural networks.", "weaknesses": "Overall, I think the paper does a good job in validating the soundness and effectiveness of the proposed approach, however, I think it can be significantly strengthened:\n1. In the exposition of the idea, the actual architecture of the generator network comes very late; with actual numbers/settings coming in appendix! I think, the simplicity of the approach must be showcased from the beginning, I highly suggest to add simple pytorch code showing how easy is to plug and play with the method\n2. Similar to the previous weakness, I think authors should consider recommending some bulletproof settings that would work for most of the cases (like number of layer in $\\phi$, number of dimensions, some rules of thumb, how long to train, optimizer settings) to easy any potential adoption: at the end of the day, the ideas have very limited impact if not used in practice. \n3. Theoretical guarantees of any sorts would be welcome as well: how universal is such a generator network? can it be used to train/finetune any neural network or it has limitations?\n4. While the papers literature review is extensive, I think the connection to hashing must be explored in depth and more papers included into review (some suggestions: Structured Multi-Hashing for Model Compression,  Lossy weight encoding for deep neural network compression)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpCgIlCijQ", "reviewer": "Reviewer_dj4q", "summary": "The paper proposes a reparameterization to train neural networks which leads to a lower memory footprint for transferring and storing the parameters. The paper achieves this by re-parameterizing the network parameters to lie on a lower dimensional spherical manifold. The method uses a “generator” network for this, mapping k-dimensional vectors onto a d-dimensional sphere. The k-dimensional vectors are then trained. The paper compares this method against multiple model compression and efficient training methods, demonstrating performance gains.", "questions": "1. How was the wasserstein distance computed for Fig 2? If I understand correctly, there is no closed form expression to compute this metric in higher dimensions. \n2. Why does ViT-S have better accuracy for smaller models in Tab 1? Is it an artifact of high variance in the results?\n3. How were reconstruction GFLOPs computed for Table 4?\n4. How much slower is training MCNC as compared to baseline LoRA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Strong empirical results, with comparisons against other compression methods on various modalities and architectures demonstrating the proposed method’s effectiveness.\n2. Extensive ablation studies for each component of the method, as well as for choices of hyper-parameters, giving some intuition on how \n3. Demonstration of real world speedups", "weaknesses": "1. My main gripe with the paper is that the method is not very well motivated. For example, the paper claims that a randomly initialized generator is enough to span the entire space of weights in a higher dimension, but never shows much empirical evidence beyond an experiment in low dimensions. It is not clear if this result also holds in higher dimensions. I would have ideally liked to see a theoretical result around the expressivity of the manifold constrained reparametrization, but even an empirical investigation showing that one does not lose out on expressivity would be great.  \n2. Continuing on my previous point, from table 10, it appears that a trained generator is better for larger models. However, this result is down-played in the text description, with the narrative claiming that the improvement is \"marginal\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)