Group and Shuffle: Efficient Structured Orthogonal Parametrization
Authors: Mikhail Gorbunov
Kolya Yudin
Vera Soboleva
Aibek Alanov
Alexey Naumov
Maxim Rakhuba
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-efficient finetuning
PEFT
orthogonal
structured matrices
convolutions
Primary area: deep_learning_architectures

Abstract: The increasing size of neural networks has led to a growing demand for methods of efficient finetuning. Recently, an orthogonal finetuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it.  We then use this parametrization to modify the orthogonal finetuning framework, improving parameter efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task finetuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.

Group and Shuffle: Efficient Structured Orthogonal
Parametrization
Mikhail Gorbunov HSE University gorbunovvmikk7700mai1... com
Nikolay Yudin HSE University
Vera Soboleva AIRI, HSE University
Aibek Alanov AIRI. HSE University
Alexey Naumov
Maxim Rakhuba
HSE University, Steklov Mathematical Institute of Russian Academy Sciences
HSE University
Abstract
The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses orthogonal matrices for adapting the weights a pretrained model. this paper, we introduce new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties this class and build structured orthogonal parametrization upon it. We then this parametrization to modify the orthogonal fine-tuning framework, improving parameter and computational efficiency. We empirically validate method on different domains, including adapting of text-to-image diffusion models and downstream task fine-tuning in language modeling. Additionally, adapt construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.
1 Introduction
Orthogonal transforms have proven useful in different deep learning tasks. For example, they were shown to stabilize CNNs [Li al., 2019, Singla and Feizi, 2021] used in RNNs combat the problem exploding/vanishiin
gradients [Arjovsky et al., 2016]. Recent works OFT (Orthogonal Fine- Fine-Tuning) and BOFT (Butterfly Orthogonal Fine-Tuning) [Qiu al., 2023, Liu al., 2024b] use learnable orthogonal matrices for training instabilities and overfitting that alternative methods like LoRA [Hu et al., 2022] suffer from.
fine-tuning of neural networks. which prevents
Nevertheless, parametrization of orthogonal matrices is challenging task, and the existing methods typically lack in either computational efficiency expressiveness. Classical methods like Cayley parametrization and matrix exponential map cannot operate under low parameter budget, while Givens rotations and Householder reflections requires computing products of several matrices, which makes their use less efficient in deep learning tasks. Alternatiee approach OFT method uses block- diagonal matrix structure in attempt be more computationally efficient and use less trainable parameters. Unfortunately. this simple structure can be too restrictive. Thus, arises the problem of constructing dense orthogonal matrix while still being parameter- caffaieterefficieett. While attempting to tackle this task, BOFT method uses a variation of butterfly matrices, parametrizing orthogonal matrices as a product of several matrices with different sparsity patterns, enforcing orthogonality each of them. This parametrization is able to construct dense matrices while still being parameter-efficien However it requires to compute product of multiple matrices (typically up which can be computationally expensive. In this paper, we aim to overcome these issues and build dense orthogonal matrices in more efficient way.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We present a novel structured matrix class parametrized by an alternating product of block-diagonal matrices and several permutations. Multiplying by these matrices can be seen as a consecutive application of independent linear transforms within certain small groups and then shuffling the elements between them, hence the name Group-and-Shuffe Shufffee matrices (or GS GS-matrices for short). This class generalizes Monarch matrices [Dao et al., 2022] and with the right permutation choices, is able to form dense orthogonal matrices more effectively compared to approach proposed in BOFT, decreasing number of matrices in the product as well as the number of trainable parameters. We build efficient structured orthogonal parametrization with this class and use to construct new parameter-efficiee fine-tur fin-ttning method named GSOFT.
Our contributions:
We introduce a new class of structured matrices, called GS, that more effective at forming dense matrices than block butterfly matrices from the BOFT method.
Using GS-matrices, we propose an efficient structured orthogonal parametrization, provide theoretical insights and study its performance in the orthogonal fine-tuning framework.
We adapt our ideas for convolutional architectures, providing framework to compress and speed-up orthogonal convolution layers.
2 Orthogonal Fine-tuning
Orthogonal Fine-tuning method (OFT) introduced in [Qiu et al., 2023] is a Parameter-Efficcin (PEFT) method which fine-tunes pre-trained weight matrices through learnable orthogonal block-diagonal matrix. Some the properties that make orthogonal transforms desirable are preservation of pair-wise angles of neurons, spectral properties and hyperspherical energy. More precisely, OFT optimizes an orthogonal matrix Q E Raxd for a pre-trained frozen weight matrix Wo E dx and modifies the multiplication y = (W") T x to (QW)) x. Note that the identity matrix I is orthogonal, which makes it a natural initialization for Q. OFT uses block-diagonal structure for Q, parameterizing it as
Q = (diag(Q), .............................Q.. Qr),,
where Q, E bx are small orthogonal matrices and br d. Orthogonality is enforced by Cayley parametrization, i.e.
Q = (I+ KI) K,)
where K₁ are skew-symmetrie: K, K, This ensures orthogonality of Q; and, hence, of Q.
Nevertheless, block-diagonal matrices can be too restrictive, as they divide neurons into independent groups based on their indices. This motivates the construction of dense parameter-efficien orthogonal matrices. To address this problem, the Orthogonal Butterfly method (BOFT) introduced [Liu et al., 2024b]. BOFT uses bottebbu structure to construct Q. Essentially, Q is parameterized as product of m orthogonal sparse matrices:
Q = Bm, Bm-1....B By.
Each matrix B, is block-diagonal matrix up permutation of rows and columns, consisting of r block matrices of sizes x b. Similarly to OFT, the orthogonality is enforced by the Cayley parametrization applied to each block. However, BOFT method has some areas for improvement as well. To construct a dense matrix, BOFT requires at least
m 1 [ro
matrices. For example, the authors of BOFT use m = or 6 matrices in the BOFT method for fine-tuning of Stable Diffusion [Rombach et al., 2022]. Large amount of stacked matrices leads to significant time and memory overhead during training. There is also room for improvement in terms of parameter-efffciinnge To overcome these issues, we introduce new class of structured matrices that we denote GS ggroupannd-suuffee that generalizes Monarch matrices [Dao et al., 2022, Fu et al., 2024] and show how to use this class to construct parameter-efffcient orthogonal parametrization. Similarly to BOFT, our approach uses block- matrices and permutations, but requires only
flog.(r) (r)
m
matrices of the same size to construct dense matrix. See details in Section 5.2. The reduced requirements on m allow us to use m = 2 in experiments to maximize computational efficiency, while still maintaining accurate results.

---

3 GS-matrices
Our motivation within this work is to utilize orthogonal matrices of the form:
A= (R.PR)
(1)
where matrices and are block-diagonal matrices with r blocks of sizes b and PL, P, PR are certain permutation matrices, c.g. Pt P .R = I in the orthogonal fine-tuning setting and PR- P, PL for convolutional architectures. Note that although the case Pt _ PR =I resembles Monarch matrices [Dao et al., 2022], they are unable to form such a structure, e.g. with equal-sized blocks in L. and R. The issue that the Monarch class has constraint that interconnects the number of blocks in matrix L and the number of blocks in matrix R (see Appendix C for details). Moreover, Monarch matrices have been considered with orthogonality constraints.
To build matrices of the form (I), we first introduce a general class of GS-matrices and study its properties. We then discuss orthogonal matrices from this class in Section
3.1 Definition of GS-matrices
Definition 3.1. m x n matrix GS P, Pn) class with kr, kn blocks and block sizes bt x bi, bR x bR if
A Pr(LPP))Rr
where diag (L1, Loz...., Lkz), L, E [R xb2, R = diag(Bis Ba, Rz..... . Rkg). Rg E x
PL, P. Pn are permutation matrices and b7, Ky b/2 kn s, bb. KL m, R kn
2
In practice, we fix Pr. P, Pn. depending the application and only make matrices L, R subject for change. GS-matrices are hardware-efficient. types of operations that can implemented efficiently: multipli- cations by alock-diagonal matrices and permutations.
they are parametrized by two simple
L
P
R
Let us also illustrate a forward pass Ax LP RR for matrix A EGS(I, P, [) building block for the more general class with two additional permutations. The first operation y = Rx consists of several cully-connected layers, applied individually to subgroups ær see Figure The next multiplication LPy ensures that these groups interact with each other. Indeed, the permutation matrix P shuffles the entries of y into new subgroups. These subgroups are then again processed by number of fully-connected layers using L. This motivates the naming for our class of matrices: Group-and-Shupp short.
I(PRr) P(Rx)
Rx
Figure GS P. matrices
GS for
with b = by = 3, b b = 2, kL = kn Edges between nodes denote nonzero weights.
Another useful insight on these matrices is that the class GS(I, consists of block matrices with low-rank blocks. The permutation matrix P is responsible for the formation of these blocks and defines their ranks (note that rank may vary from block to block). The result below formally describes our findings and is key to the projection operation that we describe afterwards.
Proposition 1. Let be a matrix from GS(I,P,J) with a permutation matrix P defined by the function a :(0..... (o.....n--1 1] > {0..... .....,, 1). Let {v]]- -be be the rows of the blocks Ri....; Rkn {u,) the columns of the blocks Li...., Lx, in the consecutive order. Then the matrix A can be written as a block matrix with kr, x KR blocks using the following formula for each block Aui
Ak
Σ uo
=k
1=k
Note that we zero-indexing for this proposition for simplicity of formulas.
3

---

U2
u20
Figure 2: Illustration of Proposition 1 that provides block low-rank interpretation of (T, P)
matrices. The matrix R contains 2 blocks and matrix L contains blocks.
Let us illustrate this proposition in Figure 2. We consider GS P, P.T) 7) with ky = and kn = blocks in L and R and with the block sizes 3 3 and 6 x respectively. Let consider the leading block Aso of the size 3 x 6. According to Proposition 1, Aoo = uooz + uzv Indeed, let us take closer look, e.g., at the term uyuz In the permutation matrix P. we have a nonzero element in the position (2, 4) as i and (4) 2. Therefore, select the third column u2 in L1 and the fifth row 2.j in R1. This leads to adding rank-one term u2t24 to Aoo as we see in the formula above.
Another direct corollary from Proposition is projection operation π. mxxx + GS(Pt, P. PR) that satisfies:
((A) arg ||A-B] BBBB,
BEGS(PL.PPPA)
where - -I|F is the Frobenius norm. Thanks to the block low-rank representation of matri- from GS PL, P. PR), the projection is simply constructed using SVD) truncations of the blocks (PL APR R ky k2 and is summarized in Algorithm 1.
Algorithm 1 Projection (T-) of A onto GS(PL,
Input: A, Pt, P.Pr.
Return: T.,
-...ky for 1 =
kr do
for h2 1... kn do
Compute SVD of (PT APP) & UΣVT;
Set r = Th At rank of block determined by P:
Take U, = U[: :l. Σ, = E[: r, r], Vr Vl:r,
Pack columns U, م into Lb, and rows of / V, into Ry, according to P:
end for
end
4 Orthogonal GS(PL, P. PR) matrices
In this section, we study the orthogonality constraint for the GS(Pr, P, PR) obtain structured orthogonal representation. This is one of the main contributions of our paper and we utilize this class in all the numerical experiments. Since we are interested only square orthogonal matrices, we additionally assume that m = n and 6) = b2 by: b/ by = bg- Similarly to parametrizations in OFT and BOFT, natural way to enforce orthogonality of GS(PL, P, Pr.t-matrices orthogonality of each block of L and R. This indeed leads an orthogonal matrix since permutation matrices are also orthogonal as well as product of orthogonal matrices. However, is not immedi- ately obvious that there exist no orthogonal matrices from GS(PL. P, Pn) that cannot be represented this way. Surprisingly, we find that such a way to enforce orthogonality indeed sufficient for covering of all orthogonal matrices from GS (1 PR).
enforce
Theorem 1. Let A be any orthogonal matrix from GS(Pr, P, PR). Then, A admits Pr(LPR)PR representation with the matrices L, R consisting of orthogonal blocks.
Proof. Matrices Pr, PR are orthogonal as they are permutation matrices. It means that it is sufficient to prove theorem in the case when A is from GS(I, P, P,I), which means that can use low block-

---

rank structure interpretation from Proposition 1, Consider a skeleton decomposition of the blocks A, U,₁Va U/ R XTis, Viz R such that U3JUU, Iru, (this can be ensured, e.g.. using the QR decomposition). Then
U1 Vikee
A=
UKL.1 k2.1
UKL Ke
Take the j-th block-column of A. Since A is an orthogonal matrix, we get:
UisVis
(Vi,UT) VEENSKReD)
Ion
Ukins Viens
Multiplying matrices in the hhs., we get V1,, Ui, t..... ......... Voui kid U kid = Lbx- Since Unj Ui = Lrg conclude Vi.jVi +.... VKL Vir.s lug. This implies that (V1.j is an orthogonal matrix. Note that if we now parameterize A LPR with the matrices Vi packed into R and Us packed into L. then (V1, permutation of rows. Therefore, every block in R is orthogonal matrix. Since we now proved that vij Vi₃ I, we can use same the derivation for the rows of A and conclude that blocks L. also orthogonal.
Vk...)
VkL.) is exactly the j-th block matrix in R up to
L
GS(PmL 15((m11...... Pi) matrices
In this section we describe an the extension GS-matrices that uses more than two block-diagonal matrices and show that with the right permutations choices GS-matrices are more effective than block butterfly matrices in forming dense matrices. Here by dense matrices we imply matrices that do not contain zero entries at all.
Definition 5.1. A is said to be in GS(P
P.) if
= Pm41 II (B.P.)
1
where each matrix B; is block-diagonal matrix with k; blocks of size b) x b?, matrices Pa are permutation matrices and by k, bţ+1 K4+1-
2 described in Section 3, we may use orthogonal blocks in B,, i i= 1,...., -m + to obtain orthogonal matrices. However, it is not clear if an analog to Theorem correct in this case as well.
Remark 1. Similarly to the
Remark 2. For each of the classes of Block Butterfly matrices [Chen al., 2022], Monarch matrices [Dao et al., 2022] and order-p Monarch matrices [Fu et al., 2024], there exist permutation matrices Pm+t,..., Pi such that GS(Pm=1,... P1) coincides with a respective class. Indeed, Monarch matrices have the form of alternating specific size constraints and sparse matrices in the product of Block Butterfly matrices can be easily transformed to block-diagonal matrices with permutations of rows and columns.
matrices and permutations with some
5.1 Choosing permutation matrices
We suggest using the following matrices with A: k; for P:- Note that this is efficient for forming dense matrices as follows from the proof of Theorem This is by contrast to the permutations used in [Fu et al., 2024] that restricted to particular matrix sizes.
Definition 5.2 ([Dao al., 2022]). Let P(k..) be a permutation matrix given by permutaatin {0, 1, 1}:
n (1)) = mod k)
5

---

Applying this permutation to vector can be viewed as reshaping input of size into an le x matrix in row-major order, transposing and then vectorizing the result back into a vector (again in row-major column). We provide several examples of such permutations in Figure 3.
O
P(a12)
Preua)
P(az)
Figure Illustraion of P(1,12) permutations for E (3,4,6,2)
5.2 Comparison to block butterfly matrices and BOFT
Block Butterfly matrices were introduced [Chen et al., 2022] and are used to construct orthogonal matrices in the BOFT method. Block Butterfly matrix class is special case of higher-order GS- matrices with k r and bj b3 = 2s and certain permutation choices. However, we sub-optimal for construction of dense matrix and
argue that the choice of these permutations using permutations from Definition 5.2 more effective. When using block-diagonal matrices with blocks, block butterfly matrices need + [log2 (r) matrices to construct dense matrix. For GS matrices we have the following result.
Theorem 2. Let ki = b, be = Then using m + [logs (r) is sufficient for the class GS(Pr,, Pir br..br).........
((r,br), PR) to form dense matrix for any Pt,, PR. Moreover, the choice of P2 =..... = Pm = P(r,br) is optimal in the sense that all matrices from GS(Pm+1..... P1) contain zero blocks for any integer m < 1 + [r]] and any permutations PI..... P Pm+1-
Proof. See Appendix D.
For example, let us consider case of constructing dense orthogonal matrix of the size 1024 x 1024. Suppose also that we use block matrices with block size 32. Constructing a dense matrix with Block butterfly matrices, which leads to 6 x 32 parameters in
Butterfly matrices requires 1 log, the representation. GS(Pt, PR) matrices with P = 1032,1024) only need two matrices to construct a dense matrix yielding 2 32 parameters. The GS(P,, f,, P. PR) parametrization is also naturally more computationally efficient as fewer number of multiplications is both faster and requires less cached memory for activations.
6 Applications
6.1 Orthogonal fine-tuning with GS(PL, P, PR) (GSOFT)
We utilize the pipeline of OFT and BOFT methods with the exception of parametrizing Q with orthogonal permuted GS(,L P, Pn) matrices. In particular, for parametrization of Q E R we utilize the GS(PT P. I) class, i.c. Q = PT LPR, where L = diag L1. L,), L, E gx R diag( (R, Ri.... R2), R2 E Rxxb, For consistency, we use the same notation for the number of blocks and block sizes as in BOFT and OFT methods. We use P(r as a permutation matrix P. To enforce orthogonality, we parameterize each block in matrices I., with the Cayley parametrization. We initialize Q an identity matrix by initializing each block be an identity matrix. Additional

---

techniques like magnitude scaling and multiplicative dropout that are used in OFT and BOFT can be utilized the same way in our method, though we only use scaling our experiments. Note that likewise in OFT, BOFT weights of the matrix Q can be merged with the pretrained weight W producing no inference overhead.
6.2 Two-sided orthogonal fine-tuning (Double GSOFT)
Consider SVD decomposition of matrix 8 UΣV Applying orthogonal fine-tuning, we get W" = (QU)EVT which is SVD decomposition for the adapted weight W". This shows that can only change left singular vectors with the standard orthogonal fine- fin-ttnning paradigm. At the same time, the LoRA method modifies both matrices U and V. Moreover, recent papers [Meng et al., 2024, Li et al., 2023] show that initializing matrices A, with singular vectors can additionally boost performance of LoRA. This motivates extension of orthogonal fine-tuning method, that can adapt both matrices U and V. We introduce simple approach that multiplies pre-trained weight matrices from both sides, rather than This method modifies forward pass from (W") ar
(Qu W Q
Where Qu and Qy are parametrized as orthogonal GS-matrices. cases where BOFT utilizes 5-6 matrices, we can leverage the fact that our method uses only 2 and adapt both sides while still using less matrices and trainable parameters than BOFT.
6.3 GS Orthogonal Convolutions
Recall, that due linearity of a multichannel convolution operation, we can express the convolution of tensor X gea xhx with kernel L E Ro Xcin xkxk L X in terms of matrix multiplication [Singla and Feizi, 2021]:
Lo,0
Lo -1
Y= L*X
+ vec(Y)=
vec(X),
(2)
L,0 1.0
Legue L,0n
where Li, is doubly Toeplitz matrix, corresponding to convolution between i-th and j-th channels and vec(X) (X) is a vectorization of tensor into a vector in a row-major order. Thus, the convolution essentially block matrix, where each block represents standard convolution operation. Using this block interpretation (2), we may apply the concept of GS matrices to convolutional layers as well. Considering each convolution between channels as element of block matrix, we can set some of these blocks to zero, obtaining some additional structure. Thus, we can construct block matrix which has block-diagonal structure, corresponding to grouped convolution (further, in all equations we will denote it as GrConv). Then, defining ChShuffle as permutation of channels, like in [Zhang al., 2018], we obtain structure, which similar GSOFT, defined in Section 6:
Y
(ChShufflee (X)))).
The proposed GS convolutional layer shuffles information between each pair of input channels and requires less parameters and FLOPs during computations. In this example we can also choose permutations of channels and change kernel size. This convolutional layer can treated as GS(Pm+1,.... Pi) matrix in vectorized view, that is why choosing permutations between con- volutional layers also very important for information transition properties. In Appendix we explain the choice of ChShuffle operation.
We can use the proposed layer to construct orthogonal convolutions (transformations with an or- thogonal Jacobian matrix) similarly to skew orthogonal convolution (SOC) architecture, that uses Taylor expansion of matrix exponential. One major downside of methods such as SOC and BCOP [Li et al., 2019] is that they require more time than basic convolution operation. For instance, in SOC method, one layer requires multiple applications of convolution (6 convolutions per layer). In our framework, we propose a parametrization of convolutional layer, in which imposing an orthogonality to convolutions has fewer number of FL.OPs and parameters thanks to the usage of grouped convolutions.
7

---

Let us discuss in more details how SOC works and the way we modify it. In SOC, a convolutional filter parametrized in the following way:
L
M ConvTranspose(MM)
where M E X XXout X is an arbitrary kernel and the ConvTranspose is the following operation:
Con/Transpose(M1 (..... M
This parametrization of filter L makes the matrix from Equation 2 skew-symmetric. As matrix exponential of skew-symmetric matrix is an orthogonal matrix, SOC the authors define convolution exponential operation, which is equivalent to matrix exponential in matrix-vector notation:
Definition 6.1. [Singla and Feizi, 2021] Let X E Rexhxx be an input tensor and L E Rexexxxx be a convolution kernel. Then, define convolution exponential *e as follows:
+
L 2
L X - x
2!
where L x² X is a convolution with kernel applied times consequently.
As mentioned above, with proper initialization we get convolutional layer with orthogonal Jacobian matrix. Using the parametrization of convolution layer from the Equation and substituting there two grouped convolution exponentials (e.g. in parametrization
have the same convolution exponential, but we have grouped convolution instead of basic one) with the parameterized kernel:
GrExpConv, (CtShufflehhhhhhtccccc (GrExpCon,
(Casaf (ChShuffle;
In our experiments we tried different layer architectures and we found that making kernel size of the second convolution equal to speeds up our convolutional layer, maintaining quality metrics. Thus, if convolutional layer consists of two grouped convolutional exponentials, the second convolutional exponential has kernel_siee
Experiments
All the experiments below conducted on NVIDIA V100-SXM2-32Gb GPU. We ran all the experiments within ~200 GPU hours.
Source code is available at: https:/ hatp:::///thhbb httpub/cccosslooo/grrop..aaaaad shuffle
7.1 Natural language understanding
We report result on the GLUE [Wang et al., 2018] benchmark with RoBERTa-base [Liu al., 2019] model. Benchmark includes several classification tasks that evaluate general language understanding. We follow training settings of [Liu al., 2024b, Zhang al., 2023]. We apply adapters for all linear layers and only tune learning rate for all methods. Table I reports best results on the evaluation set from the whole training. LoRA, OFT and BOFT are implemented with PEFT library [Mangrulkar et al., 2022]. GSOFT method outperforms OFT, BOFT and also have slight edge over LoRA. Note that even though skew-symmetric K theoretically matrix only requires approximately half the parameters of a full matrix, in practice it is parametrized as K A- A for the ease of computations. However, after fine-tuning, one can only save upper-triangular part of K. Doing this, orthogonal fine-tuning methods become approximately 2 times more efficient in terms of memory savings.
7.2 Subject-driven generation
Subject-driven generation [Ruiz et al., 2023, Gal al., 2022] is an important and challenging task in the field of generative modelling. Given several photos of particular concept, we want to introduce it the diffusion model so that we can generate this particular object in different scenes described by textual prompts. The main way to do this is to fine-tune the model. However, the large number of fine-tuning parameters together with the lack of training images make the model prone to overfitting, i.e. the model reconstructs the concept almost perfectly, but starts to ignore the textual prompt during generation. To solve this problem and stabilize the fine-tuning process, different lightweight parameterizations [Qiu et al., 2023, Liu et al., 2024b, Hu et al., 2022, Tewel et al., 2023, Han et al.,

---

Table Results on GLUE benchmark with RoBERTa-bare base model. We report Pearson correlation for STS-B B, Matthew's 's correlation for CoLA and accuracy for other tasks. # Params denotes number of trainable parameters
Method
Params
MNLI
SST-2
CoL.A
QQP
QNLI
RTE
MRPC
STS-B
ALL
FT
125M
87.62
94.38
61.97
91.5
93,06
80.14
88.97
90,91
86.07
LoRA.=s
1.33M
87.82
95.07
64.02
90.97
92.81
81,95
88.73
90.84
86.53
OFTi=16
1,41M
87.21
95.07
64.37
90.6
92.48
79.78
89.95
90.71
86.27
BOFT
1.42M
87.14
94.38
62.57
90.48
92.39
80. 14
88.97
90.67
85.84
GSOFT
1.42M
87.16
95.06
65.3
90.46
92,46
81.95
90.2
90.76
86.67
Table 2: Results on subject-driven generation. # Params denotes the number of training parameters in each parametrization. Training time is computed for 3000 iterations on single GPU V100 in hours.
Full
LoRA
BOFT
GSOFT (Ours)
Double GSOFT (Ours)
Model
rank
4
32
128
32, 4
32,6 6
16, 5
32
16
8
64
32
Params
99.9M
0.8M
6.6M
26.6M
20.4M
33.8M
13.6M
27.1M
6.5M
13.0M
25.9M
Training time
1.3
1.3
1.3
1.3
2.0
2.2
2,3
1.5
1.6
1.8
1.7
2.0
1.8
CLIP-I
0.805
0.805
0,819
0.803
0.796
0.789
0.803
0.783
0.815
0.783
CLIP-T 0,212
0.246
0.236
0,223
0.244
0.234
0.223
0.256
0.245
0,227
0.256
0,242
0.225
2023] and regularization techniques [Ruiz et al., 2023, Kumari al., 2023] widely used in this task. Therefore, we chose this setting to evaluate the effectiveness of the proposed orthogonal parameterization compared other approaches.
We use StableDiffusion [Rombach et al., 2022] and the Dreambooth [Ruiz et al., 2023] dataset for all experiments. The following parameterizations v and out.0 layers all cross- and self- attentions of the UNet are trained). LoRA [Hu et al., 2022] and BOFT [Liu al., 2024b] applied to the same layer. We use GSOFT parameterization and two-sided orthogonal GSOFT (Double GSOFT) applied to the same layers as baselines. For a more comprehensive comparison, consider different hyperparameters for the models, adjusting the total number of optimized parameters. More training and evaluation details can be found in Appendix E,
considered as baselines in this task: full (q. k.
CLIP image similarity, CLIP text similarity and visual comparison for this task are presented in Table 2 and Figure 4. As the results show, GSOFT and DoubleGSOFT are less prone to overfitting compared to the baselines. They show better alignment with text prompts while maintaining high level of concept fidelity. Furthermore, both methods with optimal hyperparameters are more efficient than BOFT and comparable to LoRA and full parameterization in terms of training time. See Appendix for more visual and quantitative comparison.
7.3 GS Orthogonal Convolutions
Following [Singla and Feizi, 2021], we train LipConvnet-n CIFAR-100 dataset. LipConvnet-n is 1-Lipschitz neural network, i.e. neural network with Lipschitz constant equal to 1, his property provides certified adversarial robustness. LipConvnet uses orthogonal convolutions and gradient preserving activations in order to maintain I-Lipschitz property.
LipConvnet-n architecture consists of 5 equal blocks, each having 5h skew orthogonal convolutions, where the last convolution at each level downsamples image size. We replace the skew orthogonal convolution layer with the structured version using GSorthogonal convolutions and test it in the setting of [Singla and Feizi, 2021], I, using the same hyperparameters (learning rate, batch size and scheduler stable during testing). In layers where we have two GrExpConv, the second convolution has kernel size equal to
We also use modified activation function Minerrmue instead of MaxMin), which uses differ- ent pairing channels. This makes activations aligned with the ChShuffle operation and grouped convolutions. The choice of permutation for ChShuf fle also slightly differs from permutations defind in Definition 5.2 because of the interplay between activations and convolutional layers. We provide definitions and intuition regarding activations and permutations for ChShuffle in Appenix F.

---

LoRA 32
BOFT 32,
GSOFT 32
Double GSOFT 64
Concept
4
V on cobblestone street"
"a purple V
"a V in purple wizzard outfit"
Figure 4: Subject-driven generation visual results on 3000 training iterations.
Table 3: Results of training LipConvnet-15 15 architecture CIFAR-100. (a, b) in "Groups" column denotes number groups in two grouped exponential convolutions (with kernel sizes and 1). (a, -) corresponds to only one GS orthogonal convolutional layer. Before each grouped layer with k groups use a ChShuffle operator.
Conv, Layer
Params
Groups
Speedup
Activation
Accuracy
Robust Accuracy
SOC
24.1M
MaxMin
43.15%
29.18%
GS-SOC
6.81M
(4,
1.64
43.48
29.26 %
GS-SOC
8.91M
(4,
1.21
MixMuntemutedd
43.42%
29.56%
GS-SOC
7.86M
(4
1.22
MaxMinPermuted
42.86%
28.98%
GS-SOC
7.3M
(4,4)
1.23
MaxMinPermuted
42.75%
28.7%
8
Concluding remarks
In this paper, we introduce a new class of structured matrices, called GS-matrices, build a structured orthogonal parametrization with them and use them in several domains within deep learning applica- tions. However, we hope that our orthogonal parametrization can be adapted to different settings in future (including tasks outside of deep learning), makes orthogonal parametrizations less of computational burden. GS-matrices without orthogonality constraints is another promising direction.
Limitations
Although our method for orthogonal fine-tuning is faster than BOFT, still slower than LoRA during training. Additionally, since our parametrization provides trade-off between expressivity and parameter-efficiency, might be unable to represent some particular orthogonal matrices, which might be required in other settings apart from parfmeter-efffcieet fine-tuning.
10

---

10 Acknowledgments
The article was prepared within the framework of the HSE University Basic Research Program. This research was supported in part through computational resources of HPC facilities at HSE University [Kostenetskly al., 2021].
References
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pages 291-301. PMLR, 09-15 Jun 2019, URL http::ttt/////ooeeee http///rooeeedeeeeeeeeeee ngs. mlr. press/ /v7/ aa//annlll 199a. html.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120-1128. PMLR, 2016.
Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations (ICLR), 2022.
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Re. Monarch: Expressive structured matrices for efficient and accurate training. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4690- mlr www..press/vlle dao22a. html.
4721. PMLR, 17-23 Jul 2022. URL settps:/proceeeee
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.106650 2022.
Dan Fu, Simran Arora, Jessica Grogan, Isys Johnson, Evan Sabri Eyuboglu, Armin Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré. Monarch mixer: A simple sub-quadratic gemm-based architecture. Advances Neural Information Processing Systems, 36, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 01611, 2022.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7323-7334, 2023.
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning, pages 1969-1978. PMLR, 2018.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:://// http:///ppeeeee.neeteeeee net.://rmffff/id nZeVKeeFYf9.
Stephanie Hyland and Gunnar Rätsch. Learning unitary operators with help from u (n). In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.

---

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. HPC resources of the higher school of economics. In Journal of Physics: Conference Series, volume 1740, page 012050. 2021.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: // //arxiv... crrxiv.org///eoo abs/ abs/1412. 6553.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104. 000.8061, 2021.
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Jörn-Henrik Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, 32, 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 11665/////0221..cc-oon......// URL https:t////.. http://acaaathhooooo org/2021
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-wwar quantization for large language models, 2023.
Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09333, 2024a.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue. Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-effiiient orthogonal finetuning via butterfly factorization. In The Twelfth International Conference on Learning Representations, 2024b. URI. https://openrevier htt//forum?id-7llggaaGyy
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692,
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-an parameter-efficien fine-tuning methods. https://githe thub. com/ huggingface, ace//fft 2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Bernd Prach, Fabio Brau, Giorgio Buttazzo, and Christoph H Lampert. 1-lipschitz layers compared: Memory speed and certifiable robustness. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 24574-24583, 2024.
12

---

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Thhirty-seeenhh Conference on Neural Information Processing Systems, 2023. URL https: //pppnreview..o////////// wtt.:/n.weee/ooomaeoeeooo0000
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine Pmlr, 2021.
learning, pages 8821-8833
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pages 10684-1069)) 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusson models with deep language understanding. Advances in neural information processing systems, 35:36479-36494 2022.
Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9756-9766. PMLR, 18-24 Jul 2021. URL attts::/proceett
mlr. wtppps/ssssvee/p v/39//33/////// w13.///ngl/ln//ee/ singla2is aatgll222ahtm.... html.
Sahil Singla, Surbhi Singla, and Soheil Feizi. Improved deterministic 12 robustness on cifar- and cifar-100. arXiv preprint arXiv: 100002000 2021.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to- image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning. pages 3570-3578. PMLR, 2017.
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 1rXiv:1804.07461, 2018
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943-15953, 2023.
Xiaojun Xu, Linyi Li, and Bo Li. Lot: Layer-wise orthogonal training improving 12 certified robustness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 18904-18915. Curran As- sociates, Inc., 2022. URL https: ctttopee/ppooeeedinggs neurips cc hwtpppppe...... lapppappleeeeeeee www.ffilspppee /21ee//paper/202// /tlo:/7782774ff02026662162222222
Paper- Ceaerr-Coofeennce.. pdf.
Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor-train adaptation for Aitra-low-parame fine-tuning of large language models, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference Learning Representations, 2023.
13

---

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018
Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu. Enhancing detail preservation for customized text-to-imas to-image generation: A regularization sererattto---eeeeeeee approach. arXiv preprint arXiv:2305.13552 2023.
14

---

A Related work
Garameter-Efficien Fine- Tane-Tuning (PEFT) With the growth of model sizes, end-to-end training became unavailable for those who want adapt powerful architectures for specific tasks, as even full fine-tuning became too expensive. This problem sparked research in the direction of parameter- efficient fine-tuning methods, including methods that focus on prompt tuning [Lester al., 2021, Li and Liang, 2021] and adapter tuning (e.g. [Houlsby et al., 2019, Karimi Mahabadi et al., 2021)), which include LoRA [Hu al., 2022] and its variations [Meng et al.. 2024, Zhang et al., 2023, Liu et al., 2024a, Dettmers et al., 2024, Li et al., 2023), that inject learnable low-rank matrices as an additive injection the weights of pretrained models. OFT [Qiu et al., 2023], BOFT [Liu al., 2024b] and our method use similar approach to LoRA, but learn multiplicative injection rather than an additive one.
Structured sparsity Structured sparsity is an approach that replaces dense weight layers with different structured ones, such as matrix factorizations or tensor decompositions in order compress or speed-up models [Dao et al., 2022, Chen et al., 2022, Novikov et al., 2015, Lebedev et al., 2015]. Some of these techniques were also adapted to PEFT methods in works like [Karimi Mahabadi et al., 2021, Edalati et al., 2022, Yang et al., 2024] or BOFT [Liu et al., 2024b] method, that utilizes variation of butterfly matrices as a parametrization for parameter-efficient orthogonal matrices, imposing orthogonality to each butterfly factor. See details in Section 2. Monarch matrices [Dao et al., 2022, Fu et al., 2024] are most relevant to our work as our proposed matrix class is their generalization that utilizes similar structure.
Subject-driven generation The emergence of large text-to-image models [Ramesh et al., 2022, 2021, Saharia et al., 2022, Rombach et al., 2022] has propelled the advancement of personalized generation techniques in the research field. Customizing based on multiple input images presents a key challenge. Various methods [Ruiz et al., 2023, Gal et al., 2022, Kumari et al., 2023, Han al., 2023, Qiu al., 2023, Zhou et al., 2023, Wei al., 2023, Tewel et al., 2023] have been proposed to address this challenge, requiring either extensive fine-tuning of the model as a whole [Ruiz al., 2023] or specific parts [Kumari al., 2023] accurately reconstruct concept-related training images. While this facilitates precise learning of the input concept. it also raises concerns regarding overfitting, potentially limiting the model's flexibility in generating diverse outputs in response to different textual prompts. Efforts to mitigate overfitting and reduce computational burden have led the development of lightweight parameterization techniques [Qiu et al., 2023, Liu et al., 2024b, Hu et al., 2022, Tewel et al., 2023, Han et al., 2023] such as those proposed among others. These methods aim to preserve editing capabilities while sacrificing some degree of concept fidelity. The primary objective is to identify parameterization strategies that enable high-quality concept learning without compromising the model's ability to edit and generate variations of the concept. Our investigation indicates that the orthogonal parameterization approach we propose represents significant step towards achieving this goal.
text-to-image model to generate specific concepts
Orthogonal transforms In papers [Vorontsov et al., 2017, Hyland and Rätsch, 2017, Helfrich al., 2018] authors work in the setting of dense matrices and try parameterize essentially the whole manifold of orthogonal matrices. For n X n matrices it requires computing inverses or exponential maps of n x n matrices at every optimization step. This takes O(n²) time, which can be computationally challenging for larger architectures. Moreover, such parametrization utilizes ((2)) trainable parameters, which makes it inapplicable for PEFT (see the OFT paper [Qiu et al., 2023] for more details). Our proposed method is different in a sense that it provides trade-off between expressivity (describing only a subset of the orthogonal manifold) and efficiency.
In [[.i et al., 2019, Singla et al., 2021] authors discuss main issues of bounding of Lipschitz constant of neural networks and provide Gradient-Norm -ree--reer-n (GNP) architecture in order avoid van- ishing of gradients while bounding Lipschitz constant. The authors propose a specific convolutional layer (Block Convolutional Orthogonal Parametrization) which Jacobian is orthogonal, also providing orthogonal activations with Lipschitz constant equal to 1. These constraints guarantee that the norm of the gradient will not change through backward pass. In other works [Singla and Feizi, 2021, Singla et al., 2021] authors provide a modification of the orthogonal convolutions (Skew Orthogonal Convolution) in terms of hardware--ffficiencyy Authors provide neural network architecture where each layer is I-Lipschitz and make comparison between these two convolutional layers. The work [Li et al., 2019] was outperformed by the SOC method that we utilize (there is a comparison in the SOC method [Singla and Feizi, 2021]). In [Xu et al., 2022], the authors utilize periodicity of
15

---

convolution and their padding is not equivalent to widely-used pero-padded convolution. Survey [Prach et al., 2024] suggests that [Xu al., 2022] and other I-Lipschitz architectures yield worse robustness metrics than SOC, which we use as a baseline in our paper.
B Proof of Prop. 1
Proof. Let R = PR. R' can be viewed as a block matrix with ky x kR blocks of sizes by x b L can be viewed block matrix with 2y x ky blocks from which only diagonal non-zero. The A can be written in the following form:
Ao.o
Ag, ) Lo
Ro,0
Bo-
Akg-1.0
Ak,-1, 1,kR
Lkk-1
R 11
Using block matrix product formulas, we get:
Ak
Lik Hi .22'
We can now rewrite Lxx, Ri, product in terms of their columns and rows:
Ta Rk = (11...92)
Dure
Columns of T.,, are just vectors uj such that 1 J= kn- Let examine the rows of , Since R' is matrix formed by permuting the rows of block-diagonal matrix R. R R can only contain rows that were in the Rbg before permutation. Formally, this means that R can only contain vector-rows u! such that kRn k2- Additionally, rows after permutation should get into the -blokkck row. That implies a(i Kz. = ky Other rows of k, are zero-rows. Notice that in (4) non-zero rows r represented by un will match exactly with columns (o(()) that represent 4. Keeping only non-zero terms in E, ler gets us to the desired conclusion.
C Comparison of Monarch matrices and GS-matrices
GS(Pt, P, P.) class is inspired by Monarch matrices [Dao al., 2022] and their primary goal is to introduce additional flexibility in the block structure of matrices L and R. Generalized Monarch matrices are parameterized P; LP R. where and R are block-diagonal matrices and Pi and Pe are certain permutations defined in Definition 5.2. This resembles GS(P1. P2,1) matrix class, however comparison monarch matrices have additional hard constraints on relation between kr and kn- Being more precise, Monarch matrices are special case of GS(P,, P2, I) matrices with additional constraints kn = b/2. kR D2.- Such constraints lead to several theoretical and practical limitations of Monarch matrices. From theoretical point of view, Monarch matrices can only describe permuted block matrices with blocks of ranks 1. In contrast GS (S-matrices with can describe matrices with different rank structure of blocks (including structures where rank of each block is equal to arbitrary r). From practical point of view, due to this constraint Monarch matrices are often unable to form a desirable block structure of matrices I. and R. For demonstration of this phenomena, consider a case of square matrices with square blocks - the structure needed Orthogonal fine-tuning paradigm. Formally, we have 6) 67, by. b)² b 6R. m n. Additional Monarch constraint would mean that 6R = kL; by = kR- This in turn means that ky kR = n. As we can see, makes impossible to stack two matrices with small number of blocks (say, 4) or large number of blocks, which is required in situations with low parameter budget. In contrast, GS parametrization allows for both of these structures, which
in our experiments.
Note, that the work [Fu et al., 2024] provides a slightly different definition for monarch matrices, introducing order-p Monarch matrices. These matrices are also special case of GS class, however they are very restrictive as they can only parametrize matrices with both sides equal to a² for some integers a, p.
16

---

o
Figure 5: Demonstration of information transition through a block structure. Each node connected to exactly consecutive nodes from the next level.
D Proof of Theorem 2
We use information transition framework from [Liu et al., 2024b], representing product m sparse d x matrices as transmitting information in grid with d x (m++1) 1) nodes. Edges between nodes j and represent that element j,j in sparse matrix is non-zero. Element i, from final matrix can only be non-zero if there exists path from the j-th node from the right column to the i-th node in the left column (see Figure 5).
Proof. Consider an information transmission graph for the matrix B, Bt (r.)).. In this graph, the first node connects with first edges, the second node connects with the edges from
to 2b and so on. Now consider graph for the product of such matrices. As shown in Figure 5, now each node from the first level has paths to 6² unique nodes from the kth-th level. It means that using [log, (d) [br]] 1 + [log, (r) matrices is sufficient reach all nodes and therefore form a dense matrix. Note that the number paths for each node is always equal
regardless of permutation choice. This observation shows that it is impossible to reach d unique elements on the final level with m + [rog
E Subject-driven generation
Training details All the models are trained using Adam optimizer with batch size 4, learning rate =0.00002, betas (0.9, 0.999) and weight decay 0.01. The Stable Diffusion-2-base model is used for all experiments.
Evaluation details We use the DreamBooth dataset for evaluation. The dataset contains 25 different contextual prompts for 30 various objects including pets, toys and furnishings. For each concept generate 10 images per contextual prompt and 30 images per base prompt "a photo of S*" resulting in 780 unique concept-prompl pairs and total of 8400 images for fair evaluation.
To measure concept fidelity, we use the average pairwise cosine similarity (IS) between CLIP ViTB/32 embeddings of real and generated images as [Gal al., 2022]. This means that the image similarity is calculated using only the base prompt, i.e. "a photo of an S*" Higher values of this metric usually indicate better subject fidelity, while keeping this evaluation scene-independent. To evaluate the correspondence between generated images and contextual prompts (TS), the average cosine similarity
17

---

between CLIP ViTB/32 embeddings of the prompt and generated images [Ruiz et al., 2023, Gal et al., 2022].
Overfitting discussion Typically, the more parameters are trained, the more easily the model overfits. This results in higher image similarity and lower text similarity. In addition, if model with large number of training parameters is trained for a long time, the image similarity can often start to decrease the model starts to collapse and artifacts with more trainable parameters having a worse score in both image and text similarity than the models with fewer ones.
appear. This sometimes leads to models
Models with fewer trainable parameters overfit less, but require longer training to capture the concept carefully, and usually have upper limit on the maximum image similarity: some point the increase image similarity becomes small, while text similarity starts to decrease dramatically. Therefore, the very common result of usual fine-tuning is either overfitting with poor context preservation or an undertraining with poor concept fidelity. The orthogonal fine-tuning shows a different behavior. The model with less trainable parameters can be trained longer (GSOFT, OFT, BOFT) and capture the concept more carefully without artifacts. At the same time, it overfits less and shows higher text similarity.
Additional results Figures 6 we show a graphical representation of the metrics for 1000 and 3000 iterations. Examples of generation for different methods presented in Figure 7, 8.
F GS Orthogonal Convolution
In this section, we provide some details and insights about the choice of the ChShuffle permutation and the activation function.
In experiments, we apply the ChShuffle operation right before grouped convolutional layers. Stack- ing several layers of that form resembles higher-order (S--marices, which motivates the usage of permutations from Definition 5.2 for optimal information transition (see Appendix D). However, in the LipConvnet architecture, the activation function also shuffle information between channels. Thus, this additional shuffling of information can negatively affect our information transition proper- ties. In the original SOC paper [Singla and Feizi, 2021], the authors MaxMin activation, firstly proposed in [Anil et al., 2019].
Definition F.1. [Singla and Feizi, 2021] ] Given a feature tensor X 2mxxxx, the MarMinnXX) activation of tensor X is defined as follows:
A X.m... B Xmm..
M MorMin(X) = mar B),
MaxMin X)m::: = min(A, B).
This activation shuffles information between different groups in convolution which harms performance of experiments, as permutations that in ChShuffle become sub-optimal in terms of information transmission. Thus, we introduce modification of MaxMin activation, that splits channels into pairs in different way. Rather than constructing pairs from different halves of input
1000
steps 3000
LoRArant-4
LoRAranies 32
0.82
raRA-ank-129
GSOFT
BaselS
0.81
G5OFT
r-15
0.80
Double G50FT
Double 650FT
0.79
B0FT ma6
B-FT m-4
0.22
0.24
0.26
0.22
0.24 TS
0.26
TS
Figure 6: Image and text similarity visualisation for different methods on subject-driven generation.
18

---

Concept
Full
LORA
LORA
BOFT
BOFT
GSOFT
Double GSOFT
$ V On acdblesione stee
V in wizzard outl
a V the jungle
the background
chef outfit"
Figure 7: Subject-driven generation visual results on 3000 training iterations.
tensor, we use neighboring channels for forming of pairs (first channel pairs with second, third with fourth and so on). With this modification information does not transfer between groups during activations, which enables more optimal information transmission in-between layers with ChShuff fee operator. In further experiments we denote this activation function as MaxMinPermuted and define below:
Definition F.2. Given a feature map X R 2mxnxn Max MarMMinPermuted(X is defined as follows:
A X B = X1:244
MaxMin Permmleed (x) )).... = maz (A, B); (X) min(A,
However, we also empirically find that it is crucial for the channels that interact within activations functions also interact during convolutions. This means that they should always stay in the same group. This motivates us to use a slightly different permutation for the ChShuff fle operation, which permutes channels in pairs. We use the following permutation
(1 mod
(i))aarrd patred (k.n)
(
+
+ (i mod 2)
2k
19

---

Concept
Full
LoRA
LoRA
BOFT
BOFT
GSOFT
Double GSOFT 64
32
32.
32
5 V on cobolestone stoot
Vn purple mizzard owild
V the Junglo
V in the snow
as y" with blve house in the background
V a chef outfi"
Figure 8: Subject-driven generation visual results on 1000 training iterations.
This permutation can be seen as an adaptation of P(R..) that operates pairs of channels instead of single channels. This permutation also optimal terms of information transition. We call this permutation "paired". Using this paired permutation as ChShuffle with our modified activation saves connection between pairs while also transmitting information in the most efficient way. We provide the results of comparison of approaches with activations and permutations in Table 4.
20

---

Table 4: Comparison of activations LipConvnet-15 architecture and CIFAR-100. (a. in "Groups" column denotes that we have two grouped exponential convolutions (the first one with kernel_st the second with kernel_siee
1). If not mentioned, we have only one GS orthogonal
convolutional layer.
Conv. Layer
# Params
Groups
Speedup
Activation
Permutation
Accuracy
Robust Accuracy
SOC
24.IM
Max
43.15%
29.18%
GS-SOC
6.81M
(4,-)
1.64
MarMuuueeeteed
Permuted
paired
43.48%
29.26 %
GS-SOC
6.81M
(4.-)
1.64
MaxMinPermmted
not paired
40.46%
26.18%
GS GS-SOC
6.81M
(4.-)
1.64
MaxMin
paired
37.99%
24.19%
GS-SOC
6.81M
(4.-)
1.64
MaxMin
not paired
39.72%
25.96%
GS-SOC
8.91M
(4, 1)
1.21
MaxMinPermuted
paired
43.42%
29.56%
GS-SOC
8,91M
(4. 1)
1.21
MaxMinnermmted
not paired
40.15%
26.4%
GS-SOC
8.91M
(4
1.21
MaxMin
paired
40.3%
26.74%
GS-SOC
8,91M
(4
1.21
MaxMin
not paired
41.7%
27.66%
GS-SOC
7.86M
(4.2)
1.22
MaxMinPermuted
paired
42.86%
28.98%
GS-SOC
7.86M
(4,2)
1.22
MixMMenerrmuted
not paired
27.53%
GS-SOC
7.86M
(4
1.22
MaxMin
paired
41.55%
27.45%
GS-SOC
7.86M
(4.2)
1.22
MaxMin
paired
41.25%
27.29%
GS-SOC
7.3M
(4.
1.23
MaxMinPermuted
paired
42.75%
28.7%
GS-SOC
7.3M
(4,4)
1.23
MaxMinPermuteed
paired
38.93%
25.59%
GS-SOC
7.3M
(4.4)
1,23
MaxMin
paired
40.34%
27.06%
GS-SOC
7.3M
(4
1.23
MaxMin
not paired
41.57%
27.48%
can be seen that using "paired" permutation used with MinMaxPPrmuted activation significantly improves quality metrics.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, all the assumptions claimed in abstract and introduction were discussed in our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
All the limitations of the work are discussed in Section 9.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Yes, we prove all theorems which were introduced in our paper or we have all the necessary references theory we used.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, provide all the necessary information to reproduce the results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: Yes, we provide link to github repo in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ httpc//guides//ooeeeeee
wwwssssoooooooooe oneioinopyyy olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips.. wtt.cc/ppbb. wwtpppplc/gguiiiiieeeeeeeooceele
httee/CCoeSSuiiiiseeee wwwihbbmissioni cesssonoolioe for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Yes, in Section 7 we provide all the necessary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We conducted the experiments within limited computational resources and demanding settings. Nevertheless, we plan to add error bars to some of the experiments on convolution NNs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: Yes, we wrote the information about the resources used for experiments in Section 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eeuripp.eeeeee cttpiiss..ppppuue wwtppplic//tttieeeeeeccee www.llic/Etiicstuudellliiiee
Answer: [Yes]
Justification: Yes, our research conforms with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, cite all papers appeared in the text.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release any assets in our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
In our paper there is no crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "xgZFYATDaJ", "reviewer": "Reviewer_dcw2", "summary": "This paper proposes a new parameterization for orthogonal matrices and applies this parameterization to orthogonal finetuning of foundation models. The new orthogonal parameterization is interesting and useful. The experimental results show some improvement over the baselines.", "questions": "See the weakness section.", "limitations": "The major concern that I has is whether the results are easily reproducible. With that being said, the authors should consider to release the source code for the orthogonal finetuning experiments.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The study of parameter-efficient orthogonal finetuning is very much in need and is an important topic at the mement. I believe this new parameterization will be useful for orthogonal finetuning in general.\n\n- The GS parameterization seems novel to me, although I am not well knowledgeable about the related orthogonal parameterization literature. However, in the application of orthogonal finetuning, the paper, to my knowledge, is the first work to apply this parameterization.\n\n- Specifcially, the GS parameterization resembles the Monarch matrices, as mentioned in the paper. To some extend, it can be viewed as a generalization of the Monarch matrices. \n\n- The experimental results look promising and demonstrate that its parameter-efficiency outperforms both OFT and BOFT in some scenarios.", "weaknesses": "- Although it is trivial to show that GS matrices can approximate arbitrary orthogonal matrices (as it generalizes the Monarch matrices), it is still important to discuss and (potentially re-use) the theoretical results from the Monarch matrices.\n\n- It can further strengthen the paper, if the authors can add the comparison of the Monarch parameterization in the orthogonal finetuning experiments.\n\n- The implementation should be released prior to its acceptance, otherwise it may be difficult for people to reproduce the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. First of all, we add a source code for our experiments (see the top-level comment). Regarding the Monarch matrices, the key drawback in the Monarch representation for us is that if one block-diagonal factor has many small blocks, then the other one has to have several large dense blocks. As a result, we are not able to obtain as few parameters as we want, which was critical for our experiments. For example, in the natural language understanding experiment, obtaining number of parameters similar to LoRa with rank 8 is impossible with Monarch Matrices, which would require at least 4x parameters with any block structure. On the other hand, additional flexibility introduced in GS matrices allows for matrices with small number of blocks in each of the matrices, making it possible to use this method with any parameter budget. \nAt the same time, the order-p generalization of Monarch matrices from the follow-up paper, is only capable of working with very specific matrix sizes (see also Appx. C). What is more, the used permutation matrices do not allow for optimally filling the matrix with non-zeroes using a given budget of parameters.\nFollowing your suggestions, we will emphasize these points in the revised version and add more discussion concerning the Monarch matrix class.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LXfecAarw", "reviewer": "Reviewer_AkW1", "summary": "This paper introduces a new way of constructing orthogonal parametrization for efficient model fine-tuning. The development of the proposed method is motivated by: naive orthogonal fine-tuning, which can be too restrictive due to how it constructs the block-diagonal orthogonal matrix; and improved methods with dense orthogonal matrix can be too expensive in terms of both computation and parameters. \nThe authors introduce a new way of constructing orthogonal matrix by an alternating product of block-diagonal matrices and several permutations, with advantages over previous methods justified theoretically. \nExperiments on language model, stable diffusion, and standard conv models are presented.", "questions": "- Does the SVD need to be performed in every optimization step?", "limitations": "The limitation discussions presented in Section 9 focus mainly on how the model compares to LoRA and BOFT. The proposed method still requires more parameters than other improved LoRA methods.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is able to significantly reduce the parameter and computation compared to the previous method on dense orthogonal parameter tuning. \n\n- The overall writing of this paper is good. And the presentation is well organized. \n\n- The core advantages of the proposed methods are justified with theoretical supports. The development of GS Orthogonal Convolutions makes it more generalizable to more applications.", "weaknesses": "My major concern with this paper is the empirical justifications. \n\n**Small experiment scale.**\nAll experiments are conducted in very small scale. Specifically, the language models used in this paper are relatively old and outdated; Only few-shot personalization experiments are reported in the stable diffusion experiments; and the final conv application is only conducted on a tiny network on cifar100. \nThis makes it very hard to objectively evaluate the proposed method's practical effectiveness.\n\n**Missing evaluations**\nIf I'm not missing anything, the entire *training time* row of Table 2 is empty. Training time and memory are the core advantages compared to the previous methods. While it is true that the method is able to reduce the parameters and computations by numbers, significant speed-up and memory reduction in practice can still be a huge support to the paper's contribution. \n\n**Missing comparisons**\nThe comparisons reported in this paper are mainly on LoRA and other orthogonal tuning methods. However, without the comparisons against other improved LoRA methods, such as DoRA, it can be hard to position the contribution and impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Small experiment scale.** We do understand your concerns regarding the scale of experiments.\n\nFor diffusion experiments, the few-shot setting is one of the most challenging tasks for fine-tuning. On the one hand, the model should carefully capture the target concept, but on the other hand, it should preserve it's ability to follow the target prompt. This setting is one of the main motivations for orthogonal fine-tuning (OFT), as it additionally helps to avoid overfitting and was used in the pioneering work with the OFT paradigm. \n\nDespite the few shot setting of a task, we report results accumulated over **30 different concepts and 25 contextual text prompts**. In total, there are 750 unique concept-prompt pairs and **a total of 7500 images for each baseline for robust evaluation**. And our evaluation includes different hyperparameters for each baseline method.\n\nNotice that BOFT only presents a visual comparison on $11$ different concepts with one fixed set of hyperparameters and does not provide a qualitative evaluation. Thus, our evaluation in this task is more comprehensive and gives more insight into how different models perform with different parameters.\n\nFor 1-Lipschitz architectures, we took a standard neural network that was specifically developed for smaller datasets. We are not aware of any other architecture baselines that guarantee exact 1-Lipschitzness and are developed for larger datasets such as ImageNet. The works we aware of do not go beyond smaller datasets in their experiments, see e.g. [1, 2, 3]. One of the factors for this absence could be the high computational complexity associated with 1-Lipschtiz layers. \n\nFinally, regarding the NLP experiment, the main concern for us was the computational resources required for running larger models. RoBERTa experiment took us multiple weeks of compute time. Conducting experiments with multi-billion models would require the amount of computational resources that we unfortunately do not have. Nevertheless, we believe that our experiments are of interest, as they show the comparison with the closest competitor BOFT on a different domain apart from diffusion experiments.\n\n**Missing evaluations.** Thank you for pointing this out. We added training times for the Table 2 (see the top-level comment). Both GSOFT and Double GSOFT are faster (and better in terms of metrics) than BOFT.\n\n**Missing comparisons.** Thank you for this comment. We did an additional experiment with DoRA for subject-driven generation. We found that DoRA gives slightly better results compared to LoRA, but still shows significantly lower text similarity compared to GSOFT and DoubleGSOFT. **See Table 1 and Figure 1 from attached PDF file.** We also plan to add DoRA comparison for NLP experiment in the revisited version. \n\n**SVD during optimization steps**. We actually did not use SVD at any step of the process. The SVD steps we describe when projecting to the class of GS matrices can be a useful tool for future applications. For example, for different initiazation strategies of GS-matrices.\n\n[1] Singla et al. \"Skew orthogonal convolutions.\" International Conference on Machine Learning. 2021\n \n[2] M. Laurent, et al. \"A dynamical system perspective for lipschitz neural networks.\" International Conference on Machine Learning. 2022\n\n[3] X. Xiaojun, et al. \"Lot: Layer-wise orthogonal training on improving l2 certified robustness.\" Advances in Neural Information Processing Systems. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "63wBggRfU3", "reviewer": "Reviewer_yzRC", "summary": "This paper follows the orthogonal fine-tuning paradigm that uses orthogonal matrices for adapting the weights of a pretrained model. It introduces a class of structure matrices, defined as GS-matrices, that can well be used to conduct orthogonal matrix. Based on GS-matrices, It proposes structured orthogonal parametrization and studies its performance in the orthogonal fine-tuning framework. Experiments are conducted on two standard setting ups: downstream task fine-tuning in language modeling and text-to-image diffusion tasks. It also claims the proposed method can be used to construct orthogonal convolutions for 1-Lipschitz neural networks.", "questions": "The questions in weakness 1", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The description of this paper is very clear, and the overall presentation is good. It clearly describes how this paper motivates from orthogonal fine-turning. The proposed method using structured matrix to conduct orthogonal matrix is interesting.", "weaknesses": "1.The experiments are not well conducted to support the claims/effectiveness of the proposed method:  \n(1) This paper spends many words to claim the efficiency of the proposed methods. However, this paper does not provide the experiments to show the wall-clock time in the natural language understanding and subject-driven generation. I think this paper should provide the wall-clock time when comparing to baselines in Table 1 and Table 2, to show the efficiency of the proposed methods, compared to the baselines. What is the results of wall-clock time of these methods in Table 1 and 2?  \n(2) This paper should conduct some ablation study to the Double GSOFT. It seems that Double GSOFT is the key to improve the performance, based on the description of this paper, but the double idea is apparently incremental. Besides, why Table 1 show the results using GSOFT, while Table 2 using Double GSOFT? Why not provide both in the experiments or using the best one (maybe Double GSOFT) with ablation study? I think this paper should address this point in the rebuttal. \n\n2.This paper claims that “Nevertheless, parametrization of orthogonal matrices is a challenging task” In Line 19. I do not think so. There are many previous solutions to conduct orthogonal matrix in DNNs before 2019[1,2,3,4], even for conducting orthogonal convolutions [5,6]. And there are merits and drawbacks among these methods. I think this paper should well discuss these previous methods and take credits to them. \n\n3.The original motivation of this paper is for parametrization efficient finetuning by using orthogonal fine-tuning. Meanwhile, it also wants to claim its contributions to the application orthogonal matrix benefits, E.g., conducting 1-Lipschitz network for robustness. \nBut this claim is not well supported by experiments (There are many other orthogonal convolution methods(e.g., [5] and [6]), not only the baselines in the paper. E.g. SOC). Besides, this paper should pay more attention on discussing the orthogonal techniques in DNNS during the related work section. \n\n\n**Ref:**  \n[1] On orthogonality and learning recurrent networks with long term dependencies, ICML 2017   \n[2] Learning unitary operators with help from u(n), AAAI 2017  \n[3] Orthogonal recurrent neural networks with scaled Cayley transform, ICML, 2018  \n[4] Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, AAAI 2018  \n[5] Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks, NeurIPS 2019  \n[6] LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness, NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1.** Thank you for your comments. We added compute times and comparison to the standard GSOFT for subject-driven generation (see the top-level comment). In the NLP tasks, we are also faster or on par with the BOFT approach. Compared to LoRa we are several times slower, but observe a slight quality boost on the scale we consider. The BOFT paper shows more quality gain for larger models like Llama. Unfortunately, such experiments require much more computational resources. \n\n**W2 and W3.** Thank you for pointing out these papers. We will be sure to make a more detailed survey in the revised version. \nTo be more specific, [1]-[3] work in the setting of dense matrices and try to parameterize essentially the whole manifold of orthogonal matrices. For $n \\times n$ matrices this requires computing inverses or exponential maps of $n \\times n$ matrices at every optimization step. This takes $O(n^3)$ time, which can be computationally challenging for larger architectures. Moreover, such a parametrization utilizes $\\mathcal{O}(n^2)$ trainable parameters, which makes it inapplicable for PEFT (see the OFT paper [Qiu et. al, NeurIPS '23] for more details). Our method is different in a sense that it provides a trade-off between expressivity (describing only a subset of the orthogonal manifold) and efficiency. For example, for a dense $n \\times n$ matrix, when setting a number of blocks equal to $\\sqrt{n}$, our method uses $O(n \\sqrt{n})$ parameters and does $O(n^2 \\sqrt{n})$ computations at optimization steps. \n\nWork [4] applies orthogonal parametrization to a reshaped convolution tensor.  Although it is indeed a widely used heuristics, it does not formally lead to an orthogonal convolution mapping and certified robustness. The work [5] was outperformed by the SOC method that we utilize (there is a comparison in the SOC paper). Finally, in [6], the authors utilize periodicity of convolution and their padding is not equivalent to a widely-used zero-padded convolution. What is more, the survey [7] suggests that [6] and other 1-Lipschitz architectures yield worse robustness metrics than SOC, which is why it serves as a baseline in our paper. \n\n[7] B. Prach, et al. \"1-Lipschitz Layers Compared: Memory Speed and Certifiable Robustness.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M021I7yvSY", "reviewer": "Reviewer_Kbzt", "summary": "In this paper, the authors proposed a new structured matrices, namely GS-matrices. It unifies and generalizes structured classes from previous works. Compared to the previous BOFT, it further improves the parameters and training efficiency. Experiments on language processing, text-image generation, and image classification have demonstrated the performance of GSOFT.", "questions": "My main concerns are listed in the weaknesses.\n1. It's better to demonstrate the performance of the proposed method on recent popular GPT series methods (e.g., Llama2, Llama3, Mixtral). This may expand the influence of this method in the field.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is easy to understand. \n2. The proposed GSOFT has been applied to various networks on various downstream tasks.", "weaknesses": "1. It's better to provide the training time comparison between BOFT, LoRA, and GSOFT. It seems that some results are lost in Table 2.\n2. It's better to discuss more about single GSOFT and double GSOFT. It seems that the results in Table 1&3 are from the single version and the results in Table 2 are from the double version.\n3. In Table 2, it seems that more learnable parameters result in lower CLIP scores. Maybe more discussions are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Training times and double GSOFT.** Thank you for raising these questions. We have fixed these issues in the top-level comment.\n\n\n**CLIP scores.** Thank you for pointing out this observation. Typically, the more parameters are trained the more easily the model overfits. This results in higher image similarity and lower text similarity. In addition, if a model with a large number of training parameters is trained for a long time, the image similarity can often start to decrease as the model starts to collapse and artefacts start to appear.\n\nModels with fewer trainable parameters overfit less, but require longer training to capture the concept carefully, and usually have an upper limit on the maximum image similarity: at some point the increase in image similarity becomes small, while text similarity starts to decrease dramatically. Therefore, the very common result of a usual fine-tuning is either an overfitting with poor context preservation or an undertraining with poor concept fidelity. \n\nThe orthogonal fine-tuning shows a different behaviour. The model with less trainable parameters can be trained longer (OFT, BOFT) and capture the concept more carefully without artefacts. At the same time, it overfits less and shows higher text similarity.\n\nIndeed, this needs to be discussed and we will add this to the paper, thank you.\n\n**GPT series methods.** While we certainly agree with your point, RoBERTa experiment took us multiple weeks of compute time. Conducting experiments with multi-billion models would require the amount of computational resources that we unfortunately do not currently have. Nevertheless, we believe that our experiments are of interest, as they show the comparison with the closest competitor BOFT on a different domain apart from diffusion experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for taking the time to review our article.
We appreciate your feedback and the constructive remarks you have provided.
Before giving detailed answers to the raised questions, we briefly reiterate the main contribution of our paper.
We propose a new class of matrices, denoted as $\mathcal{GS}$ that generalizes previous matrix classes: Monarch matrices, their order-p generalization, and Block Butterfly matrices.
We theoretically show how to choose permutations in the $\mathcal{GS}$-class to address the shortcomings of the previous approaches. 
We showcase our class on multiple application domains, including text-to-image diffusion models, language models and convolutional architectures.

In addition to the specific answers below, we address several key common questions from the reviewers in the top-level comment:

1.  We add an anonymised version of our code to ensure the reproducibility of the results (Reviewer dcw2). Anonymous link to the code was provided to AC in a separate comment following NeurIPS guidelines.
    
 2.  We extended the results in Table 2 for the whole Dreambooth dataset, that contains $30$ concepts of different categories, including pets, interior decoration, toys, backpacks, etc. For each concept, we used $25$ contextual text prompts, which include accessorisation, appearance and background modification. We also added DoRa baseline (Reviewer AkW1). As an ablation study, we conduct experiments in a non-doubled version of GSOFT (Reviewers yzRC, Kbzt). Both GSOFT and Double GSOFT show similar performance in terms of text similarity, outperforming other baselines with the same number of training parameters. However, as the number of training parameters decreases, Double GSOFT ($r=64$) provides better image similarity and shows the most balanced solution in terms of concept fidelity and context preservation among all baselines. **See Table 1 from attached PDF file.**

 3. We provide training times for the Table 2 (Reviewers AkW1, yzRC, and Kbzt). Both GSOFT and Double GSOFT are faster (and better in terms of metrics) than BOFT and comparable to DoRA. **See Table 1 from attached PDF file.**

4. We also provide more visual comparison for subject-driven generation to illustrate the performance of the baselines on different prompts and concepts. **See Figure 1 from attached PDF file.**

Meta-review: The paper introduces a novel structured matrix class, GS-matrices, which unifies and extends existing classes like Monarch and Block Butterfly matrices. The proposed approach is applied to orthogonal finetuning, showing parameter efficiency across different tasks, including text-to-image diffusion models and language modeling. Reviewer dcw2 praised the novelty and practical impact of the GS parameterization, rating it as an accept due to the strong potential for broader application in orthogonal finetuning. However, the reviewer suggested incorporating comparisons with Monarch parameterization and releasing the implementation for reproducibility, which the authors addressed by providing the code and discussing the limitations of Monarch matrices.

Reviewer AkW1 highlighted the theoretical contributions and the method’s ability to reduce parameters and computation, but expressed concerns over the small scale of experiments, lack of training time data, and missing comparisons with newer LoRA methods like DoRA. The authors responded by adding training times and conducting additional comparisons, yet AkW1 maintained a borderline reject rating due to perceived limitations in empirical validation.

Reviewer yzRC found the presentation clear but questioned the efficiency claims without wall-clock time data and suggested ablation studies for Double GSOFT. They also noted the need for a broader discussion on previous orthogonal matrix parameterization methods. The authors responded with additional data and a more detailed survey of related work, but yzRC still rated the paper as borderline reject, citing incremental contributions and limited experimental validation.

Reviewer Kbzt recognized the broad application of GSOFT but recommended additional training time comparisons and broader evaluations on recent models like Llama2. Although the authors provided further clarifications and data, Kbzt did not raise their score, keeping it at borderline accept, indicating that while their concerns were addressed, broader experimental evidence was still needed.

Overall, the paper presents a promising and technically solid contribution to orthogonal finetuning. However, the concerns regarding empirical validation and broader impact, as raised by multiple reviewers, suggest that further refinement and additional experiments would strengthen the paper. Given the addressed feedback and the potential impact, I recommend acceptance, with an emphasis on expanding the experimental evaluations in future work to fully realize the method’s contributions.
Decision: Accept (poster)