Group and Shuffle: Efficient Structured Orthogonal Parametrization

Authors: mikhail  gorbunov
nikolay  yudin
vera soboleva airi
alexey  naumov
maxim  rakhuba
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:38:10

Abstract: The increasing size of neural networks has led to a growing demand for methods of efficient finetuning. Recently, an orthogonal finetuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it.  We then use this parametrization to modify the orthogonal finetuning framework, improving parameter efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task finetuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.

Main paper: 
 introduction: 
 Orthogonal transforms have proven useful in different deep learning tasks. For example, they were shown to stabilize CNNs [Li et al., 2019, Singla and Feizi, 2021] or used in RNNs to combat the problem of exploding/vanishing gradients [Arjovsky et al., 2016] . Recent works OFT (Orthogonal Fine-Tuning) and BOFT (Butterfly Orthogonal Fine-Tuning) [Qiu et al., 2023 , Liu et al., 2024b] use learnable orthogonal matrices for parameter-efficient fine-tuning of neural networks, which prevents training instabilities and overfitting that alternative methods like LoRA [Hu et al., 2022] suffer from.Nevertheless, parametrization of orthogonal matrices is a challenging task, and the existing methods typically lack in either computational efficiency or expressiveness. Classical methods like Cayley parametrization and matrix exponential map cannot operate under low parameter budget, while Givens rotations and Householder reflections requires computing products of several matrices, which makes their use less efficient in deep learning tasks. Alternative approach in OFT method uses blockdiagonal matrix structure in an attempt to be more computationally efficient and use less trainable parameters. Unfortunately, this simple structure can be too restrictive. Thus, arises the problem of constructing dense orthogonal matrix while still being parameter-efficient. While attempting to tackle this task, BOFT method uses a variation of butterfly matrices, parametrizing orthogonal matrices as a product of several matrices with different sparsity patterns, enforcing orthogonality on each of them. This parametrization is able to construct dense matrices while still being parameter-efficient. However it requires to compute a product of multiple matrices (typically up to 6) which can be computationally expensive. In this paper, we aim to overcome these issues and build dense orthogonal matrices in a more efficient way.We present a novel structured matrix class parametrized by an alternating product of block-diagonal matrices and several permutations. Multiplying by these matrices can be seen as a consecutive application of independent linear transforms within certain small groups and then shuffling the elements between them, hence the name Group-and-Shuffle matrices (or GS-matrices for short). This class generalizes Monarch matrices [Dao et al., 2022] and with the right permutation choices, is able to form dense orthogonal matrices more effectively compared to approach proposed in BOFT, decreasing number of matrices in the product as well as the number of trainable parameters. We build efficient structured orthogonal parametrization with this class and use it to construct a new parameter-efficient fine-tuning method named GSOFT.Our contributions:• We introduce a new class of structured matrices, called GS, that is more effective at forming dense matrices than block butterfly matrices from the BOFT method. • Using GS-matrices, we propose an efficient structured orthogonal parametrization, provide theoretical insights and study its performance in the orthogonal fine-tuning framework. • We adapt our ideas for convolutional architectures, providing a framework to compress and speed-up orthogonal convolution layers. 
orthogonal fine-tuning: 
 Orthogonal Fine-tuning method (OFT) introduced in [Qiu et al., 2023] is a Parameter-Efficient Fine-Tuning (PEFT) method which fine-tunes pre-trained weight matrices through a learnable orthogonal block-diagonal matrix. Some of the properties that make orthogonal transforms desirable are preservation of pair-wise angles of neurons, spectral properties and hyperspherical energy. More precisely, OFT optimizes an orthogonal matrix Q ∈ R d×d for a pre-trained frozen weight matrix W 0 ∈ R d×n and modifies the multiplication y = (W 0 ) ⊤ x to y = (QW 0 ) ⊤ x. Note that the identity matrix I is orthogonal, which makes it a natural initialization for Q. OFT uses block-diagonal structure for Q, parameterizing it asQ = diag(Q 1 , Q 2 , . . . , Q r ),where Q i ∈ R b×b are small orthogonal matrices and br = d. Orthogonality is enforced by Cayley parametrization, i.e. Q i = (I + K i )(I -K i ) -1 , where K i are skew-symmetric: K i = -K ⊤ i . This ensures orthogonality of Q i and, hence, of Q. Nevertheless, block-diagonal matrices can be too restrictive, as they divide neurons into r independent groups based on their indices. This motivates the construction of dense parameter-efficient orthogonal matrices. To address this problem, the Orthogonal Butterfly method (BOFT) was introduced [Liu et al., 2024b] . BOFT uses block-butterfly structure to construct Q. Essentially, Q is parameterized as a product of m orthogonal sparse matrices: Q = B m B m-1 . . . B 1 . Each matrix B i is a block-diagonal matrix up to a permutation of rows and columns, consisting of r block matrices of sizes b × b. Similarly to OFT, the orthogonality is enforced by the Cayley parametrization applied to each block. However, BOFT method has some areas for improvement as well. To construct a dense matrix, BOFT requires at least m = 1 + ⌈log 2 (r)⌉ matrices. For example, the authors of BOFT use m = 5 or 6 matrices in the BOFT method for fine-tuning of Stable Diffusion [Rombach et al., 2022] . Large amount of stacked matrices leads to significant time and memory overhead during training. There is also a room for improvement in terms of parameter-efficiency. To overcome these issues, we introduce a new class of structured matrices that we denote GS (group-and-shuffle) that generalizes Monarch matrices [Dao et al., 2022 , Fu et al., 2024] and show how to use this class to construct parameter-efficient orthogonal parametrization. Similarly to BOFT, our approach uses block-diagonal matrices and permutations, but requires only m = 1 + ⌈log b (r)⌉ matrices of the same size to construct a dense matrix. See details in Section 5.2. The reduced requirements on m allow us to use m = 2 in experiments to maximize computational efficiency, while still maintaining accurate results. 
gs-matrices: 
 Our motivation within this work is to utilize orthogonal matrices of the form:A = P L (LP R)P R (1)where matrices L and R are block-diagonal matrices with r blocks of sizes b × b and P L , P , P R are certain permutation matrices, e.g. P L = P ⊤ , P R = I in the orthogonal fine-tuning setting and P R = P , P L = I for convolutional architectures. Note that although the case P L = P ⊤ , P R = I resembles Monarch matrices [Dao et al., 2022] , they are unable to form such a structure, e.g., with equal-sized blocks in L and R. The issue is that the Monarch class has a constraint that interconnects the number of blocks in matrix L and the number of blocks in matrix R (see Appendix C for details). Moreover, Monarch matrices have not been considered with orthogonality constraints.To build matrices of the form (1), we first introduce a general class of GS-matrices and study its properties. We then discuss orthogonal matrices from this class in Section 4.Definition 3.1. An m × n matrix A is in GS(P L , P , P R ) class with k L , k R blocks and block sizes b 1 L × b 2 L , b 1 R × b 2 R if A = P L (LP R)P R , where L = diag(L 1 , L 2 , . . . , L k L ), L i ∈ R b 1 L ×b 2 L , R = diag(R 1 , R 2 , . . . , R k R ), R i ∈ R b 1 R ×b 2 R , P L , P , P R are permutation matrices and b 2 L • k L = b 1 R • k R = s, b 1 L • k L = m, b 2 R • k R = n. L P R x Rx P (Rx) L(P Rx) Figure 1: GS(I, P , I) matrices with b 1 L = b 2 L = 3, b 1 R = b 2 R = 2, k L = 2, k R = 3. Edges between nodes denote nonzero weights.In practice, we fix P L , P , P R depending on the application and only make matrices L, R subject for change. GS-matrices are hardware-efficient, as they are parametrized by two simple types of operations that can implemented efficiently: multiplications by block-diagonal matrices and permutations.Let us also illustrate a forward pass Ax ≡ LP Rx for a matrix A ∈GS(I, P , I) as a building block for the more general class with two additional permutations. The first operation y = Rx consists of several fully-connected layers, applied individually to subgroups of x, see Figure 1 . The next multiplication LP y ensures that these groups interact with each other. Indeed, the permutation matrix P shuffles the entries of y into new subgroups. These subgroups are then again processed by a number of fully-connected layers using L. This motivates the naming for our class of matrices: Group-and-Shuffle or GS for short.Another useful insight on these matrices is that the class GS(I, P , I) consists of block matrices with low-rank blocks. The permutation matrix P is responsible for the formation of these blocks and defines their ranks (note that rank may vary from block to block). The result below formally describes our findings and is key to the projection operation that we describe afterwards.Proposition 1. Let A be a matrix from GS(I, P , I) with a permutation matrix P defined by the function σ : {0, . . . , n -1} → {0, . . . , n -1}. Let {v ⊤ i } -be the rows of the blocks R 1 , . . . , R k R , {u i } -the columns of the blocks L 1 , . . . , L k L in the consecutive order. Then the matrix A can be written as a block matrix with k L × k R blocks using the following formula for each block A k1,k2 :A k1,k2 = ⌊ σ(i) k L ⌋=k1 ⌊ i k R ⌋=k2 u σ(i) v ⊤ i .Note that we use zero-indexing for this proposition for simplicity of formulas. Let us illustrate this proposition in Figure 2 . We consider GS(I, P , I) with k L = 4 and k R = 2 blocks in L and R and with the block sizes 3 × 3 and 6 × 6 respectively. Let us consider the leading block A 00 of the size 3 × 6. According to Proposition 1,A 00 = u 0 v ⊤ 2 + u 2 v ⊤ 4 .Indeed, let us take a closer look, e.g., at the term u 2 v ⊤ 4 . In the permutation matrix P , we have a nonzero element in the position (2, 4) as i = 4 and σ(4) = 2. Therefore, we select the third column u 2 in L 1 and the fifth row v ⊤ 4 in R 1 . This leads to adding a rank-one term u 2 v ⊤ 4 to A 00 as we see in the formula above. Another direct corollary from Proposition 1 is a projection operation π : R m×n → GS(P L , P , P R ) that satisfies: π(A) ∈ arg minB∈GS(P L ,P ,P R ) ∥A -B∥ F ,where ∥ • ∥ F is the Frobenius norm. Thanks to the block low-rank representation of matrices from GS(P L , P , P R ), the projection π is simply constructed using SVD truncations of the blocks (P ⊤ L AP ⊤ R ) k1,k2 and is summarized in Algorithm 1.Algorithm 1 Projection π(•) of A onto GS(P L , P , P R )Input: A, P L , P , P R Return: L, R for k 1 = 1 . . . k L do for k 2 = 1 . . . k R doCompute SVD of (P T L AP T R ) k1,k2 = U ΣV ⊤ ; Set r = r k1,k2 -rank of block determined by P ; Take U r = U [: r, :], Σ r = Σ[: r, : r], V r = V [: r, :]; Pack columns of U r Σ 1/2 r into L k1 and rows of Σ 1/2 r V r into R k2 according to P ; end for end for 4 Orthogonal GS(P L , P , P R ) matricesIn this section, we study the orthogonality constraint for the GS(P L , P , P R ) to obtain structured orthogonal representation. This is one of the main contributions of our paper and we utilize this class in all the numerical experiments. Since we are interested only in square orthogonal matrices, we additionally assume that m = n and b1 L = b 2 L = b L ; b 1 R = b 2 R = b R .Similarly to parametrizations in OFT and BOFT, a natural way to enforce orthogonality of GS(P L , P , P R )-matrices is to enforce orthogonality of each block of L and R. This indeed leads an orthogonal matrix since permutation matrices are also orthogonal as well as a product of orthogonal matrices. However, it is not immediately obvious that there exist no orthogonal matrices from GS(P L , P , P R ) that cannot be represented this way. Surprisingly, we find that such a way to enforce orthogonality is indeed sufficient for covering of all orthogonal matrices from GS(P L , P , P R ). Theorem 1. Let A be any orthogonal matrix from GS(P L , P , P R ). Then, A admits P L (LP R)P R representation with the matrices L, R consisting of orthogonal blocks.Proof. Matrices P L , P R are orthogonal as they are permutation matrices. It means that it is sufficient to prove theorem in the case when A is from GS(I, P , I), which means that we can use low block-rank structure interpretation from Proposition 1. Consider a skeleton decomposition of the blocksA ij = U ij V ⊤ ij , U ij ∈ R b L ×rij , V ij ∈ R b R ×rij such that U ⊤ ij U ij = I rij (this can be ensured, e.g., using the QR decomposition). ThenA =    U 1,1 V ⊤ 1,1 . . . U 1,k L V ⊤ 1,k L . . . . . . . . . U k L ,1 V ⊤ k L ,1 . . . U k L ,k R V ⊤ k L ,k R    .Take the j-th block-column of A. Since A is an orthogonal matrix, we get:V 1,j U ⊤ 1,j . . . V k L ,j U ⊤ k L ,j    U 1,j V ⊤ 1,j . . . U k L ,j V ⊤ k L ,j    = I b R Multiplying matrices in the l.h.s. we get V 1,j U ⊤ 1,j U 1,j V ⊤ 1,j +• • •+V k L ,j U ⊤ k L ,j U k L ,j V ⊤ k L ,j = I b R . Since U ⊤ ij U ij = I rij we conclude V 1,j V ⊤ 1,j + • • • + V k L ,j V ⊤ k L ,j = I b R . This implies that (V 1,j . . . V k L ,j) is an orthogonal matrix. Note that if we now parameterize A = LP R with the matrices V ij packed into R and U ij packed into L, then (V 1,j . . . V k L ,j ) is exactly the j-th block matrix in R up to permutation of rows. Therefore, every block in R is an orthogonal matrix. Since we now proved that V ⊤ ij V ij = I, we can use same the derivation for the rows of A and conclude that blocks of L are also orthogonal.5 GS(P m+1 , . . . , P 1 ) matricesIn this section we describe an the extension of GS-matrices that uses more than two block-diagonal matrices and show that with the right permutations choices GS-matrices are more effective than block butterfly matrices in forming dense matrices. Here by dense matrices we imply matrices that do not contain zero entries at all. Definition 5.1. A is said to be in GS(P m+1 , . . . , P 1 ) ifA = P m+1 1 i=m (B i P i ),where each matrix B i is a block-diagonal matrix with k i blocks of size b 1 i × b 2 i , matrices P i are permutation matrices and b1 i • k i = b 2 i+1 • k i+1 . Remark 1.Similarly to the case m = 2 described in Section 3, we may use orthogonal blocks in B i , i = 1, . . . , m + 1 to obtain orthogonal matrices. However, it is not clear if an analog to Theorem 1 is correct in this case as well.Remark 2. For each of the classes of Block Butterfly matrices [Chen et al., 2022] , Monarch matrices [Dao et al., 2022] and order-p Monarch matrices [Fu et al., 2024] , there exist permutation matrices P m+1 , . . . , P 1 such that GS(P m+1 , . . . P 1 ) coincides with a respective class. Indeed, Monarch matrices have the form of alternating block-diagonal matrices and permutations with some specific size constraints and sparse matrices in the product of Block Butterfly matrices can be easily transformed to block-diagonal matrices with permutations of rows and columns. 
choosing permutation matrices: 
 We suggest using the following matrices with k = k i for P i . Note that this is efficient for forming dense matrices as follows from the proof of Theorem 2. This is by contrast to the permutations used in [Fu et al., 2024] that are restricted to particular matrix sizes. Definition 5.2 ( [Dao et al., 2022] ). Let P (k,n) be a permutation matrix given by permutation σ on {0, 1, . . . , n -1}:σ(i) = (i mod k) • n k + i k .Applying this permutation to a vector can be viewed as reshaping an input of size n into an k × n k matrix in a row-major order, transposing it, and then vectorizing the result back into a vector (again in row-major column). We provide several examples of such permutations in Figure 3 . P (3,Block Butterfly matrices were introduced in [Chen et al., 2022] and are used to construct orthogonal matrices in the BOFT method. Block Butterfly matrix class is a special case of higher-order GSmatrices with k i = r and b 1 i = b 2 i = b = 2s and certain permutation choices. However, we argue that the choice of these permutations are sub-optimal for construction of dense matrix and using permutations from Definition 5.2 is more effective. When using block-diagonal matrices with r blocks, block butterfly matrices need 1 + ⌈log 2 (r)⌉ matrices to construct a dense matrix. For GS-matrices we have the following result.Theorem 2. Let k i = r, b 1 i = b 2 i = b.Then using m = 1 + ⌈log b (r)⌉ is sufficient for the class GS(P L , P (r,br) , . . . , P (r,br) , P R ) to form a dense matrix for any P L , P R . Moreover, the choice of P 2 = • • • = P m = P (r,br) is optimal in the sense that all matrices from GS(P m+1 , . . . , P 1 ) contain zero blocks for any integer m < 1 + ⌈log b (r)⌉ and any permutations P 1 , . . . , P m+1 .Proof. See Appendix D.For example, let us consider a case of constructing a dense orthogonal matrix of the size 1024 × 1024. Suppose also that we use block matrices with block size 32. Constructing a dense matrix with Block Butterfly matrices requires 1 + log 2 (32) = 6 butterfly matrices, which leads to 6 × 32 3 parameters in the representation. GS(P L , P , P R ) matrices with P = P (32,1024) only need two matrices to construct a dense matrix yielding 2 × 32 3 parameters. The GS(P L , P , P R ) parametrization is also naturally more computationally efficient as fewer number of multiplications is both faster and requires less cached memory for activations. 
applications: 
 6.1 Orthogonal fine-tuning with GS(P L , P , P R ) (GSOFT)We utilize the pipeline of OFT and BOFT methods with the exception of parametrizing Q with orthogonal permuted GS(P L , P , P R ) matrices. In particular, for parametrization of Q ∈ R d×d , we utilize the GS(P ⊤ , P , I) class, i.e.Q = P ⊤ LP R, where L = diag(L 1 , . . . L r ), L i ∈ R b×b , R = diag(R 1 , . . . , R r ), R i ∈ R b×b .For consistency, we use the same notation for the number of blocks and block sizes as in BOFT and OFT methods. We use P (r,br) as a permutation matrix P . To enforce orthogonality, we parameterize each block in matrices L, R with the Cayley parametrization. We initialize Q as an identity matrix by initializing each block to be an identity matrix. Additional techniques like magnitude scaling and multiplicative dropout that are used in OFT and BOFT can be utilized the same way in our method, though we only use scaling in our experiments. Note that likewise in OFT, BOFT weights of the matrix Q can be merged with the pretrained weight W producing no inference overhead.Consider SVD decomposition of a matrix W 0 = U ΣV ⊤ . Applying orthogonal fine-tuning, we get W ′ = (QU )ΣV ⊤ , which is an SVD decomposition for the adapted weight W ′ . This shows that we can only change left singular vectors U with the standard orthogonal fine-tuning paradigm. At the same time, the LoRA method modifies both matrices U and V . Moreover, recent papers [Meng et al., 2024 , Li et al., 2023] show that initializing matrices A, B with singular vectors can additionally boost performance of LoRA. This motivates an extension of orthogonal fine-tuning method, that can adapt both matrices U and V . We introduce a simple approach that multiplies pre-trained weight matrices from both sides, rather than one. This method modifies forward pass from z = (W 0 ) ⊤ x toz = (Q U W 0 Q V ) ⊤ xWhere Q U and Q V are parametrized as orthogonal GS-matrices. In cases where BOFT utilizes 5-6 matrices, we can leverage the fact that our method uses only 2 and adapt both sides while still using less matrices and trainable parameters than BOFT.Recall, that due to linearity of a multichannel convolution operation, we can express the convolution of tensor X ∈ R cin×h×w with a kernel L ∈ R cout×cin×k×k L ⋆ X in terms of matrix multiplication [Singla and Feizi, 2021] :EQUATIONwhere L i,j is doubly Toeplitz matrix, corresponding to convolution between i-th and j-th channels and vec(X) is a vectorization of tensor into a vector in a row-major order. Thus, the convolution is essentially a block matrix, where each block represents a standard convolution operation. Using this block interpretation (2), we may apply the concept of GS matrices to convolutional layers as well. Considering each convolution between channels as an element of our block matrix, we can set some of these blocks to zero, obtaining some additional structure. Thus, we can construct block matrix which has block-diagonal structure, corresponding to grouped convolution (further, in all equations we will denote it as GrConv). Then, defining ChShuffle as a permutation of channels, like in [Zhang et al., 2018] , we obtain structure, which is similar to GSOFT, defined in Section 6:EQUATIONThe proposed GS convolutional layer shuffles information between each pair of input channels and requires less parameters and FLOPs during computations. In this example we can also choose permutations of channels and change kernel size. This convolutional layer can be treated as GS(P m+1 , . . . , P 1 ) matrix in vectorized view, that is why choosing permutations between convolutional layers is also very important for information transition properties. In Appendix F we explain the choice of ChShuffle operation.We can use the proposed layer to construct orthogonal convolutions (transformations with an orthogonal Jacobian matrix) similarly to skew orthogonal convolution (SOC) architecture, that uses Taylor expansion of a matrix exponential. One major downside of methods such as SOC and BCOP [Li et al., 2019 ] is that they require more time than basic convolution operation. For instance, in the SOC method, one layer requires multiple applications of convolution (6 convolutions per layer).In our framework, we propose a parametrization of a convolutional layer, in which imposing an orthogonality to convolutions has fewer number of FLOPs and parameters thanks to the usage of grouped convolutions.Let us discuss in more details how SOC works and the way we modify it. In SOC, a convolutional filter is parametrized in the following way:L = M -ConvTranspose(M ),where M ∈ R cin×cout×r×s is an arbitrary kernel and the ConvTranspose is the following operation:ConvTranspose(M ) i,j,k,l = M j,i,r-k-1,s-l-1This parametrization of filter L makes the matrix from Equation 2 skew-symmetric. As matrix exponential of skew-symmetric matrix is an orthogonal matrix, in SOC the authors define convolution exponential operation, which is equivalent to matrix exponential in matrix-vector notation: Definition 6.1. [Singla and Feizi, 2021] Let X ∈ R c×h×w be an input tensor and L ∈ R c×c×k×k be a convolution kernel. Then, define convolution exponential L ⋆ e X as follows:L ⋆ e X = X + L ⋆ X 1! + L ⋆ 2 X 2! + . . .where L ⋆ i X is a convolution with kernel L applied i times consequently.As mentioned above, with proper initialization we get a convolutional layer with orthogonal Jacobian matrix. Using the parametrization of convolution layer from the Equation 3 and substituting there two grouped convolution exponentials (e.g. in our parametrization we have the same convolution exponential, but we have grouped convolution instead of basic one) with the parameterized kernel:Y = GrExpConv 2 (ChShuffle 2 (GrExpConv 1 (ChShuffle 1 (X))))In our experiments we tried different layer architectures and we found that making kernel size of the second convolution equal to 1 speeds up our convolutional layer, maintaining quality metrics. Thus, if convolutional layer consists of two grouped convolutional exponentials, the second convolutional exponential has kernel_size = 1 × 1 
experiments: 
 All the experiments below were conducted on NVIDIA V100-SXM2-32Gb GPU. We ran all the experiments within ∼2000 GPU hours.Source code is available at: https://github.com/Skonor/group_and_shuffleWe report result on the GLUE [Wang et al., 2018] benchmark with RoBERTa-base [Liu et al., 2019] model. Benchmark includes several classification tasks that evaluate general language understanding. We follow training settings of [Liu et al., 2024b , Zhang et al., 2023] . We apply adapters for all linear layers and only tune learning rate for all methods. Table 1 reports best results on the evaluation set from the whole training. LoRA, OFT and BOFT are implemented with PEFT library [Mangrulkar et al., 2022] . GSOFT method outperforms OFT, BOFT and also have a slight edge over LoRA. Note that even though skew-symmetric K theoretically matrix only requires approximately half the parameters of a full matrix, in practice it is parametrized as K = A -A T for the ease of computations. However, after fine-tuning, one can only save upper-triangular part of K. Doing this, orthogonal fine-tuning methods become approximately 2 times more efficient in terms of memory savings.Subject-driven generation [Ruiz et al., 2023 , Gal et al., 2022] is an important and challenging task in the field of generative modelling. Given several photos of a particular concept, we want to introduce it to the diffusion model so that we can generate this particular object in different scenes described by textual prompts. The main way to do this is to fine-tune the model. However, the large number of fine-tuning parameters together with the lack of training images make the model prone to overfitting, i.e. the model reconstructs the concept almost perfectly, but starts to ignore the textual prompt during generation. To solve this problem and stabilize the fine-tuning process, different lightweight parameterizations [Qiu et al., 2023 , Liu et al., 2024b , Hu et al., 2022 , Tewel et al., 2023, Han et al 2023] and regularization techniques [Ruiz et al., 2023 , Kumari et al., 2023] are widely used in this task. Therefore, we chose this setting to evaluate the effectiveness of the proposed orthogonal parameterization compared to other approaches.We use StableDiffusion [Rombach et al., 2022] and the Dreambooth [Ruiz et al., 2023] dataset for all our experiments. The following parameterizations were considered as baselines in this task: full (q, k, v and out.0 layers in all cross-and self-attentions of the UNet are trained), LoRA [Hu et al., 2022] and BOFT [Liu et al., 2024b] applied to the same layer. We use our GSOFT parameterization and a two-sided orthogonal GSOFT (Double GSOFT) applied to the same layers as baselines. For a more comprehensive comparison, we consider different hyperparameters for the models, adjusting the total number of optimized parameters. More training and evaluation details can be found in Appendix E.CLIP image similarity, CLIP text similarity and visual comparison for this task are presented in Table 2 and Figure 4 . As the results show, GSOFT and DoubleGSOFT are less prone to overfitting compared to the baselines. They show better alignment with text prompts while maintaining a high level of concept fidelity. Furthermore, both methods with optimal hyperparameters are more efficient than BOFT and comparable to LoRA and full parameterization in terms of training time.See Appendix E for more visual and quantitative comparison.Following [Singla and Feizi, 2021] , we train LipConvnet-n on CIFAR-100 dataset. LipConvnet-n is 1-Lipschitz neural network, i.e. neural network with Lipschitz constant equal to 1, his property provides certified adversarial robustness. LipConvnet uses orthogonal convolutions and gradient preserving activations in order to maintain 1-Lipschitz property.LipConvnet-n architecture consists of 5 equal blocks, each having n 5 skew orthogonal convolutions, where the last convolution at each level downsamples image size. We replace the skew orthogonal convolution layer with the structured version using GSorthogonal convolutions and test it in the setting of [Singla and Feizi, 2021] , using the same hyperparameters (learning rate, batch size and scheduler stable during testing). In layers where we have two GrExpConv, the second convolution has kernel size equal to 1.We also use a modified activation function (MaxMinPermuted instead of MaxMin), which uses different pairing of channels. This makes activations aligned with the ChShuffle operation and grouped convolutions. The choice of permutation for ChShuffle also slightly differs from permutations defind in Definition 5.2 because of the interplay between activations and convolutional layers. We provide definitions and intuition regarding activations and permutations for ChShuffle in Appenix F.  
concluding remarks: 
 In this paper, we introduce a new class of structured matrices, called GS-matrices, build a structured orthogonal parametrization with them and use them in several domains within deep learning applications. However, we hope that our orthogonal parametrization can be adapted to different settings in future (including tasks outside of deep learning), as it makes orthogonal parametrizations less of a computational burden. GS-matrices without orthogonality constraints is another promising direction. 
limitations: 
 Although our method for orthogonal fine-tuning is faster than BOFT, it is still slower than LoRA during training. Additionally, since our parametrization provides a trade-off between expressivity and parameter-efficiency, it might be unable to represent some particular orthogonal matrices, which might be required in other settings apart from parameter-efficient fine-tuning.Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856, 2018.Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu. Enhancing detail preservation for customized text-to-image generation: A regularization-free approach. arXiv preprint arXiv:2305.13579, 2023.Parameter-Efficient Fine-Tuning (PEFT) With the growth of model sizes, end-to-end training became unavailable for those who want to adapt powerful architectures for specific tasks, as even full fine-tuning became too expensive. This problem sparked research in the direction of parameterefficient fine-tuning methods, including methods that focus on prompt tuning [Lester et al., 2021, Li and Liang, 2021] and adapter tuning (e.g. [Houlsby et al., 2019 , Karimi Mahabadi et al., 2021] ), which include LoRA [Hu et al., 2022] and its variations [Meng et al., 2024 , Zhang et al., 2023 , Liu et al., 2024a , Dettmers et al., 2024 , Li et al., 2023] , that inject learnable low-rank matrices as an additive injection to the weights of pretrained models. OFT [Qiu et al., 2023] , BOFT [Liu et al., 2024b] and our method use similar approach to LoRA, but learn multiplicative injection rather than an additive one.Structured sparsity Structured sparsity is an approach that replaces dense weight layers with different structured ones, such as matrix factorizations or tensor decompositions in order to compress or speed-up models [Dao et al., 2022 , Chen et al., 2022 , Novikov et al., 2015 , Lebedev et al., 2015] . Some of these techniques were also adapted to PEFT methods in works like [Karimi Mahabadi et al., 2021 , Edalati et al., 2022 , Yang et al., 2024] or BOFT [Liu et al., 2024b] method, that utilizes a variation of butterfly matrices as a parametrization for parameter-efficient orthogonal matrices, imposing orthogonality to each butterfly factor. See details in Section 2. Monarch matrices [Dao et al., 2022 , Fu et al., 2024] are most relevant to our work as our proposed matrix class is their generalization that utilizes similar structure.The emergence of large text-to-image models [Ramesh et al., 2022 , 2021 , Saharia et al., 2022 , Rombach et al., 2022] has propelled the advancement of personalized generation techniques in the research field. Customizing a text-to-image model to generate specific concepts based on multiple input images presents a key challenge. Various methods [Ruiz et al., 2023 , Gal et al., 2022 , Kumari et al., 2023 , Han et al., 2023 , Qiu et al., 2023 , Zhou et al., 2023 , Wei et al., 2023 , Tewel et al., 2023] have been proposed to address this challenge, requiring either extensive fine-tuning of the model as a whole [Ruiz et al., 2023] or specific parts [Kumari et al., 2023] to accurately reconstruct concept-related training images. While this facilitates precise learning of the input concept, it also raises concerns regarding overfitting, potentially limiting the model's flexibility in generating diverse outputs in response to different textual prompts. Efforts to mitigate overfitting and reduce computational burden have led to the development of lightweight parameterization techniques [Qiu et al., 2023 , Liu et al., 2024b , Hu et al., 2022 , Tewel et al., 2023 , Han et al., 2023] such as those proposed among others. These methods aim to preserve editing capabilities while sacrificing some degree of concept fidelity. The primary objective is to identify parameterization strategies that enable high-quality concept learning without compromising the model's ability to edit and generate variations of the concept. Our investigation indicates that the orthogonal parameterization approach we propose represents a significant step towards achieving this goal.In papers [Vorontsov et al., 2017 , Hyland and Rätsch, 2017 , Helfrich et al., 2018] authors work in the setting of dense matrices and try to parameterize essentially the whole manifold of orthogonal matrices. For n × n matrices it requires computing inverses or exponential maps of n × n matrices at every optimization step. This takes O(n 3 ) time, which can be computationally challenging for larger architectures. Moreover, such a parametrization utilizes O(n 2 ) trainable parameters, which makes it inapplicable for PEFT (see the OFT paper [Qiu et al., 2023] for more details). Our proposed method is different in a sense that it provides a trade-off between expressivity (describing only a subset of the orthogonal manifold) and efficiency.In [Li et al., 2019 , Singla et al., 2021] authors discuss main issues of bounding of Lipschitz constant of neural networks and provide Gradient-Norm-Preserving (GNP) architecture in order to avoid vanishing of gradients while bounding Lipschitz constant. The authors propose a specific convolutional layer (Block Convolutional Orthogonal Parametrization) which Jacobian is orthogonal, also providing orthogonal activations with Lipschitz constant equal to 1. These constraints guarantee that the norm of the gradient will not change through backward pass. In other works [Singla and Feizi, 2021, Singla et al., 2021] authors provide a modification of the orthogonal convolutions (Skew Orthogonal Convolution) in terms of hardware-efficiency. Authors provide neural network architecture where each layer is 1-Lipschitz and make a comparison between these two convolutional layers. The work [Li et al., 2019] was outperformed by the SOC method that we utilize (there is a comparison in the SOC method [Singla and Feizi, 2021] ). In [Xu et al., 2022] , the authors utilize periodicity of convolution and their padding is not equivalent to a widely-used zero-padded convolution. Survey [Prach et al., 2024] suggests that [Xu et al., 2022] and other 1-Lipschitz architectures yield worse robustness metrics than SOC, which we use as a baseline in our paper.B Proof of Prop. 1Proof. Let R ′ = P R. R ′ can be viewed as a block matrix withk L × k R blocks of sizes b L 2 × b R 2 .L can be viewed as a block matrix with k L × k L blocks from which only diagonal are non-zero. The A can be written in the following form:   A 0,0 . . . A 0,k R -1 . . . . . . . . . A k L -1,0 . . . A k L -1,k R -1    =    L 0 . . . 0 . . . . . . . . . 0 . . . L k L -1       R ′ 0,0 . . . R ′ 0,k R -1 . . . . . . . . . R ′ k L -1,0 . . . R ′ k L -1,k R -1    .Using block matrix product formulas, we get:A k1,k2 = L k1 R ′ k1,k2 . We can now rewrite L k1 R ′ k1,k2product in terms of their columns and rows:EQUATIONColumns of L k1 are just vectors u j such that ⌊ j k L ⌋ = k 1 . Let us examine the rows of R ′ k1,k2 . Since R ′ is a matrix formed by permuting the rows of block-diagonal matrix R, R ′ k1,k2 can only contain rows that were in the R k2 before permutation. Formally, this means that R ′ k1,k2 can only contain vector-rowsv T i such that ⌊ i k R ⌋ = k 2 .Additionally, rows after permutation should get into the k 1 -block row. That implies ⌊ σ(i) k L ⌋ = k 1 . Other rows of R ′ k1,k2 are zero-rows. Notice that in (4) non-zero rows r ⊤ t represented by v ⊤ i will match exactly with columns u σ(i) that represent l t . Keeping only non-zero terms in t l t r ⊤ t gets us to the desired conclusion.C Comparison of Monarch matrices and GS-matrices GS(P L , P , P r ) class is inspired by Monarch matrices [Dao et al., 2022] and their primary goal is to introduce additional flexibility in the block structure of matrices L and R. Generalized Monarch matrices are parameterized as P 1 LP 2 R, where L and R are block-diagonal matrices and P 1 and P 2 are certain permutations defined in Definition 5.2. This resembles GS(P 1 , P 2 , I) matrix class, however in comparison monarch matrices have additional hard constraints on relation between k L and k R . Being more precise, Monarch matrices are a special case of GS(P 1 , P 2 , I) matrices with additional constraintsk L = b 1 R , k R = b 2 L .Such constraints lead to several theoretical and practical limitations of Monarch matrices. From theoretical point of view, Monarch matrices can only describe permuted block matrices with blocks of ranks 1. In contrast GS-matrices with can describe matrices with different rank structure of blocks (including structures where rank of each block is equal to arbitrary r). From practical point of view, due to this constraint Monarch matrices are often unable to form a desirable block structure of matrices L and R. For demonstration of this phenomena, consider a case of square matrices with square blocks -the structure needed in Orthogonal fine-tuning paradigm. Formally, we have b1 L = b 2 L = b L , b 1 R = b R 2 = b R , m = n. Additional Monarch constraint would mean that b R = k L ; b L = k R . This in turn means that k L • k R = n.As we can see, it makes impossible to stack two matrices with small number of blocks (say, 4) or large number of blocks, which is required in situations with low parameter budget. In contrast, GS parametrization allows for both of these structures, which we use in our experiments.Note, that the work [Fu et al., 2024] provides a slightly different definition for monarch matrices, introducing order-p Monarch matrices. These matrices are also a special case of GS class, however they are very restrictive as they can only parametrize matrices with both sides equal to a p for some integers a, p. We use information transition framework from [Liu et al., 2024b] , representing product of m sparse d × d matrices as an transmitting information in a grid with d × (m + 1) nodes. Edges between nodes j and i represent that element i, j in sparse matrix is non-zero. Element i, j from final matrix can only be non-zero if there exists a path from the j-th node from the right column to the i-th node in the left column (see Figure 5 ).Proof. Consider an information transmission graph for the matrix B i P (r,br) . In this graph, the first node connects with b first edges, the second node connects with the edges from b + 1 to 2b and so on. Now consider a graph for the product of m such matrices. As shown in Figure 5 , now each node from the first level has paths to b k unique nodes from the kth-th level. It means that using m = ⌈log b (d)⌉ = ⌈log b (br)⌉ = 1 + ⌈log b (r)⌉ matrices is sufficient to reach all nodes and therefore form a dense matrix. Note that the number of paths for each node is always equal to b m regardless of permutation choice. This observation shows that it is impossible to reach d unique elements on the final level with m < 1 + ⌈log b (r)⌉.Training details All the models are trained using Adam optimizer with batch size = 4, learning rate = 0.00002, betas = (0.9, 0.999) and weight decay = 0.01. The Stable Diffusion-2-base model is used for all experiments.Evaluation details We use the DreamBooth dataset for evaluation. The dataset contains 25 different contextual prompts for 30 various objects including pets, toys and furnishings. For each concept we generate 10 images per contextual prompt and 30 images per base prompt "a photo of an S*", resulting in 780 unique concept-prompt pairs and a total of 8400 images for fair evaluation.To measure concept fidelity, we use the average pairwise cosine similarity (IS) between CLIP ViTB/32 embeddings of real and generated images as in [Gal et al., 2022] . This means that the image similarity is calculated using only the base prompt, i.e. "a photo of an S*". Higher values of this metric usually indicate better subject fidelity, while keeping this evaluation scene-independent. To evaluate the correspondence between generated images and contextual prompts (TS), the average cosine similarity between CLIP ViTB/32 embeddings of the prompt and generated images [Ruiz et al., 2023 , Gal et al., 2022] .Overfitting discussion Typically, the more parameters are trained, the more easily the model overfits. This results in higher image similarity and lower text similarity. In addition, if a model with a large number of training parameters is trained for a long time, the image similarity can often start to decrease as the model starts to collapse and artifacts start to appear. This sometimes leads to models with more trainable parameters having a worse score in both image and text similarity than the models with fewer ones.Models with fewer trainable parameters overfit less, but require longer training to capture the concept carefully, and usually have an upper limit on the maximum image similarity: at some point the increase in image similarity becomes small, while text similarity starts to decrease dramatically. Therefore, the very common result of a usual fine-tuning is either an overfitting with poor context preservation or an undertraining with poor concept fidelity. The orthogonal fine-tuning shows a different behavior. The model with less trainable parameters can be trained longer (GSOFT, OFT, BOFT) and capture the concept more carefully without artifacts. At the same time, it overfits less and shows higher text similarity.In Figures 6 we show a graphical representation of the metrics for 1000 and 3000 iterations. Examples of generation for different methods are presented in Figure 7 , 8.In this section, we provide some details and insights about the choice of the ChShuffle permutation and the activation function.In experiments, we apply the ChShuffle operation right before grouped convolutional layers. Stacking several layers of that form resembles higher-order GS-matrices, which motivates the usage of permutations from Definition 5.2 for optimal information transition (see Appendix D). However, in the LipConvnet architecture, the activation function can also shuffle information between channels. Thus, this additional shuffling of information can negatively affect our information transition properties. In the original SOC paper [Singla and Feizi, 2021] , the authors use MaxMin activation, firstly proposed in [Anil et al., 2019] . Definition F.1. [Singla and Feizi, 2021] Given a feature tensor X ∈ R 2m×n×n , the M axM in(X) activation of a tensor X is defined as follows:A = X :m,:,: , B = X m:,:,: , M axM in(X) :m,:,: = max(A, B), M axM in(X) m:,:,: = min(A, B).This activation shuffles information between different groups in convolution which harms performance of our experiments, as permutations that we use in ChShuffle become sub-optimal in terms of information transmission. tensor, we use neighboring channels for forming of pairs (first channel pairs with second, third with fourth and so on). With this modification information does not transfer between groups during activations, which enables more optimal information transmission in-between layers with ChShuffle operator. In further experiments we denote this activation function as MaxMinPermuted and define it below: Definition F.2. Given a feature map X ∈ R 2m×n×n . M axM inP ermuted(X) is defined as follows:A = X ::2,:,: , B = X 1::2,:,: , M axM inP ermuted(X) ::2,:,: = max(A, B), M axM inP ermuted(X) 1::2,:,: = min(A, B)However, we also empirically find that it is crucial for the channels that interact within activations functions to also interact during convolutions. This means that they should always stay in the same group. This motivates us to use a slightly different permutation for the ChShuffle operation, which permutes channels in pairs. We use the following permutation This permutation can be seen as an adaptation of P (k,n) that operates on pairs of channels instead of single channels. This permutation is also optimal in terms of information transition. We call this permutation "paired". Using this paired permutation as a ChShuffle with our modified activation saves connection between pairs while also transmitting information in the most efficient way. We provide the results of comparison of approaches with activations and permutations in Table 4 . Justification: Yes, we wrote the information about the resources used for experiments in Section 7.σ(i) paired (k,n) = i 2 mod k • n k + 2 • i 2k + (i mod 2)Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes, our research conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, we cite all papers appeared in the text.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
 
 Appendix:  


Decision: The paper introduces a novel structured matrix class, GS-matrices, which unifies and extends existing classes like Monarch and Block Butterfly matrices. The proposed approach is applied to orthogonal finetuning, showing parameter efficiency across different tasks, including text-to-image diffusion models and language modeling. Reviewer dcw2 praised the novelty and practical impact of the GS parameterization, rating it as an accept due to the strong potential for broader application in orthogonal finetuning. However, the reviewer suggested incorporating comparisons with Monarch parameterization and releasing the implementation for reproducibility, which the authors addressed by providing the code and discussing the limitations of Monarch matrices.

Reviewer AkW1 highlighted the theoretical contributions and the method’s ability to reduce parameters and computation, but expressed concerns over the small scale of experiments, lack of training time data, and missing comparisons with newer LoRA methods like DoRA. The authors responded by adding training times and conducting additional comparisons, yet AkW1 maintained a borderline reject rating due to perceived limitations in empirical validation.

Reviewer yzRC found the presentation clear but questioned the efficiency claims without wall-clock time data and suggested ablation studies for Double GSOFT. They also noted the need for a broader discussion on previous orthogonal matrix parameterization methods. The authors responded with additional data and a more detailed survey of related work, but yzRC still rated the paper as borderline reject, citing incremental contributions and limited experimental validation.

Reviewer Kbzt recognized the broad application of GSOFT but recommended additional training time comparisons and broader evaluations on recent models like Llama2. Although the authors provided further clarifications and data, Kbzt did not raise their score, keeping it at borderline accept, indicating that while their concerns were addressed, broader experimental evidence was still needed.

Overall, the paper presents a promising and technically solid contribution to orthogonal finetuning. However, the concerns regarding empirical validation and broader impact, as raised by multiple reviewers, suggest that further refinement and additional experiments would strengthen the paper. Given the addressed feedback and the potential impact, I recommend acceptance, with an emphasis on expanding the experimental evaluations in future work to fully realize the method’s contributions.

Reviews: review: paper_summary: paper_summary: This paper proposes a new parameterization for orthogonal matrices and applies this parameterization to orthogonal finetuning of foundation models. The new orthogonal parameterization is interesting and useful. The experimental results show some improvement over the baselines.
strength_weakness: strength_weakness: - Although it is trivial to show that GS matrices can approximate arbitrary orthogonal matrices (as it generalizes the Monarch matrices), it is still important to discuss and (potentially re-use) the theoretical results from the Monarch matrices.

- It can further strengthen the paper, if the authors can add the comparison of the Monarch parameterization in the orthogonal finetuning experiments.

- The implementation should be released prior to its acceptance, otherwise it may be difficult for people to reproduce the results.
questions: questions: See the weakness section.
limitations: limitations: The major concern that I has is whether the results are easily reproducible. With that being said, the authors should consider to release the source code for the orthogonal finetuning experiments.
paper_summary: paper_summary: This paper introduces a new way of constructing orthogonal parametrization for efficient model fine-tuning. The development of the proposed method is motivated by: naive orthogonal fine-tuning, which can be too restrictive due to how it constructs the block-diagonal orthogonal matrix; and improved methods with dense orthogonal matrix can be too expensive in terms of both computation and parameters. 
The authors introduce a new way of constructing orthogonal matrix by an alternating product of block-diagonal matrices and several permutations, with advantages over previous methods justified theoretically. 
Experiments on language model, stable diffusion, and standard conv models are presented.
strength_weakness: strength_weakness: My major concern with this paper is the empirical justifications. 

**Small experiment scale.**
All experiments are conducted in very small scale. Specifically, the language models used in this paper are relatively old and outdated; Only few-shot personalization experiments are reported in the stable diffusion experiments; and the final conv application is only conducted on a tiny network on cifar100. 
This makes it very hard to objectively evaluate the proposed method's practical effectiveness.

**Missing evaluations**
If I'm not missing anything, the entire *training time* row of Table 2 is empty. Training time and memory are the core advantages compared to the previous methods. While it is true that the method is able to reduce the parameters and computations by numbers, significant speed-up and memory reduction in practice can still be a huge support to the paper's contribution. 

**Missing comparisons**
The comparisons reported in this paper are mainly on LoRA and other orthogonal tuning methods. However, without the comparisons against other improved LoRA methods, such as DoRA, it can be hard to position the contribution and impact of this work.
questions: questions: - Does the SVD need to be performed in every optimization step?
limitations: limitations: The limitation discussions presented in Section 9 focus mainly on how the model compares to LoRA and BOFT. The proposed method still requires more parameters than other improved LoRA methods.
paper_summary: paper_summary: This paper follows the orthogonal fine-tuning paradigm that uses orthogonal matrices for adapting the weights of a pretrained model. It introduces a class of structure matrices, defined as GS-matrices, that can well be used to conduct orthogonal matrix. Based on GS-matrices, It proposes structured orthogonal parametrization and studies its performance in the orthogonal fine-tuning framework. Experiments are conducted on two standard setting ups: downstream task fine-tuning in language modeling and text-to-image diffusion tasks. It also claims the proposed method can be used to construct orthogonal convolutions for 1-Lipschitz neural networks.
strength_weakness: strength_weakness: 1.The experiments are not well conducted to support the claims/effectiveness of the proposed method:  
(1) This paper spends many words to claim the efficiency of the proposed methods. However, this paper does not provide the experiments to show the wall-clock time in the natural language understanding and subject-driven generation. I think this paper should provide the wall-clock time when comparing to baselines in Table 1 and Table 2, to show the efficiency of the proposed methods, compared to the baselines. What is the results of wall-clock time of these methods in Table 1 and 2?  
(2) This paper should conduct some ablation study to the Double GSOFT. It seems that Double GSOFT is the key to improve the performance, based on the description of this paper, but the double idea is apparently incremental. Besides, why Table 1 show the results using GSOFT, while Table 2 using Double GSOFT? Why not provide both in the experiments or using the best one (maybe Double GSOFT) with ablation study? I think this paper should address this point in the rebuttal. 

2.This paper claims that “Nevertheless, parametrization of orthogonal matrices is a challenging task” In Line 19. I do not think so. There are many previous solutions to conduct orthogonal matrix in DNNs before 2019[1,2,3,4], even for conducting orthogonal convolutions [5,6]. And there are merits and drawbacks among these methods. I think this paper should well discuss these previous methods and take credits to them. 

3.The original motivation of this paper is for parametrization efficient finetuning by using orthogonal fine-tuning. Meanwhile, it also wants to claim its contributions to the application orthogonal matrix benefits, E.g., conducting 1-Lipschitz network for robustness. 
But this claim is not well supported by experiments (There are many other orthogonal convolution methods(e.g., [5] and [6]), not only the baselines in the paper. E.g. SOC). Besides, this paper should pay more attention on discussing the orthogonal techniques in DNNS during the related work section. 


**Ref:**  
[1] On orthogonality and learning recurrent networks with long term dependencies, ICML 2017   
[2] Learning unitary operators with help from u(n), AAAI 2017  
[3] Orthogonal recurrent neural networks with scaled Cayley transform, ICML, 2018  
[4] Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, AAAI 2018  
[5] Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks, NeurIPS 2019  
[6] LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness, NeurIPS 2022
questions: questions: The questions in weakness 1
limitations: limitations: Yes
paper_summary: paper_summary: In this paper, the authors proposed a new structured matrices, namely GS-matrices. It unifies and generalizes structured classes from previous works. Compared to the previous BOFT, it further improves the parameters and training efficiency. Experiments on language processing, text-image generation, and image classification have demonstrated the performance of GSOFT.
strength_weakness: strength_weakness: 1. It's better to provide the training time comparison between BOFT, LoRA, and GSOFT. It seems that some results are lost in Table 2.
2. It's better to discuss more about single GSOFT and double GSOFT. It seems that the results in Table 1&3 are from the single version and the results in Table 2 are from the double version.
3. In Table 2, it seems that more learnable parameters result in lower CLIP scores. Maybe more discussions are required.
questions: questions: My main concerns are listed in the weaknesses.
1. It's better to demonstrate the performance of the proposed method on recent popular GPT series methods (e.g., Llama2, Llama3, Mixtral). This may expand the influence of this method in the field.
limitations: limitations: N/A

Comments: title: Thanks for Response
Thanks for the response
comment: We would like to thank the reviewer for the response!
Please let us know if you have any follow-up questions that need further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that might be helpful.
Please let us know if you have any follow-up questions that need further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that might be helpful.
We would like to thank the reviewer for the response!
Thanks for the authors' response. My main concerns have been addressed. Thus, I will keep my score.
My concerns are properly addressed. I think this paper proposes an interesting parameterization for orthogonal matrices and is worth discussing at the conference. I will keep my current score and recommend for acceptance.