Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Authors: Zeju Qiu
Weiyang Liu
Haiwen Feng
Yuxuan Xue
Yao Feng
Zhen Liu
Dan Zhang
Adrian Weller
Bernhard Schölkopf
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-image
diffusion models
finetuning
generative models
orthogonality
TL;DR: A simple yet effective method for finetuning large text-to-image diffusion models

Abstract: Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.

Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Zeju Qiu
Weiyang Liu 1255 Liu
Haiwen Feng', Yuxuan Xue"
Yao Feng' Bernhard Schölkopf'
Zhen Liu
Dan Zhang
Adrian Weller
'MPI for Intelligent Systems-
Tubingen
University of Cambridge
*University of Tübingen "The Alan Turing Institute
Mila, Université Montréal
'ossh Center Artificial Intelligence
'Equal contribution 'Project lead
oft. att.wyliu.com
Abstract
Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyper- spherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic gen- eration ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes additional radius constraint to the hypersphere. Specifically, consider two important finetuning text-to-image tasks: subject- generation where the goal is to generate subject-specific images given few images of subject and text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that OFT framework outperforms existing methods generation quality and convergence speed.
1 Introduction
Recent text-to-ing text-to-image diffusion models [45, 50, 53] achieve impressive performance in text- eext-guided control for high-fidelity image generation. Despite strong results, text guidance can still be ambiguous and insufficient to provide fine-graine graingainn and accurate control to the generated images. To address this shortcoming, we target two types of text-to-image generation tasks this paper:
Subject-driven generation [51]: Given just few images of subject, the task is to generate images the same subject in a different context using the guidance text prompt.
Controllable generation [38, 68]: Given an additional control signal (e.g., canny edges, segmenta- tion maps), the task is to generate images following such a control signal and a text prompt.
Both tasks essentially boil down to how to effectively finetune text-to-image diffusion models without losing the pretraining generative performance. We summarize the desiderata for an effective finetuning method as: (1) training efficiency: having fewer trainable parameters and number of training epochs, and (2) generalizability preservation: preserving the high-fidelity and diverse generative performance. To this end, finetuning is typically done either by updating the neuron weights by small learning rate (e.g., [51]) by adding small component with re- re-parameterized neuron weights (e.g.. [22, 68]).
Despite simplicity, neither finetuning strategy is able to guarantee the preservation of pretraining generative performance. There is also a lack of principled understanding towards designing good finetuning strategy and finding suitable hyperparameters such as the number of training epochs. key difficulty is the lack of measure for quantifying the preservation of pretrained generative ability. Existing finetuning methods implicitly assume that smaller Euclidean distance between the
This work was finished when ZQ was a research intern hosted by WL at MPI for Intelligent Systems.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Toll prumpe a [V] dlg with mountain in the background
Text peoupt: a house
Drcunfoood
ContooNer
Groand
Ixpul imagos
EaRA
Control signal
11penpance (2et7)
Narbor of Feraioos
Iuratos 400
Recation 1460
leratice 2300
lurnice 2000
LoRA GFT
5% dirz
10% data
50% Lura
Subject-driven generation
(b) Controllable generation
Figure Subject-driven generation: OFT preserves the hyperspherical energy and yields more stable finetuning performance across different number of iterations, while both DreamBooth [51] and LoRA [22] do not, OFT can preserve hyperspherical energy and perform stable tinetuning. is more sample efficient training and converges well only 5% of the original dataset, while hoth ControlNet [68] and LoRA [22] cannot converge until 50% of the data is present. The hyperspherical energy comparison between LoRA and OFT fair, since they finetune the same layers. ControlNet uses different layer finctuning strategy, so hyperspherical energy not compurable. The detailed settings given in the experiment section and Appendix A.
while both LoRA DreamBaavih are unable. (b) Controllable generation:
finetured model and the pretrained model indicates better preservation of the pretrained ability. Due to the same reason, finetuning methods typically work with a very small learning rate. While this assumption may occasionally hold, we argue that the Euclidean difference to the pretrained model alone unable to fully capture the degree of semantic preservation, and therefore a more structural measure to characterize the difference between the finetuned model and the pretrained model can greatly benefit the preservation of pretraining performance as well as finetuning stability.
Inspired by the empirical observation that hyperspherical similarity encodes semantic information well [7, 35, 36], we use hyperspherical energy [32] characterize the pairwise relational structure among neurons. Hyperspherical energy is defined as the sum of hyperspherical similarity (e.g.. cosine similarity) between all pairwise neurons in the same layer, capturing the level of neuron uniformity on the unit hypersphere [34]. We hypothesize that good finetuned model should have a minimal difference in hyperspherical energy compared to the pretrained model. naive way is to add a regularizer such that the hyperspherical energy remains the same during the finetuning stage, but there no guarantee that the hyperspherical energy difference can be well minimized. Therefore, we take advantage of an invariance property of hyperspherical energy the pairwise hyperspherical similarity provably preserved if we apply the same orthogonal transformation for all neurons. Motivated by such an invariance, we propose Orthogonal Finetuning (OFT) which adapts large eext-to-image diffusion models to a downstream task without changing its hyperspherical energy. The central idea is to learn a layer-shared orthogonal transformation for neurons such that their pairwise angles are preserved. OFT can also be viewed as adjusting the canonical coordinate system for the neurons in the same layer. By jointly taking into consideration that smaller Euclidean distance between the finetuned model and the pretrained model implics better preservation of pretraining performance, we further propose OFT variant Constrained Orthogonal Finetuning (COFT) which constrains the finetuned model within the hypersphere of a fixed radius centered on the pretrained neurons.
The intuition for why orthogonal transformation works for finetuning neurons partially comes from 2D Fourier transform, with which an image can be decomposed as magnitude and phase spectrum.
The phase spectrum, which is angular information between input and basis, preserves the major part of semantics. For example, the phase spectrum of an image, along with a random magnitude spectrum, can still reconstruct the original image without losing its semantics. This phenomenon suggests that changing the neuron directions is the key to semantically modifying the generated image (which is the goal of both subject-driven and controllable generation). However, changing neuron directions with large degree of freedom will inevitably destroy the pretraining generative performance. To constrain the degree of freedom, we propose to preserve the angle between any pair of neurons, largely based on the hypothesis that the angles between neurons are crucial for representing the knowledge of neural networks. With this intuition, it is natural to learn layer-shared orthogonal transformation for neurons in each layer such that the hyperspherical energy stays unchanged.
We also draw inspiration from orthogonal werrppaametterized training [33] which trains classification neural networks from scratch by orthogonally transforming a randomly initialized neural network. This is because a randomly initialized neural network yields provably small hyperspherical energy in

---

expectation and the goal of [33] is to keep hyperspherical energy small during training (small energy leads to better generalization in classification [30, 321). [33] shows that orthogonal transformation sufficiently flexible to train generalizable neural networks for classification problems. In contrast, we focus on finetuning text-to-image diffusion models for better controllability and stronger downstream generative performance. We emphasize the difference between OFT and [33] in two aspects. First, while [33] is designed to minimize the hyperspherical energy, OFT aims to preserve the same hyperspherical energy as the pretrained model so that the intrinsic pretrained structure will not be destroyed by finetuning. In the case of finetuning diffusion models, minimizing hyperspherical energy could destroy the original semantic structures. Second, OFT seeks to minimize the deviation from the pretrained model, which leads to the constrained variant. In contrast, [33] imposes no such constraints. The key to finetuning is to find a good trade-off between flexibility and stability, and we argue that our OFT framework effectively achieves this goal. Our contributions are listed below:
We propose novel finetuning method Orthogonal Finetuning for guiding text-to-image diffusion models towards better controllability. To further improve stability, we propose a constrained variant which limits the angular deviation from the pretrained model.
Compared to existing finetuning methods, OFT performs model finetuning while provably pre- serving the hyperspherical energy, which we empirically find to be an important measure of the generative semantic preservation of the pretrained model.
We apply OFT to two tasks: subject-driven generation and controllable generation. We conduct comprehensive empirical study and demonstrate significant improvement over prior work in terms of generation quality, convergence speed and finetuning stability. Moreover, OFT achieves better sample efficiency, as it converges well with a much smaller number of training images and epochs.
For controllable generation, lability. This core idea is to estimate the control signal from the generated image and then compare it with the origin control signal. The metric further validates the strong controllability of OFT.
introduce new control consistency metric to evaluate the control-
2 Related Work
Text-to-image diffusion models. Tremendous progress [16, 39, 45, 50, 53] has been made in ext-to-image generation, largely thanks to the rapid development in diffusion-basee generative models [12, 20, 55, 56] and vision-language representation learning [1. 28, 29, 37, 44, 54, 57, 61]. GLIDE [39] and Imagen [53] train diffusion models in the pixel space. GLIDE trains the text encoder jointly with diffusion prior using paired text-image data, while Imagen uses a frozen pretrained text encoder. Stable Diffusion [50] and DALL-E2 [45] train diffusion models in the latent space. Stable Diffusion uses VQ-GAN [14] to learn a visual codebook as its latent space, while DALL- E2 adopts CLIP [44] to construct a joint latent embedding space for representing images and text. Other than diffusion models, generative adversarial networks [27, 48, 65, 67] and autoregressive models [13, 46, 62, 66] have also been studied in text-to-image generation. OFT inherently a model-agnostic finetuning approach and can be applied to any text-to-image tomee diffusion model.
Subject-driven generation. To prevent subject modification, [2. 39] consider given mask from users as an additional condition. Inversion methods [8, 12, 15, 45] can be applied to modify the context without changing the subject. [18] can perform local and global editing without input masks. The methods above are unable to well preserve identity-relate
details of the subject. In Pivotal Tuning [49], generator is finetuned around an initial inverted latent code with an additional regularization to preserve the identity. Similarly, [41] learns a personalized generative face prior from collection of a person's is face images. [6] can generate difference variations of an instance, but it may lose the instance-specific details. With customized token and a few subject images, DreamBooth [51] finetunes the text-to-image diffusion model using a reconstruction loss and a class- specific prior preservation loss. OFT adopts the DreamBooth framework. but instead of performing naive finetuning with a small learning rate, OFT finetunes the model with orthogonal transformations.
Controllable generation. The task of songe-to-image translation can be viewed as a form of controllable generation, and previous methods mostly adopt conditional generative adversarial networks [9, 23, 42, 60, 71]. Diffusson models are also used for image-to-image translation [52, 58, 59]. More recently, ControlNet [68] proposes to control a pretrained diffusion model by finetuning and adapting it to additional control signals and achieves impressive controllable generation performance. Another concurrent and similar work, T2I-Adapter [38], also finetunes a pretrained diffusion model in order to gain stronger controllability for the generated images. Following the same task setting
3

---

in [38, 68], we apply OFT to finetune pretrained diffusion models, yielding consistently better controllability with fewer training data and less finetuning parameters. More significantly, OFT does not introduce any additional computational overhead during test-time inference.
Model finetuning. Finetuning large pretrained models on downstream tasks has been increasingly popular nowadays [3, 11, 17J. As a form of finetuning, adaptation methods (e.g., [21, 22, 43J) are heavily studied in natural language processing, LoRA [22] is the most relevant work OFT, and assumes a low-rank structure for the additive weight update during finetuning. In contrast, OFT uses layer-shared orthogonal transformation to update neuron weights in a multiplicative manner, and provably preserves the pair-wise angles among neurons in the same layer, yielding better stability.
3
Orthogonal Finetuning
3.1 Why Does Orthogonal Transformation Make Sense?
We start by discussing why orthogonal transformation is desirable in finetuning eexttto-image diffusion models. We decompose this question into two smaller ones: (1) why we want to finetune the angle of neurons (i.e., direction), and (2) why we adopt orthogonal transformation to finetune angles.
For the first question, we draw inspiration from the empir- ical observation in [7, 35] that angular feature difference well characterizes the semantic gap. SphereNet [36] shows that training neural network with all neurons normalized onto a unit hypersphere yields comparable capacity and even better generalizability. implying that the direction of neurons can fully capture the most important information from data. To better demonstrate the importance of neuron angles, we conduct toy experiment in Figure 2 where we train a standard convolutional autoencoder on some flower images. In the training stage, we use the standard inner product to produce the feature map (z denotes the element output of the convolution kernel and æ is the input in the sliding window). In the testing stage, we compare three ways to generate the feature map: (a) the inner product used in training, (b) the magnitude information, and (c) the angular information. The results in Figure 2 show that the angular information of neurons can almost perfectly recover the input images, while the magnitude of neurons contains no useful information. We emphasize that we do not apply the cosine activation (c) during training, and the training is done only based on inner product. The results imply that the angles (directions) of neurons play the major role in storing the semantic information of the input images. In order to modify the semantic information of images, finetuning the neuron directions will likely be more effective.
z- w TIx
Convoluticn
Decioivolation
(b) z- 1l-I-I1x0l
== coc(
Feature map
(a) Inner product
Magnitude
Angle
Figure luy experiment demonstrate impur- tance of angular information. The autoencoder is trained in standard way using inner product activation, and (a) shows the standard reconstruction. In testing, the angular information of neurons alone can well recover the input image, even the autocncoder not trained with angles.
For the second question, the simplest way to finetune direction of neurons is to simultancously rotate / reflect all the neurons (in the same layer), which naturally brings in orthogonal transformation.
vol otthograal w-la orihogon
Onginil image Contel signal
Epoch
It may be more flexible to use some other angular transformation that rotates different neurons with different angles, but find that orthogonal transformation is a sweet spot between flexibility and regularity. Moreover, [33] shows that orthogonal transformation is sufficiently powerful for learning neural networks. To support our argument, we perform an experiment to demonstrate the effective regularization induced by the orthogonality constraint. We perform the controllable generation experiment using the setting of Con- trolNet [68], and the results are given in Figure 3. We can observe that our standard OFT performs quite stably and achieves accurate control after the training is finished (epoch 20). In comparison, OFT without the orthogonality constraint fails to generate any realistic image and achieve no control effect. The experiment validates the importance of the orthogonality constraint in OFT.
Epoch
Figure 3: Controllable generation with or without orthogonality, Middle column the original OFT, and the right columm is given hy OFT without the orthagonality constraint.
3.2 General Framework
The conventional finetuning strategy typically uses gradient descent with small learning rate to update model (or certain layers of model). The small learning rate implicitly encourages a small
4

---

deviation from the pretrained model, and the standard finetuning essentially aims to train the model while implicitly minimizing || | where M is the finetuned model weights and M is the pretrained model weights. This implicit constraint makes intuitive sense, but it can still be too flexible for finetuning large model. To address this, LoRA introduces an additional low-rank constraint for the weight update, i.e., rank M - M =r' where r' is set to be some small number. Different from LoRA, OFT introduces a constraint for the pair-wise neuron similarity: [[HE(M) -BBE) HE(M)) | where HE(-) denotes hyperspherical energy of weight matrix. As an illustrative example, we consider fully connected layer W= .............................. w } E R where w, E R is the i-th neuron (W the pretrained weights). The output vector # E R" of this fully connected layer is computed by z= W x where x E IRd is the input vector. OFT can be interpreted as minimizing the hyperspherical energy difference between the finetuned model and the pretrained model:
min HE(WW - HB(W")|l
min
Σ Lē. - ,ا, Σ |100 - 66-
(I)
where w;:=w;/ //::|| denotes the i-th normalized neuron, and the hyperspherical energy of a fully connected layer W is defined as HE( W): Eii; |,;0; One can easily observe that the attainable minimum is zero for Eq. (I). The minimum can be achieved as long as W and W differ only up to a rotation or reflection, i.e. W RW in which dddd is an orthogonal matrix (The determinant 1 or -1 1 means rotation reflection, respectively). This is exactly the idea of OFT, that we only need to finetune the neural network by learning layer-shared orthogonal matrices to transform neurons in each layer. Formally, OFT seeks to optimize the orthogonal matrix R E Rdd for a pretrained fully connected layer W E Rdxn, changing the forward pass from z= (W") x to
z W x. = (R. WT) x., s.t. RR R RR =I
(2)
where W denotes the OFT-finetuned weight matrix and I is an identity matrix. OFT is illustrated in Figure 4. Similar to the zero initialization in LoRA, we need to ensure OFT to finetune the pretrained model exactly from W To achieve this, we initialize the orthogonal matrix R to be an identity matrix so that the finetuned model starts with the pretrained weights. To guarantee the orthogonality of the matrix R, we can use differential orthogonalization strategies discussed in [26, 33]. We will discuss how to guarantee the orthogonality a computationally efficient way.
3.3 Efficient Orthogonal Parameterization
Standard orthogonalization such as Gram- Schmidt method, despite differentiable, is often too expensive compute in prac- tice [33]. For better efficiency, we adopt Cayley parameterization to generate the orthogonal matrix. Specifically, we con- struct the orthogonal matrix with R= (I+Q) Q) where Q is a skew-symmetric matrix satisfying Q QT Such an efficiency comes at a small price - the Cayley parameterization can only produce orthogonal matrices with determinant 1 which belongs to the special orthogonal group. Fortunately, we find that such a limita- tion does not affect the performance in practice. Even if we use Cayley transform parameterize the orthogonal matrix, R can still be very parameter-ineffiieen with a large d. To address this, we propose to represent R with a block-diagonal matrix with blocks, leading to the following form:
Orthogonal
Pretmined Weight Matrix
K R
Pretrained Weight Manrix 1p"
Matrix
X
X
R
Orthogoend Maria
(n) Original OFT
OFT with block-diagonal It
Figure (a) Original OFT without diagonal structure. (b) OFT diagonal blocks of the same size. When r = 1, the case of (b) recovers the case of (a).
R1 E 0(5)
R (Ri, R2.... ,R.)
E O
(3)
R. 0
where O(d) denotes the orthogonal group in dimension d, and Re and R, ERd//rx//r Vi are orthogonal matrices. When 1= then the block-diagnn orthogonal matrix becomes a standard unconstrained one. For an orthogonal matrix with size d x d, the number of parameters is d(d-1)/ 1)/2. resulting in a complexity of OdP). For an r-block diagonal orthogonal matrix, the number of parameter is d(d/r 1)/2, resulting in complexity of (dl /r). We can optionally share the block matrix to further reduce the number of parameters, i.e., R, RR=E, R,, Vi This reduces the parameter complexity to (d²/r2). Despite all these strategies to improve parameter efficiency, we note that the resulting matrix R remains orthogonal, so there is sacrifice in preserving hyperspherical energy.
5

---

We discuss how OFT compares to LoRA in terms of parameter efficiency. For LoRA with a low-rank parameter r', we have its number of trainable parameters as r' nn). If we consider both r and r' to be dependent on the neuron dimension d (e.g.. r- r' = ad where 0 < is some constant), then the parameter complexity of LoRA becomes Od + dn) and the parameter complexity of OFT becomes (d). We illustrate the difference in complexity between OFT and LoRA with a concrete example. Suppose we have a weight matrix with size 128 x 128, LoRA has 2, 048 trainable parameters with r'=8, while OFT has 960 trainable parameters with r 8 (no block sharing is applied).
3.4 Constrained Orthogonal Finetuning
We can further limit the flexibility of original OFT by constraining the finetuned model be within small neighborhood of the pretrained model. Specifically, COFT uses the following forward pass:
W æ = (R. WO) æ. S.L. R R RR = I, R 5
(4)
which has an orthogonality constraint and an c-deviation constraint to an identity matrix. The orthogonality constraint can be achieved with the Cayley parameterization introduced in Section 3.3. However, it nontrivial to incorporate the e-deviation constraint to the orthogonal matrix. To gain more insights on the Cayley transform, we apply the Neumann series to ap- as R I + +Q ((Q)) (under the assumption that the Neumann
proximate R= (I ()
series converges in the operator norm). Therefore, we can move the constraint R Se inside the Cayley transform, and the equivalent constraint is ||Q -|Q-0|| 0 e' where 0 denotes an all-zero ma- trix and e' is another error hyper-
Text prompt: a [V] toy on the beach
6-D
e-1e-7d
8-1e-6d
6-1e-5-d
e-le-4d
OFT (6=x)
Input image
Figure
How affeects the Blxxiiliity of COFT subject driven generation.
parameter (different than c). The new constraint on the matrix Q can be easily enforced by projected gradient descent. To achieve identity initialization for the orthogonal matrix R. we initialize Q as an all-zero matrix. COFT can be viewed as combination of two explicit constraints: minimal hyperspherical energy difference and constrained deviation from the pretrained model. The second constraint is usually implicitly used by existing finetuning methods, but COFT makes it explicit one. Despite the excellent performance of OFT, we observe that COFT yields even better finetuning stability than OFT due to this explicit deviation constraint. Figure 5 provides example how e affects the performance of COFT. We can observe that controls the fiexibility of finetuning. model resembles the OFT-finctuned model. With smaller the
With larger e, the COFT-finetuncd model behaves increasingly similar to the pretrained text-to-image diffusion model.
3.5 Re-scaled Orthogonal Finetuning
We propose a simple extension to the original OFT by additionally learning a magnitude scaling coefficient for each neuron. This is motivated by the fact that re-scaling neurons does not change the hyperspherical energy (the magnitude will be normalized out). Specifically, we use the forward pass: == =DDRW00) x where D D= diag((11 ... , sa) Radd is a diagonal matrix with all the diagonal element $1, Ca larger than zero. In contrast to OFT's original forward pass in Eq. (2) where only R is learnable, we have both the diagonal matrix D and the orthogonal matrix R learnable. The re-scaled OFT will further improve the flexibility of OFT with neglectable number of additional trainable parameters. Note that, we still stick the original OFT in the experiment section order to demonstrate the effectiveness of orthogonal transformation alone, but actually find that the re-scaled OFT can generally improve performance (see Appendix C for empirical results).
4 Intriguing Insights and Discussions
OFT is agnostic to different architectures. We can apply OFT to any type of neural network in principle. For Transformers, LoRA is typically applied to the attention weights [22]. To compare fairly to LoRA, we only apply OFT to finetune the attention weights in our experiments. Besides fully connected layers, OFT also well suited for finetuning convolution layers, because the block- diagonal structure of R has interesting interpretations in convolution layers (unlike LoRA). When we use the same number of blocks as the number of input channels, each block only transforms a unique neuron channel, similar to learning depth-wise convolution kemels [10]. When all the blocks in are shared, OFT transforms the neurons with an orthogonal matrix shared across channels. We conduct preliminary study on finetuning convolution layers with OFT in Appendix D

---

Connection to LoRA. By adding low-rank matrix, LoRA prevents the information in the pretrained weight matrix from shifting dramatically. In contrast, OFT controls the transform that applies to the pretrained weight matrix to be orthogonal (full-rank), which prevents the transform to destroy the pretraining information. We can rewrite OFT's forward pass as 2 == (RW") T ax (Wo + (R I) where (R- I) W is analogous LoRA's low-rank weight update. Since W is typically full-rank, OFT also performs low-rank weight update when R R I- is low-rank. Similar to LoRA that has rank parameter r', OFT has a diagonal block parameter r' to reduce the number of trainable parameters. More interestingly, LoRA and OFT represent two distinct ways to be -effnnttt--ffiiinttt LoRA exploits the low-rank structure to reduce the number of trainable parameters, while OFT takes a different route by exploiting the sparsity structure (i.e., block-diagonal orthogonality).
Why OFT converges faster. On one hand, we can see from Figure 2 that the most effective update to modify the semantics to change neuron directions, which is exactly what OFT is designed for. On the other hand, OFT can be viewed as finetuning neurons on a smooth hypersphere manifold, which yields better optimization landscape. This is also empirically verified in [33].
Why not minimize hyperspherical energy. One of the key difference [33] is that we do not aim to minimize hyperspherical energy. This is quite different from classification problems where neurons without redundancy are desired. The minimum hyperspherical energy means all neurons are uniformly spaced around the hypersphere. This is not a meaningful objective to finetune a pretrained generative model, since will likely destroy the pretraining information.
Trade-off between flexibility and regularity in finetuning. We discover an underlying trade-off be- tween flexibility and regularity. Standard finetuning is the most flexible method, but it yields poor sta- bility and easily causes model collapse. Being surprisingly simple, OFT finds a good balance between flexibility and regularity by preserving the pairwise neuron angles. The block-diagonal parameteriza- tion can also be viewed as a regularization of the orthogonal matrix. Although limits the flexibility, it brings additional regularity and stability. COFT is also another example of limiting the flexibility.
No additional inference overhead. Unlike ControlNet, our OFT framework introduces no additional inference overhead the finetuned model. In the inference stage, we can simply multiply the learned orthogonal matrix R into the pretrained weight matrix 8 and obtain an equivalent weight matrix W RRW Thus the inference speed is the same as the pretrained model.
5 Experiments and Results
General settings. In all the experiments, we use Stable Diffusion v1.5 [50] as our pretrained text- to-image diffusion model. All the compared methods finetune the same pretrained model under the same setting. For fairness, we randomly pick generated images from each method. For subject-driven generation, we generally follow DreamBooth [51]. For controllable generation, we generally follow ControlNet [68] and T21-Adapter [38]. To ensure a fair comparison to LoRA, we only apply OFT or COFT to the same layer where LoRA is used. More results and details are given in Appendix A.
5.1 Subject-driven Generation
Text prompt: a [V] cat wearing yellon shirt .etteoeneeenn
loput imagess
Settings. We use DreamBooth [51] and LoRA [22] as the baselines. All the methods adopt the same loss function as in DreamBooth. For DreamBooth and LoRA, we generally follow the original paper and use the best hyperparameter setup. More results are provided in Appendix F.E.F.J.
Uier 400
Finetuning stability and convergence. We first evalu- ate the finetuning stability and the convergence speed for DreamBooth, LoRA, OFT and COFT. Results are given in Figure I and Figure 6. We can observe that both COFT and OFT are able finetune the diffusion model quite stably, After 400 iterations, both DreamBooth and OFT variants achieve good control, while LoRA fails to preserve the subject identity. After 2000 iterations, DreamBooth starts to generate collapsed images, and LoRA fails to generate yellow shirt (and instead generates yellow fur). In contrast, both OFT and COFT are still able to achieve stable and consistent control over the generated image.
Iuer 1000
Iee 2000
DreamRooth
LoRA
COFT
OFT
Figure 6: Generated images across different iterations.
These results validate the fast yet stable convergence of our OFT framework in subject-driven genera-
7

---

Impul
[V] beed with blur home the background
Input imssgas
A the beach
a [V] boxl with tree and nutumu kaaves the backgrsand
A cleck en top green gras with sunflower
DreamBooth
LoRA
COFT
OFT
DreamBoooh
LoRA
COFT
OFT
Imput images
[V] dog in police ouetit
Input imagas
[V] staffed animaal top of pink fabric
wearlng black
stuffed animal
DreamBooth
LoRA
COFT
OFT
DreamBooth
LoRA
COFT
OFT
Figure 7: Qualitative comparison of subject driven generation among DreamBooth, LoRA, COFT and OFT. Results are generated with the same finetuned model from each method. All examples randomly picked. The figure best viewed digitally, color and significanly zoomed in.
tion. We note that the insensitivity to the number of finetuning iteration is quite important, since it can effectively alleviate the trouble of tuning the iteration number for different subjects. For both OFT and COFT, we can directly set a relatively large iteration number without carefully tuning it.
For COFT with a proper e, both the learning rate and the iteration number become effortless to set.
Quantitative comparison. Following [51], we conduct a quantitative comparison to evaluate subject fidelity (DINO [5], CLIP-I [44]), text prompt fidelity (CLIP- T [44J) and sample diversity (LPIPS [69]). CLIP-I com- putes the average pairwise cosine similarity of CLIP embeddings between generated and real images. DINO is similar to CLIP-I, except that we use VIT S/16 DINO embeddings. CLIP-T is the average cosine similarity of CLIP embeddings between text prompt and generated images. We also evaluate average LPIPS cosine simi- larity between generated images of the same subject with the same text prompt. Table show that both COFT and OFT outperforms DreamBooth and LoRA in the DINO and CLIP-I metrics by a considerable margin, while achieving slightly better or comparable performance in prompt fidelity and diversity metric. For each method, we repeatedly finetune the same pretrained model with 30 different random seeds minimize randomness. The results show that our OFT framework not only achieves better convergence and stability, but also yields consistently better final performance.
Method
DINO 1
CLIP-1
CLIP-T 1 LPIPS
Real Images
0,703
0,864
0.695
DreamBovit
0.614
0.778
0.239
LoRA
0.613
0.765
0.237
0.744
COFT
0.630
0.783
0.235
0.744
OFT
0.632
0.785
0.237
0.746
Table 1: Quantitative comparison subject fidelity (DINO,
CLIP-D. D). prompt fidelity (CLIP-T) and diversity metric (I.PIPS). evaluation images and prompts are
[25] subjects with 30 text prompts each subject).
Qualitative comparison. To have more intuitive understanding of OFT's benefits, show some randomly picked examples for subject-driven generation in Figure 7. For a fair comparison, all the examples are generated from the same finetuned model using each method, so no text prompt will be separately optimized for its final results. For each method, we select the model that achieves the best validation CLIP metrics. From the results in Figure 7, we can observe that both OFT and COFT deliver excellent semantic subject preservation, while LoRA often fails to preserve the subject identity (e.g., LoRA completely loses the subject identity in the bowl example). In the meantime, both OFT and COFT have much more accurate control using text prompts, while DreamBooth. despite its preservation of subject identity, often fails to generate the image following the text prompt (e.g., the first row of the bowl example). The qualitative comparison demonstrates that our OFT framework achieves better controllability and subject preservation at the same time. Moreover, the number of iterations is not sensitive in OFT, so OFT performs well even with large number of iterations, while neither DreamBooth nor LoRA can. More qualitative examples are given Appendix F. Moreover, we conduct human evaluation in Appendix H which further validates the superiority of OFT.
5.2 Controllable Generation
Settings. We use ControlNet [68], T21-Adapter [38] and LoRA [22] as the baselines. We consider three challenging controllable generation tasks in the main paper: Canny edge to image (C2I) on the
8

---

COCO dataset [3 I], segmentation map to image (S2I) on the ADE20K dataset [70] and landmark to face (L2F) on the CelebA-HQ dataset [25, 63). All the methods are used to finetune Stable Diffusion (SD) v1.5 on these three datasets for 20 epochs. More results are given in Appendix F,G.J.
Convergence. We evaluate the convergence speed of ControlNet, T21- Adapter, LoRA and COFT on the L2F task. We provide both quantitative and qualitative evaluation. Specifically for the evaluation metric, we compute the mean & distance between control face landmarks and pre- dicted face landmarks. In Figure 8, we plot the face landmark error obtained by the model finetuned with different number of epochs. We can observe that both COFT and OFT achieve significantly faster con- vergence. It takes 20 epochs for LoRA to converge to the performance of our OFT framework at the 8-th epoch. We note that OFT and COFT use a similar number of trainable parameters to LoRA (much fewer than NontrolNet), while being much more efficient to converge than existing methods. On the other hand, the fast convergence of OFT is also validated by the results in Figure 1. The right example in Figure I shows that OFT is much more data-efficient than ControlNet and LoRA, since OFT can converge well with only 5% of the ADE20K dataset. For qualitative results, we focus on comparing OFT, COFT and ControlNet, because ControlNet achieves the closest landmark error to ours. Results in Figure show that both OFT and COFT converge stably and the generated face pose is gradually aligned with the control landmarks. In contrast to our stable and smooth convergence, the controllability in ControlNet suddenly emerges after the 8-th epoch, which perfectly matches the sudden convergence phenomenon observed in [68]. Such a convergence stability makes our OFT framework much easier to use in practice, since the training dynamics of OFT is far more smooth and predictable. Thus it will be easier to find good OFT's hyperparameters.
10
Landmok emorr
v. 11
Adlesii
ContooNNei
LoRA
COFT
.OFT
Number epochs
Figure 8: Face landmark ervor.
man with long bair wearing
CommoNeet
Quantitative comparison. We introduce control consistency metric to evaluate the performance of controllable generation. The basic idea is to com- pute the control signal from the generated image and then compare it with the original input control sig- nal. For the C2I task, we compute IoU and Fl score, For the S2I task, compute mean IoU, mean and overall accuracy. For the L.2F task, we compute the mean lg distance between control landmarks and predicted landmarks. More details regarding the consistency metrics are given in Appendix A. For all the compared method, we use the best possible hyperparameter settings. Results in Table 2 show that both OFT and COFT yield much stronger and accurate control than the other methods. We observe that the adapter-based approaches (e.g., T21-Adapter and ControlNet) converge slowly and also yield worse final results. Compared ControlNet, LoRA performs better in the S2I task and worse in the C2I and L2F tasks. In general, we find that the performance ceiling of LoRA is relatively low, even we have carefully tuned its hyperparameters. As comparison, the performance of our OFT framework has not yet saturated, since we empirically find that it still gets better as the number of trainable parameters gets large. We emphasize that our quantitative evaluation in controllable generation is one of our novel contributions, since it can accurately evaluate the control performance of the finetuned models (up to the accuracy of the off-the-shelf segmentation/dellction model).
shirt
COFT
Control
Ground truth
Epoch
Epoch 6
Epoch
Epoch 10
Figure 9: Qualitative examples with different number of epachs.
Qualitative comparison. We also qual- itatively compare OFT and COFT to current of-te-of-the-ar methods, includ- ing ControlNet. T2I-Adapter and LoRA. Randomly generated images in Fig- ure 10 show that OFT and COFT not only yield high-fidelity and realistic im- age quality, but also achieve accurate control. In the S2I task, we can see that LoRA completely fails to generate images following the input segmentation map, while Control- Net, OFT and COFT can well control the generated images. In contrast to ControlNet, both OFT and COFT are able to generate high-fidelity images with more vivid details and more reasonable
Task
Metric
SD
ControlNet
T21-Adapee
LoRA
COFT
OFT
IoU
0.049
0.189
0.078
0.168
0.195
0.193
C21
F
0.093
0.317
0.143
0.286
0.325
m
7.72
20.88
16.38
22.98
26.92
27.06
mAce
14.40
30.91
26.31
35.52
40.08
40.09
aAce 1
33.61
61.42
$1.63
58.03
62.96
62.42
L2F
Errer 4
146.19
7.61
23.75
7.68
6.92
7.07
Table 2: Quantitative comparison of control signal consistency for three control tasks (Canny edge image, segmentation to image landmark face).

---

tropical inland
building
building
truck
women with curly
Landmark
wearing
red shirt
Ground tuth
Concrol
ControlNet
T21 121-Adapter
LoRA
COFT
OFT
Figure 10: Qualitative comparison controllable generation. The ligure best viewed digitally. color and signilicantly zoomed
geometric structures with far less model parameters. In the C2I task, both OFT and COFT are able to hallucinate semantically similar images based rough Canny edges, while T21-Adapter and LoRA perform much worse. In the L2F task, our method produces the most accurate pose control for the generated faces even under challenging face poses. all three control tasks, show that both OFT and COFT produce qualitatively better images than the state-of-the-art baselines, demonstrating the effectiivneess of our OFT framework in controllable generation. To give a more comprehensive qualitative comparison, we provide more qualitative examples for all the three control tasks in Appendix F.2, and moreover, we demonstrate OFT can perform well on more control tasks (including dense pose to human body, sketch to image and depth to image) in Appendix G.
6 Concluding Remarks and Open Problems
Motivated by the observation that angular information among neurons crucially determines visual semantics, we propose a simple yet effective finetuning method orthogonal finetuning for controlling text-to-image diffusion models. Specifically, we target two text-to-image applications: subject-drives generation and controllable generation. Compared existing methods, OFT demonstrates stronger controllability and finetuning stability with fewer number of finetuning parameters. More importantly, OFT does not introduce additional inference overhead, leading to an efficient deployable model.
OFT also introduces few interesting open problems. First, OFT guarantees the orthogonality via Cayley parametrization which involves a matrix inverse. It slightly limits the scalability of OFT. Although we address this limitation using block diagonal parametrization, how speed up this matrix inverse in differentiable way remains a challenge. Second, OFT has unique potential in compositionality, in the sense that the orthogonal matrices produced by multiple OFT finetuning tasks can be multiplied together and remains an orthogonal matrix. Whether this set of orthogonal matrices preserve the knowledge all the downstream tasks remains an interesting direction to study. Finally, the parameter efficiency of OFT is largely dependent on the block diagonal structure which inevitably introduces additional biases and limits the flexibility. How to improve the parameter efficiency in more effective and less biased way remains important open problem.
10

---

Acknowledgement
The authors would like to sincerely thank Luigi Gresele, Yandong Wen, Yuliang Xiu and many other colleagues at Max Planck Institute for Intelligent Systems for many helpful suggestions.
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tubin- gen AI Center, FKZ: 011S18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. WL was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP XX, project number: 276693517. AW acknowledges support from Turing AI Fellowship under grant EP/VO22279/1, The Alan Turing Institute, and the Leverhulme Trust via CFI.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022. 3
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint wxXiv2226.002799 2022. 3
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. In NeurlPS, 2020. 4
[4] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In ICCV, 2017. 17
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 8
[6] Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal. and Adriana Romero Soriano. lnstance-conditionedd gan. In NeurlPS, 2021.
[7] Beidi Chen, Weiyang Liu, Zhiding Yu, Jan Kautz, Anshumali Shrivastava, Animesh Garg, and Animashrec Anandkumar, Angular visual hardness. ICML, 2020. 4
[8] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
[9] Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Jung-Woo Ha. Sunghun Kim, and Jacgul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-iman imaae translation. In CVPR, 2018.
[10] François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 6
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 181.0008,, 2018. 4
[12] Prafulla Dhariwal and Alexander Nichol. Diffiusion models beat gans on image synthesis. In NeurlPS, 2021. 3
[13] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering texi-to-image generation via transformers. In NeurlPS. 2021. 3
[14] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis.
In CVPR, 2021. 3
[15] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Patashník, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 2rXiv:2288.066/33 2022. 3
[16] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for texl-to-image synthesis. In CVPR, 2022. 3
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022. 4

---

[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman. Yael Pritch, and Daniel Cohen-Or Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint urXiv:2208.. 0/6266, 2022. 3
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two simel update rule converge to local nash equilibrium. NewrIPS, 2017. 17
[20] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[21] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficice transfer learning for nlp. In ICML, 2019. 4
[22] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 1, 2, 4. 6, 7, 8, 8,20
[23] Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros, Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 3
[24] Yuming Jiang. Shuai Yang, Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022. 16
[25] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICIRR, 2018. 9, 16
[26] Mario Lezcano-Casaan and David Martinez- Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In ICML, 2019. 5
[27] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. In NeurlPS, 2019.
[28] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301 125977 2023.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image image pre- ge-training for unified lision-language understanding and generation. In JCML, 2022. 3, 16
[30] Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James M Rehg. Li Xiong, and Le Song. Regularizing neural networks via minimizing hyperspherical energy. In CVPR, 2020. 3
[31] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 9, 16
[32] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. In NeurlPS, 2018. 2,
[33] Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg. Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal over- per-parameterized training. In CVPR, 2021. 2, 3, 5,7
[34] Weiyang Liu, Rongmei Lin, Zhen Liu, Xiong, Bernhard Schölkopf, and Adrian Weller. Learning with hyperspherical uniformity. In AISTATS, 2021. 2
[35] Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang. James M Rehg, and Le Song. Decoupled networks. In CVPR, 2018. 2,4
[36] Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In NIPS, 2017. 2,
[37] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining aask-agnostic visiolinguistic representations for vision-and-language tasks. In NeurlPS, 2019. 3
[38] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for texi-to-image diffuuson models. arXiv preprint arXiv:2302.08453, 2023. 3.3,4. 4, 8
[39] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, 1lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
12

---

[40] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics & Image Processing, 2008. 16
[41] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. ACM Transactions Graphics (TOG), 11(6);1-10, 2022.
[42] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In CVPR, 2019. 3
[43] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adaptcrfusion: Non-destructive task composition for transfer learning, arXiv preprint arXiv:2005.002 2020. 4
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021. 3, 8
[45] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.00 06125, 2022. 1,3
[46] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 3
[47] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler. and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE trunsactions on pattern onalysis and machine intelligence, (4(3):1623-163) 2020. 16
[48] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICMI, 2016. 3
[49] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics, 42(1):1 13, 2022. 3
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 3,
[51] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 1, 1, 3, 8. 16
[52] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH 2022, pages 1-10, 2022. 3
[53] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. In NeurIPS, 2022. 1,3
[54] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08400. 2022.
[55] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 3
[56]
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. based generative modeling through stochastic differential equations. In ICLR, 2021.
[57] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2020. 3
[58] Andrey Voynov, Ktir Aberman, and Daniel Cohen-Or. iketch-guided text-to-image diffusion models. arXiv preprint arXiv:2211. 2022. 3
[59] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image -ommaae translation. arXiv preprint rxXv:2205.11952. 2022. 3
[60] Ting- Ting-Chun Wang. Ming-Yo Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018. 3
[61] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.1000 2021.
13

---

[62] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022. 3
[63] Weihao Xia, Yujiu Yang, Jan Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In CVPR, 2021. 9
[64] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021. 17
[65] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR. 2018.
[66] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022. 3
[67] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017. 3
[68] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXin:3302.055444 2023. 1, 2, 3, 7, 8, 9, 16
[69] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 8
[70] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 9, 16
[71] Jun-Yan 7hu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal maage-to-image translation. In NewrIPS, 2017. 3
14

---

Appendix
Table of Contents
A Experimental Details
16
B Effeet of Different Number of Diagonal Blocks
18
C Experiments on Re-scaled OFT
19
D Applying OFT to Convolution Layers
20
E
Comparison between COFT and OFT
21
F
More Qualitative Results
23
F.1
Subject-driven Generation
23
F.2
Controllable Generation
25
G
More Controllable Generation Tasks
34
G.1 Dense Pose to Human Body
34
G.2 Sketch to Image
36
G.3 Depth to Image
38
H Human Evaluation
40
Style Transfer by Adapting Stable Diffusion with Orthogonal Finetuning
41
J Failure Cases
42
J.1 Failure Cases in Subject-driven Generation
42
J.2 Failure Cases in Controllable Generation
43
15

---

A Experimental Details
To verify the effectiveness of our Orthogonal Fine-tuning (OFT) method, we extensively evaluate the performance our method in two common text-to-image generation tasks: subject-driven generation and controllable generation. More specifically, we use the exact same task setting as ControlNet [68] and Dreambooth [51] and the baseline implementations were sourced from the GitHub repository Diffusers' and ControlNet?.
Data and Model. For training the convolutional autoencoder from Figure 2, we use 1000 random images from the Oxford 102 Flower dataset [40]. For the task of subject-driven generation, we use the official DreamBooth dataset, which consists of 30 subjects from 15 different classes. For each subject, there are several images and 25 different text prompts. For generating the image-control-caption combinations, we use BLIP [29] to automatically caption the images (pre-trained model weight and code for captioning based the GitHub repository BLIP'). Note, although COCO provides captions for the training and validation split, to be consistent with other image-control coptrollaattoooionnn combinations, instead the SL.P-generaate captions as text prompts. For the C2I task, we the whole COCO 2017 dataset [31] with in total of 180K images; we generate canny edge images as the control signal using the same canny edge detector as ControlNet. For the S2I task,
the semantic segmentation dataset ADE20K [70] with in total of 24K image-segmentation mask pairs. For the L2F dataset, we use the CelebA-HQ dataset [25], which contains 30K images. Additionally, we demonstrate that OFT also works well in other controllable generation tasks, including depth-to- image (D2I), densepose-to-image (P2I), and sketch-to-image (SK2I). For the D2I task, we also use the COCO dataset and employ MiDaS [47] to generate depth maps; the pre-trained weights are obtained from the GitHub repository MiDaS For the P2I task, we use the Caa-Faasion-uutiiMaaee
dataset [24] with in total of 44K clothed human images with the corresponding densepose. For the Sk2I task, we use subset of the LAION-AAsthen dataset with approximately 350K images to learn sketch-guidee image generation. We use the Stable Diffusion v1.5$ as the base model.
Subject-driven generation. For training our subject-driven generation diffusion model, we follow the training objective of Dreambooth. More specifically, we use the class-specific prior preservation loss fine-tune our orthogonal matrices:
E,,,,e,e', [w (| (aqæ ofe, c) x112 Awa @ (ap + oy Gpr) -pp (2))
(5)
with Cpr being the class conditioning vector. For calculating the prior-preservation loss, we additionally need generate 200 images using the subject's class prompt. Similar to LoRA, we inject our trainable orthogonal matrices into the attention modules of the stable diffusion model. To be comparable with LoRA, we choose the exact same linear layers as LoRA to affect upon: the linear layers Wg. We. SXM2-32GB GPU using a learning rate of 6 x 10-5, batch size of 1, and train for approximately 1000 iterations. In the case of COFT, we use 6 = 6 x 10-5 to constrain the orthogonal matrices.
We and W.. We perform training on 1 Tesla
Controllable generation. Apart from injecting the trainable OFT weights into the stable diffusion model, we need to add small encoding model to stable diffusion to encode the control signal. To be comparable with ControlNet [68], we use the same encoding module, which a shallow 8-layer convolutional network with Scaled Exponential Linear Unit (SELU) activation functions. We also the same training objective as ControlNet. The control signal is encoded and concatenated once with the input to the stable diffusion U-Net. For the LoRA baseline, we use the same encoding module to encode the control signal. For $21, L21 and P21, we fine-tune the model for 20 epochs; for C2I and D2I we fine-tune the model for 10 epochs; for Sk21 we fine-tune the model for 8 epochs. We perform training on 4 NVIDIA A SXM4-80GB GPUs using a learning rate of 1 X 10 batch size of 4 for L2I and batch size of 16 for the rest of tasks. For fine-tuning with COFT,
1 10TH
http://gghub..oom/uggggggg////// P http://hhhhh.ooollllssss/loootto 3 4 http:hhhwhccmmiisoo////iiaaaaii
5tblob/main/v11--ppunndd
16

---

Evaluation. When evaluating the effectiveness of controllable generation, we primarily focus on evaluating the controllability. Using the consistency metrics introduced in the main paper, we can effectively compute the difference between the control signal and the generated image. For the C2I task, we apply the identical canny filter on the generated image to determine a canny image of the predicted image. Both the control signal canny image and the canny image obtained from the generated images are pixel-wise Intersection over Union (IoU) and FI score between these two canny predictions. For the S2I task, we compute mean IoU, mean and overall accuracy by deploying a pre-trained semantic segmentation model. More specifically, ADE20K (Segformer-Bes BB), to perform semantic segmentation on our generated images. We use the segmentation accuracy
images, with pixel values being either or 1. We evaluate the
use the Segformer" [64] model, which is trained on
indication for the overall semantically and structural resemblance of the generated images to the ground truth image. For the 1,2F task, we compute the mean f2 distance between the input control landmarks and the landmarks estimated from generated images using facial landmark detector [4].
We also evaluate the generation performance by calculating Fréchet Inception Distance (FID) [19], we use the default setting of the GitHub repository pytorch-fid'. The FID is metric quantifying the similarity between two image dataset. It utilizes 2048-dimensions features, which are derived from the final average pooling layer of pretrained Inception V3 network trained on ImageNet dataset. A lower FID score indicates a higher similarity between the datasets.
17

---

B Effect of Different Number of Diagonal Blocks
We note that the number of diagonal blocks is an important hyperparameter that effectively controls the number of trainable parameters. It is necessary to perform sensitivity study on this hyperparameter. Following the same settings as the main paper, we evaluate how r affects OFT the S2I task. Results in Table show that smaller r (closer to recovering the standard orthogonal matrix) generally works better than larger т. However, we find that a good trade-off between flexibility and parameter-efficieng indeed exists. Empirically, we find that we can use much bigger r if the dataset simple, leading to better parameter-efficinncy and faster convergence. because we find that 4 works well across datasets and tasks. Note that, in terms of the number of inference parameters, both LoRA and OFT have the exact same number of parameters, which is equal to the number of parameters of the underlying stable diffusion model, while ControlNct has an additional control model with 361M parameters.
the main paper,
always
ControlNet
2
16
Trainable Parameters
361.3
29.5 M
16.3
9.7 M
Inference Parameters
1.42 B
1.06 B
1,06
1,06
1.06 B
minU f
20.88
27.18
27.06
24.09
21.0
mAccc
30.91
39.39
40.09
36.95
32.55
a/ccc 1
61.42
65.24
62.96
60.25
55.5
Table 3: How the number of diagonal blocks affects the control capability of OFT.
18

---

C Experiments on Re-scaled OFT
Since both OFT and COFT transform neurons with orthogonal matrices and do not affect the magnitude of neurons, their magnitude may be sub-optimal with their updated orientations. To address this issue, we propose a re-scaled OFT where the neuron magnitude is refined using the same set of data in the downstream task. Specifically, re-scaled OFT further finetunes the magnitude of neurons without changing their directions. re-scaled OFT can performed in two manners: (1) joint filling: magnitude fitting can be used simultaneously with OFT or COFT, and (2) Post-stage fitting: magnitude fitting can be used after OFT or COFT is finished. An important motivation for re-scaled OFT comes from Figure 2, where perfectly preserves visual structures, but also results in a certain degree color distortion. We hypothesize that this minor color distortion is caused by magnitude loss and fixing this issue can potentially improve the visual quality of generated images.
observe that constructing images only with angular information
Notably, re-scaled OFT does not change the hyperspherical energy since it does not change the direction of neurons all the nice properties OFT and COFT on hyperspherical energy are still perfectly preserved. Therefore, the advantage of structural preservation is also inherited.
To simplify the experiments and validate the effectiveness of re-scaled OFT, perform post-stage magnitude fitting on the COFT model and compare the FID between the original validation images and the generated images (using the control signals extracted from validation images). The reason we use FID here is that FID is more sensitive to color distortion, while the consistency metric only measures the structural preservation. Table 4 shows that magnitude fitting can indeed improve the FID of COFT and is beneficial to COFT.
Magnitude fitting is lightweight and can be implemented by simply adding trainable parameter for each layer we modify; the parameter has the shape of (N x 1), with corresponds to the number of neurons in that specific layer. The performance gain shown in Table 4 is achieved by performing Post-stage fitting on a COFT-finetuned model for only one additional epoch. Moreover, we expect that the joint fitting re-scaled OFT can lead to better performance.
ControlNet
FID
41.2
30.9
33.1
30.9
30,8
30.2
Table 4: FID on the segmentation to image task (ADE20K), = 4 is used here.
19

---

D Applying OFT to Convolution Layers
In the original setting [22]. LoRA is only applied to the linear layers of the attention modules. To be fair comparison, we also apply OFT to these weights. However, OFT is not limited to linear layers but can easily be adapted convolution layers by transforming the convolutional neurons. We highlight the compatibility of OFT and COFT for finetuning convolution layers. More interestingly, sharing parameters diagonal blocks
becomes interpretable convolution layers. With a suitable setup, orthogonal matrices with sharing diagonal blocks can transform the convolution kernel in channel-sharing manner (or a spatial manner), implying that the same orthogonal transformation applied to all channels. This shares similar intuition with depth- wise convolution.
For this ablation experiment, we study the performance of applying OFT to the convolution layers in the ResNet blocks of the stable diffusion model. In this experiment, we use COFT as the baseline method and consider the controllable generation (segmentation to image) as an example. We have both quantitative (Table 5) and qualitative results (Figure 11). We can empirically observe that by only fine-tuning the convolutional layers, we can also achieve some degree of control. By simultaneously fine-tuning both linear and convolutional layers, we achieve slightly better FID score. Note, for fine-tuning convolutional layers, we let r be equal to the number of channels convolutional neurons in that layer.
COFT (attention)
COFT (conv)
COPT (extended)
FID
30.8
39.8
30.4
Table FID resalts applying COFT different types layers. (with 4)
Text prompt: the inside of airplane
Text prompt: a room with paintings on wall
Text prompt: a sunset
Text prompt: city
Original image
Control
Convolation
Attention
Both
Figure 11: Controllable generation results of applying COFT to differrnn types of layers.
20

---

E Comparison between COFT and OFT
We have already provided many qualitative examples for COFT and OFT in the main paper. One may question the fundamental difference between OFT and COFT. Based on the intuition behind COFT, the deviation constraint introduced improve the training stability. We demonstrate the training stability of COFT with qualitative example in subject-driven generation. Results in Figure 12 and Figure 13 show that, despite being much more stable than existing methods, OFT will eventually generate collapsed images at the 9000-th iteration. In contrast, COFT still produces visually appealing images. We train both OFT and COFT with learning rate of 10 and constrain COFT with
1 x 10-5
Text prompt: [V] dog top of wooden floor
COFT
OFT
Text prompt: a [V] dog wearing yellow shirt
COFT
Input images
OFF
Text prompt: a [V] teapot on top of dirt road
COFT
OFF
Text prompt: a [V] teapot on top of a dirt road
COFT
Input images
OFF
Iteration 2000
Iteration 9000
Figure 12: Qualitative comparison hetween COFT and OFT on subject driven generation.
21

---

Text prompt: [V] vase top of the sidewalk a crowded street
COFT
Text prompt: [V] vase with city in the back ground
COFT
OFT
Text prompt: a [V] vase on top of dirt road
COFT
Input images
Text prompt: [V] vase with a tree and autumn leaves in the background
COFT
OFT
Iteration 2000
Iteration 9000
Figure 13: Qualitative comparison between COFT OFT on subject-driven generation.
22

---

More Qualitative Results
F1
Subject-driven Generation
prompt: backpack floating on top of
Text prompt: [V] backpack the snow
Text prompt: backpack the beach
Text prompt: pupple backpack
Text prompt: [V] cat in firefighter outfit
Text prompt: [V] cat on top of purple rug in a forest
Text prompt: [V] cat wearing red bat
Text prompt: a wet [V] cut
Ingul inages
DreamBooth
LoRA
COFT
OFT
Figure 14: More qualitative results on subject-driven generation.
23

---

Text prompt: [V] dog in snow
Text prompt: [V] dog wearing yellow shirt
Text prompt: [V] blue house the background
prompt: police
Text prompt: [V] vase cubblestone street
Text prompt: [V] vasc on the beach
Text prompt: [V] wheat field the background
Text prompt: purple [V] vase
Input images
DreamBooth
LoRA
COFT
OFT
Figure 15: More qualitative results subject driven generation.
24

---

F22
Controllable Generation
F.2.1
Segmentation to Image
Text prompt: an auditorium
Text prompt: two beds in room
Text prompt: gallery
Text prompt: thomass train disneyland world
Text prompt: small room with a bed and shelves
Text prompt: bathroom with sink and toilet
Text prompt: house
Original image
Control
ControlNet
COFT
OFT
Figure 16: More qualitative results of OFT and COFT segmentation image generation
25

---

prompt: living
Text prompt: living
Text prompt: road the middle of nowhere
Text prompt: a hallway building
Text prompt: the sun shining through clouds over a lake
Text prompt: person kitehen
Text prompt: bears
Original image
Control
ControlNet
COFT
OFT
Figure 17: More qualitative results of OFT and COFT on the segmentation to image generation task.
26

---

Text prompt: bedroom
Text prompt: dining with table and chairs
Text prompt: the living room
Text prompt: living room with vase of flowers
Text prompt: u volleyball court
Text prompt: bedroom
Text prompt: bedroom
Original image
Control
ControlNet
COFT
OFT
Figure 18: More qualitative results of OFT and COFT on the segmentation to image generation task.
27

---

F.2.2 Canny Edge to Image
prompt: with the words
ME
SEOT BROWN RRA
SLoT
SCOTT BROWN SPAR
SPOWL
ROON
BROWN
HERPER
HEAFERA
BROWN
MAR
Text prompt: sign pole
BICYCLES SKATES SRATEBOARDS
JICCLLSS SEATES SHATPOLARDS
BICYCLES. SKATES SKATEBOARDS
BICYCLES SKATES SHATEBOARDS
Text prompt: elephunts in the wild
Text prompt: a man holding a skateboard
Text prompt: woman red hair
Text prompt: big and the parliument
Text prompt:
Original image
Control
ControlNet
COFT
OFT
Higure 19: More qualitative result of OFT and COFT's on the Canny edge to image generation task.
28

---

prompt: train going through tunnel
Text prompt: sandwich
Text prompt: people on the beach at sunset
Text prompt: a slice of pizza on table
Text prompt: bathroom with toilet and a shower
Text prompt: bathroom toilet bucket
Text prompt: living room
Original image
Control
ControlNet
COFT
OFT
Figure 20: More qualitative results OFT and COPT on the Canny edge to image generation task.
29

---

Text prompt:
Text prompt: bowl food table
Text prompt: two white horses in field
Text prompt: a vase on table
Text prompt: a fire truck parked parking
Text prompt: bathroom
Text prompt: bus parked park
Original image
Control
ControlNet
COFT
OFT
Figure 21: More qualitative results OFT and COPT on the Canny edge to image generation task.
30

---

F2.3
Landmark to Face
Text prompt: woman with red hair
Text prompt: a man wearing a hat
Text prompt: wearing glasses
D
Text prompt: man in tuxedo
Text prompt: man with glasses smiling
Text prompt: woman with blonde hair
X
on
Text prompt: a woman with long brown hair
Original image
Control
ControlNet
COFT
OFT
Figure 22: More qualitative results OFT and COFT the landmark face generation task.
31

---

prompt: beard
Text prompt: wearing glasses
Text prompt: a man in a suit and tie
N
Text prompt: man suit and
Text prompt: man long
Text prompt: in suit and tie
Text prompt: with long
Original image
Control
ControlNet
COFT
OFT
Figure 23: More qualitative results OFT and COPT on the landmark to face generation task.
32

---

prompt: woman with long
Text prompt: red lipstick
Text prompt: a woman with blonde hair
u
#
Text prompt: young man with brown hair
Text prompt: young man
Text prompt: beautiful
Text prompt: cating food
Original image
Control
ControlNet
COFT
OFT
Figure More qualitative results OFT and COPT on the landmark to face generation task.
33

---

G More Controllable Generation Tasks
G.1 Dense Pose to Human Body
Text prompt: young man wearing a grey t-shirt and black jeans
Text prompt: wearing grey t-slirt
prompt: white shirt green
Text prompt: man white and black jeans
Text prompt: man black jucket and jeuns
prompt: young man wearing black white jeans
Text prompt: young in white t-sbirt and rod
Text prompt: man wearing white r-shirt and jeans
prompt: wearing shirt
jeans
Original image
Control
ControlNet
T21-Adapter
LoRA
COFT
OFT
Figure Qualitative comparison among
dense pose body generation
34

---

Text prompt: man in black sweater and jeans
Text prompt: man grey swenter blue jeans
Text prompt: man wearing denim shirt white pants
Text prompt: a man wearing white t-shirt and jeans
Text prompt: man wearing denim jacket and jeans
Reconstruction
Control signal
Figure 26: More qualitative results COPT and OFT the dense human hody task.
35

---

G.
Sketch to Image
Text prompt: group people standing front of the white
Text prompt: boathouse
Text prompt: boat the beach
Text prompt: mountain with it
Text prompt: old driving down snowy road
Text prompt: painting of castle
HERL
Text prompt: family with dog
Text prompt: the golden gate bridge
Text prompt: deer in the snow
Original image
Control
ControlNet
T21-Adapler
LoRA
COFT
OFT
Figure Qualitative comparison among
image generation
36

---

Text prompt: mountain lake with rocks and trees
Text prompt: river with ship in the distance
Text prompt: a river the mountains
Text prompt: an old town with church in the background
Text prompt: castle with dragon flying over it
Text prompt: a boat with lightning in the sky
Text prompt: a man with blonde hair
Original image
Control
ControlNet
COFT
OFT
Figure 28: More qualitative comparison the sketch image generation task.
37

---

G.3
Depth to Image
prompt:
Text prompt: motoreycle display
Text prompt: toilet bathroom
Text prompt: city
Text prompt: hospital
Text prompt: painting
Text prompt: airplane the airport
Text prompt: cut in suiteuse
Text prompt: kitchen
Original image
Control
ControlNet
T21-Adapler
LoRA
COFT
OFT
Figure Qualitative comparison among different depth to image generation
38

---

Text prompt: mountain lake with rocks and trees
Text prompt: river with ship in the distance
Text prompt: river the mountains
Text prompt: an old town with church in the background
Text prompt: castle with dragon flying over it
Text prompt: a boat with lightning in the sky
Text prompt: a man with blonde hair
Original image
Control
ControlNet
COFT
OFT
Figure 30: More qualitative comparison the depth to image geacration task.
39

---

H
Human Evaluation
Human evaluation settings. We also carried out a structured human evaluation for the subject- driven generation task, involving 50 participants. Here's s a breakdown of our evaluation process:
Selection of subjects: picked subjects from the DreamBooth dataset at random.
Image and prompt: for every subject, unique text prompts were chosen at random. This resulted total of 28 distinct subject-prompl combinations. For every single one of the tasks, we randomly sampled an image generated by each of the three methods DreamBooth, LoRA, and OFT.
Every participant was asked to answer three single-selection questions for each task:
Subject fidelity: which image best preserves the identity of the subject? In other words, which generated image resembles the most the original subject?
Text alignment: which image matches the given text description the best?
Overall image quality: out of the options, which image has the best overall quality?
The methods were assessed at two specific points during their fine-tuning phase: at the 1000th iteration, checkpoint where these methods typically exhibit best performance, and at the 10,000th iteration, a checkpoint used to measure the stability the finetuning process over an extended period.
The results are given in Table 6, indicating the proportion of participants who chose partic- ular method based on the above criteria. We can see that OFT more favored after finetuning Stable Diffusion with 1000 iterations and after 10000 iterations. We note that OFT delivers significantly better image quality and text-following ability than both DreamBooth and LoRA after relatively large number of finetuning iterations.
Results.
Iteration 1000
Iteration 10000
Metric
DreamBooth
LoRA
OFT
DreamBonth
LoRA
OFT
Subject Fidelity
42.0%
15.4%
42.6%
22.4%
1.4%
76.2%
Text Alignment
18.6%
24.7%
56.7 %
2.6%
1.4%
96.0%
Overall Image Quality
35.7%
19.2%
45.1%
0.8%
Table Participant voting percentages subject lidelity. aligument overall image quality.
40

---

I Style Transfer by Adapting Stable Diffusion with Orthogonal Finetuning
Stable Diffusion can generate images based on the input text prompts. Without any adaptation. inputting text prompts to pre-trained Stable Diffusion model will result in images that resemble natural images. We can finetune the pre-trained Stable Diffusion model on custom dataset, to adapt the style of the generated images to the custom dataset. To demonstrate the effectiveness of orthogonal finetuning, we show qualitative results of adapting Stable Diffusion on the Sketch Scene dataset after finetuning for 20000 iterations in Figure 31 and on the Wikiart dataset after finetuning for 30000 iterations Figure 32, We train on a single NVIDIA A100-SXM4- SXM4-80GB GPU using
learning rate of 1 10-1 batch size of 1, and 4 as the number of gradient accumulation steps.
Genernted imaage
Text prompt
Before OFT finetaning
Afler fimetuning
Bench kepe
the hill
A House in the field
strect the snow
PONCHON
walking graes
dog grabbing flying
hard of achow
buy riding like the soad
a big new and bench neor the hillside
a train
Example image pairs the sketch seene dacaset
Text-to generation aher finetuning the sketch-seene dataset
Figure 31: Direct OFT Finetuning of Stable Diffusion on the sketch scene dainset.
Generated image
Text prompt
Before OFT finctuning
After OFT finetuning
landscape the style of impressioaism
a cityscape in the style post impresiionis
restauratt on city strect
portalt the landscape the syyl of Improsiosiom
lundscape style realism
place
portmit the style impresiuniim
autrrcea night lime
Example image- pains wikiart caption
Text-te image generation finetuning wikiart caption
Figure 32: Direct OFT Finctuning of Stable Diffusion the wikiart-caption dataset.
>hupss//huggingffcc http://hhngggggiiccoowwwsssttett c//ttnss////hhhhhcrccccc.ce hhup:://inniiiiiicccoooossssssin
41

---

Failure Cases
We also show few failure cases OFT and COFT. Figure 33 gives three failure cases in subject- driven generation. Figure 34 gives three failure cases in controllable generation.
J.1 Failure Cases in Subject-driven Generation
In subject-driven generation, OFT and COFT will sometimes fail to ground the text attribute the intended object. In the cat example, both OFT and COFT will sometimes generate other red objects, instead of generating a red cat.
Text prompt: cube shaped [V] dog
COFT
Text prompt: fleating in milk
COFT
OFT
Text prompt: [V]
COFT
OFF
Input images
Figure 33: Some failure cases in subject-driven generation.
42

---

J.2 Failure Cases in Controllable Generation
Both OFT and COFT will sometimes hallucinate complicated structural details large region with the same semantics. Despite still being visually plausible, these generated images cannot match the original segmentation maps.
Text prompt: field with trees and mountains in the background
Text prompt: living
prompt: building leading up
Original image
Control
COFT
OFT
Figure 34: Some failure cases in conirollable generation.
43


Reviews:
[{"review_id": "S1Rb1vMcJg", "reviewer": "Reviewer_styH", "summary": "This paper proposes a new method for finetuning text-to-image diffusion models for better guidance and control. The proposed method OFT is a low-cost finetuning method even in comparison with LoRA, and it can preserve semantic generation ability well. Furthermore, this work proposes COFT to improve the stability of OFT. Experimental results demonstrate impressive performance of the finetuned model on subject driven generation and controllable generation.", "questions": "1. Is this method generalizable to other models, say large language models?", "limitations": "The authors did not discuss the limitations and introduce their future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method is light-weight but yield highly stable finetuning performance in both subject driven generation and controllable generation.\n2. The method in simple and well correspond to the motivation. This paper explains the motivation about orthogonal transformation well.", "weaknesses": "1. Unfortunately, this paper is incomplete, which does not consist of further discussion and conclusion.\n2. In comparison with Dreambooth, the advantages are relatively small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the recognition of our contribution, motivation and novelty. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n---\n\n**Q1: Unfortunately, this paper is incomplete, which does not consist of further discussion and conclusion.**\n\nA1: We are very sorry for not including a conclusion section due to the page limit. We will add the conclusion in the revision. We have a discussion section in Section 4. We would like to emphasize that the lack of a conclusion section does not affect the completeness and soundness of our method and results.\n\n**Conclusion and future work**: Motivated by the observation that angular information among neurons crucially determines visual semantics, we propose a simple yet effective finetuning method – orthogonal finetuning for controlling text-to-image diffusion models. Specifically, we target two text-to-image applications: subject-driven generation and controllable generation. Compared to existing methods, OFT demonstrates stronger controllability and finetuning stability with fewer number of finetuning parameters. More importantly, OFT does not introduce additional inference overhead, leading to an efficient deployable model.\n\nOFT also introduces a few interesting open problems. First, OFT guarantees the orthogonality via Cayley parametrization which involves a matrix inverse. It slightly limits the scalability of OFT. Although we address this limitation using block diagonal parametrization, how to speed up this matrix inverse in a differentiable way remains a challenge. Second, OFT has unique potential in compositionality, in the sense that the orthogonal matrices produced by multiple OFT finetuning tasks can be multiplied together and remains an orthogonal matrix. Whether this set of orthogonal matrices preserve the knowledge of all the downstream tasks remains an interesting direction to study.\n\n---\n\n**Q2: In comparison with Dreambooth, the advantages are relatively small.**\n\nA2: Thanks for the comment. Our method serves as a finetuning strategy which uses the same object function as DreamBooth, but unlike DreamBooth that finetunes the entire network, OFT only finetunes a small fraction of the parameters. With a small number of finetuning iterations, OFT performs slightly better than DreamBooth. However, the finetuning stability of OFT is much stronger than DreamBooth, since OFT performs significantly better than DreamBooth with a large number of finetuning iterations. \n\nBesides the quantitative metrics shown in the paper, we also conduct a human study to evaluate the quality in subject-driven generation. We compare DreamBooth, LoRA and OFT in two settings: (1) iteration 1000 and (2) iteration 10000. The settings and results are given below. We note that the insensitivity to the iteration number is actually important, since determining a good number of finetuning iterations is nontrivial and case-specific in subject-driven generation.\n\n**Human evaluation settings**: we randomly select 7 subjects. For each subject, we randomly select 4 text prompts. Then we have 28 subject-driven generation tasks. For each task, we have one random sample from each method (DreamBooth, LoRA, OFT), and then ask each participant three multiple-choice questions: (1) which image has the subject's identity been preserved the best? I.e., whether the subject in the generated image resembles the subject in the sample images; [Subject Fidelity] (2) which image matches the text description the most? [Text Alignment] (3) which image has the best overall quality? [Overall Image Quality]. We have 50 participants in total. We evaluate the methods at two different stages of fine-tuning: at iteration 1000 (where the evaluated methods achieve the overall best performance) and at iteration 10000 (where we evaluate the finetuning stability by continuing the finetuning process). The number below is the percentage of the participants that select the corresponding image.\n\n|| | Iteration 1000| | | Iteration 10000| |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n|**Metric**|**DreamBooth**|**LoRA**|**OFT**|**DreamBooth**|**LoRA**|**OFT**|\n|**Subject Fidelity**|42.0%|15.4%|**42.6%**|22.4%|1.4%|**76.2%**|\n|**Text Alignment**|18.6%|24.7%|**56.7%**|2.6%|1.4%|**96.0%**|\n|**Overall Image Quality**|35.7%|19.2%|**45.1%**|11.6%|0.8%|**87.6%**|\n\nWe will add the full experimental details and results in the revision.\n\n---\n\n**Q3: Is this method generalizable to other models, say large language models?**\n\nA3: Great insight, and indeed as the reviewer mentioned, our method can be naturally used to finetune large language models. Based on the reviewer’s suggestion, we have conducted some preliminary experiments based on RoBERTa. The results are given below. However, this is beyond the scope of this paper, and we defer the complete study of finetuning large language models to future investigation.\n\n**NLP experimental settings**: We test OFT on the widely used natural language understanding benchmark (GLUE) by finetuning the pretrained large language model (RoBERTa). The results for LoRA are obtained by running their official implementation. The settings (e.g. the backbone architecture, batch size) for LoRA and OFT are exactly the same for fair comparison. Higher numbers indicate better performance below.\n\n|Task|RoBERTa w/ LoRA|RoBERTa w/ OFT|\n|---|:---:|:---:|\n|MNLI|**87.1**|86.0|\n|SST-2|93.8|**94.3**|\n|MRPC|89.0|**91.2**|\n|CoLA|60.3|**64.5**|\n|QNLI|**92.9**|**92.9**|\n|QQP|90.8|**90.9**|\n|RTE|74.4|**77.3**|\n|STS-B|90.6|**90.7**|\n|Average|84.9|**86.0**|\n"}, {"review_id": "tbPpyaOCo5", "reviewer": "Reviewer_i8ZS", "summary": "The paper introduces a principled finetuning method called Orthogonal Finetuning (OFT) for adapting text-to-image diffusion models to different downstream tasks. The key idea of the proposed approach is to preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. The paper claims that this property is crucial for preserving the semantic generation ability of T2I models. The paper additionally proposes  Constrained Orthogonal Finetuning (COFT) to improve finetuning stability. Experiments on subject-driven generation and controllable generation demonstrate the utility of the proposed approach.\n", "questions": "Please see weaknesses.", "limitations": "Authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The key idea of the paper to learn a layer-shared orthogonal transformation for neurons such that their pairwise angles are preserved is interesting and well-presented. The proposed finetuning approach could be potentially useful/applied to other domains/tasks. The experiment results on subject-driven generation and controllable generation are solid with sufficient comparison to the baselines.", "weaknesses": "I do not find any major weaknesses with this work except for few clarification questions: (a) On OFT in section 3.2 and COFT in section 3.4: Can they help learning weight matrices that are identity matrices? From equation (4), it seems COFT can learn this. Did you perform any analysis on the weights to see if the weight matrix can be identity? Would be great if you provide some details in the rebuttal. (b) In section 5.1, it is not clear if the paper applies OFT and COFT to the CLIP Text encoder as well? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the positive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n---\n\n**Q1: On OFT in section 3.2 and COFT in section 3.4: Can they help learning weight matrices that are identity matrices? From equation (4), it seems COFT can learn this. Did you perform any analysis on the weights to see if the weight matrix can be identity? Would be great if you provide some details in the rebuttal.**\n\nA1: Thanks for the question. If the reviewer is referring to learning a transformation matrix that is an identity matrix, then both OFT and COFT can learn this. In fact, both OFT and COFT initialize the transformation matrix $R$ as an identity matrix such that the finetuning will begin exactly from the pretrained model (similar to the role of zero initialization in existing finetuning methods). The reviewer has the right insight in the sense that, compared to COFT, OFT typically deviates more from the identity matrix due to the lack of identity constraint. After the OFT/COFT finetuning, the transformation matrix $R$ is usually not an identity matrix, otherwise the finetuned model will be exactly the same as the pretrained model. For the weight matrices in the pretrained neural networks, they are usually rectangular matrices and are not identity matrices in practice (especially when they are pretrained).\n\nAs an additional benefit of OFT/COFT, we can effortlessly remove the transformation matrix to recover the original pretrained model, while the full parameter finetuning can not easily recover the pretrained model. We also have conducted an empirical analysis to see whether the orthogonality of the transformation matrix is preserved. The reviewer can refer to Figure 1 in the main paper for this result.\n\nIf the reviewer still has questions regarding this issue, we are more than happy to provide more details on any follow-up questions.\n\n---\n\n**Q2: In section 5.1, it is not clear if the paper applies OFT and COFT to the CLIP Text encoder as well?**\n\nA2: Thanks for the question. In our experiments, we don’t apply OFT and COFT to the CLIP text encoder. We generally follow the common practice in LoRA which only finetunes the cross-attention layers. Our method finetunes exactly the same layers as LoRA in order to ensure a fair comparison. We will add more related experimental details and also release our source code to ensure full reproducibility.\n"}, {"review_id": "yzrLUgohik", "reviewer": "Reviewer_YBiq", "summary": "The paper proposes a method for efficiently fine-tuning pre-trained text-to-image generation models, good results are obtained in experiments.", "questions": "1. What is the $\\theta$ in $z=cos(\\theta)$ in Figure 2 of section 3.1?\n\n2. Can the authors provide reference for the statement finetuning essentially aims to train the model while implicitly minimizing $\\Vert M - M^0\\Vert $?\n\n3. Can the authors provide fine-tuning time in seconds/minutes of different methods, and comparison of sampling time with same number of steps at testing?\n\n4. In Figure 5, there are many details lost in the given examples. How will the results be if using a much larger $\\epsilon$? For this example, can the author add results of OFT and DreamBooth (which is provided in their paper Appendix) into Figure 5?", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes OFT and COFT, which can efficiently fine-tune the pre-trained text-to-image generation model as less parameters needed to be fine-tuned compared to baseline methods. \n\nGiven the fact that text-to-image generation models normally requires massive training samples and computation resources to train, an efficient fine-tuning method could benefit many researchers.\n\nAccording to the experiments, the proposed methods lead to better performance on different tasks.", "weaknesses": "Currently, the experiments are not convincing enough. For the proposed methods, one of the most important downstream applications is adapting pre-trained models to a target domain/dataset. Different from customizing with few samples, fine-tuning on downstream dataset can be easily evaluated by some commonly used metrics like FID, KID. Comparing supervised FID on a given dataset, for instance MS-COCO, of proposed methods, LoRA, full parameter fine-tuning is an important experiment.\n\nCustomize a pre-trained model with few images is an important downstream task, while it is actually very hard to evaluate the performance of different methods. Evaluating with pre-trained models (e.g. CLIP) may not be convincing enough because the novel concepts contained in new images could be outliers. It is actually hard to judge whether identity of target concept is well preserved. Thus human evaluation is suggested to be added. What's more, the method should be able to generate images of different styles (pencil drawing, oil painting), which are shown in related papers, these results are also suggested to be added in the paper.\n\n\nSome experimental details are not clear enough (please see some questions below).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the positive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n---\n\n**Q1: Comparing supervised FID on a given dataset, for instance MS-COCO, of proposed methods, LoRA, full parameter fine-tuning is an important experiment.**\n\nA1: Great suggestion! Our paper targets controlling text-to-image diffusion, but we do agree with the reviewer that finetuning to a target domain is useful. Following the reviewer’s suggestion, we finetune the pretrained Stable Diffusion with the text-to-image diffusion loss on the MS-COCO training set. For evaluation, we use the text prompt from the MS-COCO validation set to generate images, and then compare it to the original MS-COCO validation set to calculate the FID score. The results are summarized below:\n\n||SD (no finetuning)|SD (full param)|SD (LoRA)|SD (OFT)|\n|---|:---:|:---:|:---:|:---:|\n|**FID**|29.0|25.9|25.0|**24.6**|\n\nFrom the results, we observe that OFT outperforms both full parameter finetuning and LoRA. For LoRA and OFT, we use the best possible hyperparameters. We will add this experiment to the revision.\n\n---\n\n**Q2: Human evaluation is suggested to be added. The method should be able to generate images of different styles, these results are also suggested to be added.**\n\nA2: Great suggestion! We agree with the reviewer that human evaluation is useful. We conduct the suggested user study.\n\n**Human evaluation settings**: we randomly select 7 subjects. For each subject, we have 4 text prompts. Then we have 28 subject-driven generation tasks. For each task, we have one random sample from each method (DreamBooth, LoRA, OFT), and then ask each participant 3 multiple-choice questions: (1) which image has the subject's identity been preserved the best? I.e., whether the subject in the generated image resembles the subject in the sample images; [Subject Fidelity] (2) which image matches the text description the most? [Text Alignment] (3) which image has the best overall quality? [Overall Image Quality]. We have 50 participants in total. We evaluate the methods at two different stages of fine-tuning: iteration 1000 (where the evaluated methods achieve the overall best performance) and iteration 10000 (where we evaluate the finetuning stability by continuing the finetuning process). The number below is the percentage of the participants that select the corresponding image.\n\n|| | Iteration 1000| | | Iteration 10000| |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n|**Metric**|**DreamBooth**|**LoRA**|**OFT**|**DreamBooth**|**LoRA**|**OFT**|\n|**Subject Fidelity**|42.0%|15.4%|**42.6%**|22.4%|1.4%|**76.2%**|\n|**Text Alignment**|18.6%|24.7%|**56.7%**|2.6%|1.4%|**96.0%**|\n|**Overall Image Quality**|35.7%|19.2%|**45.1%**|11.6%|0.8%|**87.6%**|\n\n**Finetuning to different styles**: In addition, we evaluate OFT by finetuning the pretrained Stable Diffusion on other target datasets, to generate images of different styles. We consider two scenarios, including generating sketches (by finetuning on the **sketch-scene** dataset, 10k caption-sketch pairs) and art images (by finetuning on the **wikiart_captions** dataset, 81.4k caption-image pairs). For testing, we run inference on the finetuned model, using the captions from the MS-COCO validation set. The qualitative results can be found in the attached **Rebuttal PDF** (Figure 2&3).\n\n---\n\n**Q3: what is $\\theta$ in Figure 2 of section 3.1**\n\nA3: $\\theta$ is the angle between the neuron weight $w$ and the feature $x$. The experiment in Figure 2 is to replace the inner product in the convolution layer with a cosine similarity and to show that angles alone can perfectly reconstruct the original image without retraining. We will add more details in the revision for better clarity.\n\n---\n\n**Q4: Why finetuning aims to train the model while implicitly minimizing $||M-M^0||$**\n\nA4: The claim comes from a simple derivation. We have one-step gradient descent as $M^1=M^0-\\eta\\frac{\\partial \\ell}{\\partial M}$, where $\\eta$ is the learning rate. Then we have that\n$||M^1-M^0||=||M^0-\\eta\\frac{\\partial \\ell}{\\partial M}-M^0||=\\eta||\\frac{\\partial \\ell}{\\partial M}||$. Therefore, a smaller learning rate $\\eta$ implicitly enforces a smaller difference between $M$ and $M^0$.\n\n---\n\n**Q5: comparison of fine-tuning time and sampling time at testing**\n\nA5: Great suggestion! The finetuning runtime of OFT is generally comparable to LoRA. The inference runtime of OFT is the same as LoRA and the pretrained model. We note that, despite the similar finetuning runtime, OFT achieves much faster convergence than LoRA (shown in Figure 8 of the main paper), so the effective time-wise convergence of OFT is much better than LoRA. \n\nOFT does not introduce any computational overhead at inference, making the sampling time at testing exactly the same as the pretrained model (with the same # of steps).\n\nAs an example, we compare OFT and LoRA in controllable generation. The runtime comparison is given below:\n\n|Method|C2I|S2I|L2F|Sampling Time|\n|---|:---:|:---:|:---:|:---:|\n|**LoRA**|2010 min|560 min|680 min|18s|\n|**OFT**|2230 min|610 min|730 min|18s|\n\nWe will add the full results and detailed settings to the revision.\n\n---\n\n**Q6: In Figure 5, How will the results be if using a much larger $\\epsilon$? For this example, can the author add results of OFT and DreamBooth into Figure 5?**\n\nA6: Thanks for the suggestion. Larger $\\epsilon$ in COFT will increasingly resemble OFT and eventually reduce to OFT. We note that Figure 5 aims to show how different $\\epsilon$ affects the flexibility of COFT, rather than comparing performance of different methods. Therefore, in this simple example, DreamBooth and COFT/OFT perform similarly.  We provide the updated result in the figure 1 of the **Rebuttal PDF**. As suggested, we will also update Figure 5 and add the details in the revised paper."}, {"review_id": "fRPPGZdyJB", "reviewer": "Reviewer_q6NS", "summary": "This work proposes Orthogonal Finetuning (OFT), a new finetuning method for t2i diffusion models, centered on maintaining the hyperspherical energy from the pretrained model during finetuning to preserve semantic structures. It also introduces Constrained Orthogonal Finetuning (COFT), a variant that limits angular deviation from the pretrained model, thus preserving generative performance from pretraining.\nOFT is different from LoRA, since it does not assume a low-rank structure for weight updates and instead udpates weights using layer-shared orthogonal transformations.\nOFT shows good results for subject-driven and controllable generation tasks.", "questions": "N/A", "limitations": "Societal impact seems to be missing, although is an extension of other personalization methods such as Dreambooth etc.\nFailure cases would be important for this paper - specifically, when does OFT do less well than LoRA or DB?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. From my experience, neuron angle is indeed very important and it's interesting that this paper brings this to light with rigorous arguments and some nice toy experiments.\n2. The idea of OFT is solid and well-motivated. But there are questions as to the effectiveness compared to other less-well motivated alternatives such as LoRA.\n3. I do think the exploration of this idea is worthwhile and this is the reason that I am still voting positively, even given my reservations on evaluation.\n4. Supp. material is solid with many more examples and some good nuggets of insight (e.g. tuning conv layers is not very impactful).", "weaknesses": "1. DB does not add additional inference overhead (L. 293)\n2. Although the idea makes sense, and is an interesting way to see the problem, it is hard to know whether the empirical improvements are enough. Specifically, it seems like some metrics are better (subject fidelity, quality) and some are worse (prompt fidelity, by very little, could be negligible). But there is no user study, which could put these results in perspective. These metrics are not nearly as strong as a user study and a user study could show the real benefits of the method. This is my main reservation with regards to this work.\n3. My second reservation is that, for low ranks, it would be great to have comparisons with LoRA (w/ equal number of parameters, since OFT is more parameter efficient at equal rank)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the positive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n---\n\n**Q1: DB does not add additional inference overhead (L. 293)**\n\nA1: Good catch and sorry for the mistake. DreamBooth indeed does not introduce inference overhead, but it finetunes much more parameters than OFT. We will correct this mistake in the revision.\n\n---\n\n**Q2: Although the idea makes sense, and is an interesting way to see the problem, it is hard to know whether the empirical improvements are enough. Specifically, it seems like some metrics are better (subject fidelity, quality) and some are worse (prompt fidelity, by very little, could be negligible). But there is no user study, which could put these results in perspective. These metrics are not nearly as strong as a user study and a user study could show the real benefits of the method. This is my main reservation with regards to this work.**\n\nA2: Great suggestion! We follow the reviewer’s suggestion and perform user study to evaluate the generated results in subject-driven generation.\n\n**Human evaluation settings**: we randomly select 7 subjects. For each subject, we randomly select 4 text prompts. Then we have 28 subject-driven generation tasks. For each task, we have one random sample from each method (DreamBooth, LoRA, OFT), and then ask each participant three questions: (1) which image has the subject's identity been preserved the best? I.e., the subject resembles the subject in the sample images; [Subject Fidelity] (2) which image matches the text description the most? [Text Alignment] (3) which image has the best overall quality? [Overall Image Quality]. We have 50 participants in total. We evaluate the methods at two different stages of fine-tuning: at iteration 1000 (where the evaluated methods achieve the overall best performance) and at iteration 10000 (where we evaluate the finetuning stability by continuing the finetuning process). The number below is the percentage of the participants that select the corresponding image.\n\nThe results are given as follows:\n\n|| | Iteration 1000| | | Iteration 10000| |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n|**Metric**|**DreamBooth**|**LoRA**|**OFT**|**DreamBooth**|**LoRA**|**OFT**|\n|**Subject Fidelity**|42.0%|15.4%|**42.6%**|22.4%|1.4%|**76.2%**|\n|**Text Alignment**|18.6%|24.7%|**56.7%**|2.6%|1.4%|**96.0%**|\n|**Overall Image Quality**|35.7%|19.2%|**45.1%**|11.6%|0.8%|**87.6%**|\n\n\nWe will add all the experimental details and the complete results to the revised paper.\n\n---\n\n**Q3: My second reservation is that, for low ranks, it would be great to have comparisons with LoRA (w/ equal number of parameters, since OFT is more parameter efficient at equal rank)**\n\nA3: Great suggestion! The current experimental setting is to select the best possible hyperparameter (e.g., rank in LoRA, number of blocks in OFT) for all the finetuning methods. For the published methods, we generally start with the official repository and tune the hyperparameter via cross-validation. For example, we find that setting rank as 8 in LoRA yields the best performance, and keeping increasing the rank will actually hurt the performance. Therefore, our OFT already compares with the best setting of LoRA.\n\nIn order to fully address the reviewer’s concern, we vary the number of finetuning parameters in LoRA (i.e. varying the rank), and compare their performance to OFT in the segmentation-to-image controllable generation task. The results are as follows \n\n|Method|LoRA (r=32)|LoRA (r=64)|OFT (block=4)|\n|---|:---:|:---:|:---:|\n|**# of params**|13.1 M|23.1 M|16.3 M|\n|**mIoU**|23.12|23.05|**27.06**|\n|**mAcc**|35.19|35.34|**40.09**|\n|**aAcc**|57.64|57.63|**62.42**|\n\n---\n\n**Q4: Societal impact seems to be missing, although is an extension of other personalization methods such as Dreambooth etc. Failure cases would be important for this paper - specifically, when does OFT do less well than LoRA or DB?**\n\nA4: Great suggestion! We have provided some failure cases in Appendix H. In general, we hardly observe cases where DreamBooth can succeed but OFT fails, but as our qualitative results show, there are many cases where DreamBooth fails but OFT works well. The failure cases observed in our experiments usually happen in both DreamBooth and OFT. One common failure case of OFT is the incorrect text grounding in images. For example, we enter a text prompt “a red cat on a black carpet”, OFT will sometimes output images like “a black cat on a red carpet”. However, this also happens in DreamBooth. This is likely due to the grounding problems in CLIP, rather than the problems in finetuning.\n\nTherefore, we definitely agree with the reviewer that failure cases are important, and we will add more failure cases, in-depth discussions and societal impact to the revision.\n"}, {"review_id": "6nehzaXqpg", "reviewer": "Reviewer_jibc", "summary": "This paper proposes a parameter-efficient fine-tuning algorithm that preserves the so-called \"hyperspherical energy\" to achieve a great trade-off between prevervation and generalization and high training efficiency. The authors conduct experiments on subject-driven generation and controllable generation, which both can improve the image generation quality as compared with other SOTA methods.", "questions": "1. In Sec. 3.1, you say \"orthogonal transformation is a sweet spot between flexibility and regularity.\" How can you draw the conclusion? lack some theoretical or experimental proof.\n2. Figure 3 is the \"controllable generation experiment using the setting of ControlNet\". The results of controlnet without orthogonal finetuning (the original controlnet training ?) seem to collapse totally. Is it reasonable?\n3. In line 197, I don't understand why you define HE(W) as this equation. Any reference?\n4. In line 218, how to ensure that Q is a skew-symmetric matrix during code implementation? There is no explanation here or in the appendix.", "limitations": "This paper seems no conclusion and limitation section. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents the orthogonal finetuning idea clearly. It provides two toy examples to help readers understand the motivation of this paper.\n2. The core idea of orthogonal finetuning is straightforward and easy to understand. The authors also cite some theoretical papers about \"hyperspherical energy\" to support their motivation and algorithmic foundation.\n3. This paper provides another perspective for the field of PEFT other than LoRA, which may excite more interesting works and discussions on this important topic.\n4. The authors conduct two experiments: subject-driven generation and controllable generation based on Dreambooth and Controlnet, both of which demonstrate their effectiveness.", "weaknesses": "1. The \"subject-driven generation\" introduced in related work is inadequate. Most works you mentioned are about image-editing and some recent dreambooth-like subject-driven finetuing works are not included, such as P+, DisenBooth, Cones \n  \nVoynov, Andrey, et al. \"$ P+ $: Extended Textual Conditioning in Text-to-Image Generation.\" arXiv preprint arXiv:2303.09522 (2023).\n  \nLiu, Zhiheng, et al. \"Cones: Concept neurons in diffusion models for customized generation.\" arXiv preprint arXiv:2303.05125 (2023).\n  \nChen, Hong, et al. \"DisenBooth: Disentangled Parameter-Efficient Tuning for Subject-Driven Text-to-Image Generation.\" arXiv preprint arXiv:2305.03374 (2023).\n\nAlthough this work may be close to your submission time, you'd better add some discussion or comparisons with this kind of work.\n\n2. In Table 1 and Table 2, the statistical difference between COFT and OFT is very marginal; how can you draw the conclusion that COFT is better than OFT?\n\n3. In the experiment section, no specific-domain finetuning comparisons between OFT and LoRA  are provided. Since LoRA is widely used in many downstream specific-domain finetuning, I'd like to see some further comparisons and discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for the positive comments on our work. We take every comment seriously and hope our response can address the reviewer’s concerns. If there are any remaining questions, we are more than happy to address them.\n\n---\n\n**Q1: The related work of subject-driven generation**\n\nA1: We appreciate the reviewer for these highly related papers. We will cite and discuss them in the revision. In fact, some of these papers are parallel contributions compared to ours. For example, DisenBooth uses LoRA to finetune the neural network, and it can naturally be replaced by OFT.\n\n---\n\n**Q2: Difference between COFT and OFT**\n\nA2: The motivation of COFT is to constrain the deviation from the pretrained model in order to improve finetuning stability. In terms of the best possible performance, OFT and COFT may be similar, but COFT generally yields better finetuning stability and is less sensitive to the number of finetuning iterations. Determining a good finetuning iteration is actually highly nontrivial yet important. We have empirically verified that COFT can finetune the model with a very large number of iterations (e.g. 100,000 iterations) while still outputting very plausible results. In contrast, OFT is usually unable to do so. Please refer to Appendix E in our paper for more results.\n\nThe constrained version of OFT is in fact another advantage over LoRA, because it is nontrivial in LoRA to add a constraint such that the finetuned weight will not deviate too much from the pretrain weight. In OFT, setting a good $\\epsilon$ for the difference between the orthogonal matrix and the identity matrix is simple, while in LoRA, it is nontrivial to set a good $\\epsilon$ between the pretrained weight matrix and the finetuned weight matrix.\n\n---\n\n**Q3: no specific-domain finetuning comparisons between OFT and LoRA**\n\nA3: Great suggestion! We follow the reviewer’s suggestion and add a downstream finetuning experiment. For quantitative comparison, we finetune the original Stable Diffusion with the text-to-image diffusion loss on the MS-COCO training set. For evaluation, we use the text prompt from the MS-COCO validation set to generate images, and then compare it to the original MS-COCO validation set to calculate the FID score. The results are summarized in the following table:\n\n|Metric|SD (no finetuning)|SD (full param)|SD (LoRA)|SD (OFT)|\n|---|:---:|:---:|:---:|:---:|\n|**FID**|29.0|25.9|25.0|**24.6**|\n\nFrom the FID results, we can observe that OFT outperforms both full parameter finetuning and LoRA.\n\nFor qualitative comparison,we additionally evaluate our proposed method by finetuning the pre-trained Stable Diffusion model on other datasets, to generate images of different styles. We evaluated two different scenarios, including generating sketches (by finetuning on the **sketch-scene** dataset, which includes 10k caption-sketch pairs) and art images (by finetuning on the **wikiart_captions** datasheet, 81.4k caption-image pairs). For testing, we simply run inference on the finetuned model, using the captions from the MS-COCO validation set. The qualitative results can be found in Figure 2&3 in the **Rebuttal PDF**.\n\nWe will add these experiments and results to the revision.\n\n---\n\n**Q4: Orthogonal transformation is a sweet spot between flexibility and regularity. How can you draw the conclusion?**\n\nA4: Sorry for the confusion. Our conclusion that orthogonal transformation is a good tradeoff between flexibility and regularity mostly comes from the empirical evidence in Figure 3 of the main paper, where unconstrained linear transformation completely fails to finetune the model. While identity matrices do not have any flexibility of finetuning the model, we observe that orthogonal matrices have sufficient regularity to achieve flexible yet stable finetuning. Another motivation that drives us to use orthogonal transformation is its perfect preservation of hyperspherical energy, as shown in Figure 1 of the main paper. We will add more discussion and justification in the revision.\n\n---\n\n**Q5: In Figure 3, the results of controlnet without orthogonal finetuning (the original controlnet training?) seem to collapse totally. Is it reasonable?**\n\nA5: Sorry for the confusion. This experiment is not conducted with the ControlNet pipeline (which uses an additional branch to add control signals). Instead, we perform this controllable generation task with the finetuning pipeline (what LoRA and OFT use). This experiment aims to show that, without constraining the transformation matrix $R$ (in Eq.2) to be orthogonal, the multiplicative finetuning will fail completely. In contrast, enforcing $R$ to be orthogonal achieves good finetuning performance. The results show the importance of orthogonality in OFT. We apologize for missing experimental details, and we will provide all the details as well as the source code for reproducing this experiment in the revised paper.\n\n---\n\n**Q6: In line 197, why you define HE(W) as this equation**\n\nA6: Thanks for the comment. We define this energy for the sake of notation simplicity. This definition also follows from the paper [Learning towards minimum hyperspherical energy, NeurIPS 2018].\n\n---\n\n**Q7: In line 218, how to ensure that Q is a skew-symmetric matrix in implementation?**\n\nA7: Thanks for the comment. Since the skew-symmetric matrix $A$ is defined to satisfy $A^\\top=-A$, where the diagonal is zero. Therefore, we only need to initialize the upper triangular part of $A$ and make them trainable. The lower triangular part of $A$ will simply be copied as the negative version of the upper triangular part. Our source code will be made publicly available. \n\n---\n\n**Q8: no conclusion & limitation section.**\n\nA8: We are very sorry for not including a conclusion section due to the page limit. We will add a conclusion and limitation section in the revision. For the intended conclusion section, please refer to Q1 of Reviewer styH."}]

Author rebuttal:
Dear Reviewers and ACs,

We are deeply appreciative of all the reviewers and ACs for spending time on our submission. **Rebuttal PDF** contains the figures that are referred in the response to reviewers.

Best,
Authors

Meta-review: All the reviewers recommend the acceptance of the work. Reviewers appreciated the effective fine-tuning technique for T2I models and the good results. Reviewers raised several clarification questions which authors addressed well in their responses. The reviewers did raise some valuable concerns and suggestions that should be addressed in the final camera-ready version of the paper, which include adding the relevant rebuttal discussions and revisions in the main paper. The authors are encouraged to make the necessary changes to the best of their ability.
Decision: Accept (poster)