Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models
Authors: Wonguk Cho
Seokeon Choi
Debasmit Das
Matthias Reisser
Taesup Kim
Sungrack Yun
Fatih Porikli
Venue: NeurIPS 2024 poster (2024)
Keywords: Hollowed Net
On-Device Learning
Memory-Efficient Fine-Tuning
LoRA Personalization
Text-to-Image Diffusion Models
Primary area: diffusion_based_models

Abstract: Recent advancements in text-to-image diffusion models have enabled the personalization of these models to generate custom images from textual prompts. This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net to temporarily remove a fraction of its deep layers, creating a hollowed structure. This approach directly addresses on-device memory constraints and substantially reduces GPU memory requirements for training, in contrast to previous methods that primarily focus on minimizing training steps and reducing the number of parameters to update. Additionally, the personalized Hollowed Net can be transferred back into the original U-Net, enabling inference without additional memory overhead. Quantitative and qualitative analyses demonstrate that our approach not only reduces training memory to levels as low as those required for inference but also maintains or improves personalization performance compared to existing methods.

Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models
Wonguk Cho
Seokeon Choi
Debasmit Das
Matthias Reisser'
Taesup Kim²
Sungrack Yun Fatih Porikli'
Qualcomm Al Research
Scoul National University
(wongcho, seokchoi, debadas, mreisser, sungrack, fporikli)@qti qww..uuu.acooomcmommmm com {wwnguukcho, taesup. kim)@smu.ac..K
Abstract
Recent advancements in text-to-image diffusion models have enabled the person- alization of these models to generate custom images from textual prompts. This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on conssrainnd devices. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net to temporarily remove fraction of its deep layers, creating a hollowed structure. This approach directly addresses on-device memory constraints and substantially reduces GPU memory requirements for training. in contrast to previous methods that primarily focus on minimizing training steps and reducing the number of parameters to update. Additionally, the personalized Hollowed Net can be transferred back into the original U-Net, enabling inference without addi- tional memory overhead. Quantitative and qualitative analyses demonstrate that approach not only reduces training memory to levels as low as those required for inference but also maintains improves personalization performance compared existing methods.
Introduction
Recent research text-to-image (T2I) diffusion models 1 2 which generate high-resolution images from text prompts, has increasingly focused on personalizing and customizing these generative models effectively 3 4 [ 7 A primary approach, termed subject-driven generation [5], involves fine- tuning pre-trained diffusion models with a few user-specific images to generate varied representations of subject using simple text prompts. This allows users to create personalized images of specific subjects, such as family, friends, pets, or personal items, with preferred appearances, backgrounds, and styles. Such capabilities enable creative applications including art renditions, property modifications, and accessorization.
From a practical standpoint, implementing subject-driven generation on-device offers significant benefits in efficiency and privacy. By operating independently of congested cloud servers networks, users can generate personalized images anywhere at no additional cost and do not need to compromise their privacy as data and personal information remain on the device.
WWrk done during an internship at Qualcomm Al Research. 1 Qualcomm Al Research initiative of Qualcomm Technologies, Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite extensive research aimed at efficiently personalizing diffusion models, limited attention has been paid memory I/O, a critical bottleneck in on-device learning. Recent studies have mainly explored two strategies: (1) decreasing the number of training steps and (2) reducing the number of updating parameters. The first methods [8] 9 trained models to generate a set of personalized Low-Rank Adaptation (LoRA) parameters [3]. text embeddings, or image prompts from a user-specific image. This strategy provides a better initial setup for personalizing the diffusion models, effectively reducing required training steps. Some models TO IT 12] even support zero-shot personalization, although they underline that further fine-tuning can enhance personalization quality and address failure cases. Nonetheless, these methods are not viable for environments with severely limited computational resources, as they necessitate additional inference using large pre-trained models (e.g.. 2.7B parameters for BLIP-2 in BLIP- Diffusion 12] and 2.5B for apprentice models in Su'TI [T], which are substantially larger than standard diffusion models (e.g., IB for Stable Diffusion v2 2 )) making their application challenging in on-device settings.
[2] utilize additional large pre-
The second approach 714, often involving LoRA, aims to reduce the number of updating parameters by limiting updates to specific layers or decomposing weight matrices. However, even with fewer parameters to update, these parameters reside within large pre-trained models, and thus the backward pass through the large models is required to compute gradients. Given limited computational resources, where even simple inference tasks with diffusion models can strain GPU memory, performing backpropagation while keeping the entire diffusion model in GPU memory remains a significant limitation.
which fine-tunes a smaller auxiliary network rather than directly updating the parameters of large pre-trained network. This method significantly reduces the heavy memory costs associated with computing backpropagation on the larger network. Particularly for Natural Language Processing (NLP) tasks, Ladder Side Tuning (LST) 18 has proven effective, reducing the memory costs required for fine- tuning large language models (LLMs) by 69 percent. However, applying LST directly to diffusion U-Nets presents significant challenges. Unlike transformer layers in LLMs, which maintain consistent input and output dimensions, diffusion U-Nets have varying spatial dimensions and channels, as well as skip-connections across different blocks. Additionally, the requirements for structural pruning and weight initialization to build side-tuning networks further complicate the rapid adaptability of L.ST to personalization tasks across different subjects and domains.
promising approach to address these challenges side-tuning 15 16
To this end, we introduce a novel personalization technique called Hollowed Net, which is illustrated in [ig.[] [ Based on our observation that deep layers in the middle of diffusion U-Nets play significantly less important roles than the rest of the layers, we propose to fine-tune LoRA parameters for the personalization using Hollowed Net, a layer-pruned U-Net featuring a central hollow, which is constructed by temporarily removing the middle deep layers from the pre- pre-trained diffusion U-Net. By utilizing the symmetrical "U-shape" architecture of the diffusion U-Net, we avoid complicated processes of applying structural pruning and weight initialization to build a side network, and neither additional models nor extensive pre-training with large datasets are required.
By fine-tuning LoRA parameters using Hollowed Net, we can significantly reduce the memory needed for storing model weights in GPU. Once the LoRA parameters are fine-tuned with Hollowed Net, they can be seamlessly transferred back to the original Diffusion U-Net for inference, without requiring any additional memory beyond the small set of transferred parameters, Our experiments demonstrate that Hollowed Net enables achieving performance that is comparable to or better than the direct fine-tuning with LoRA, while using 26 percent less GPU memory, which is only 11 percent increased GPU memory relative to inference.
To the best of our knowledge, Hollowed Net is the first technique that addresses subject-driven generation in terms of memory efficiency. Our method shows how T2I diffusion models can be fine-tuned under extremely limited computational resources with as low GPU memory as required for inference. Furthermore, it is important to note that our method does not preclude the use of previously described strategies for efficient personalization. Both enhanced parameter-efficient strategies and improved initializations with additional pre-trained models can be integrated with our approach to further increase efficiency according to given resource constraints.
Our contributions can be summarized as follows:
2

---

Stage 1: Pre-computing
"A dog
Frozen
Unloaded
8/
85
Trainable
LofAA
Forward
Backward
save
Data Storage
Stage 2: Fine-tuning
load
Hollowed
0:
85
8
"A [V] dog
Figure 1: The LoRA personalization with Hollowed Net for resource--constrained environments. The input image from the DrcamBooth dataset
We introduce Hollowed Net, a novel personalization technique for T2I diffusion models under limited computational resources. Our method significantly reduces the memory demands on GPU to levels as low as those required for inference, while maintaining high-fidelity personalization capacity. This demonstrates its potential as a feasible on-device learning solution for resuurce-oonttrained devices.
Our method provides scalable and controllable solution for on-device learning. As this method does not require any additional models pre-training with large datasets, it is easily scalable to other architectures such as SDXL and Transformers. Moreover, we can simply adjust the fraction of hollowed layers to control the trade-offs between performance and memory requirements, depending on the target application and resources.
Unlike previous side-tuning methods, Hollowed Net does not need to be retained for inference. The LoRA parameters fine-tuned with Hollowed Net can be seamlessly transferred back to its original network, enabling inference with no additional memory cost.
Related Works
2.1 Efficient Personalization of T2I Diffusion Models
Recent research the personalization of T2I diffusion models has introduced various methods to fine-tune the models for generating diverse images of user-specific subjects from few given images. Two foundational works in this area are Textual Inversion and DreamBooth 55. Textual Inversion 3 aims to learn new text embeddings to represent given subject, while DreamBooth 5 proposes fine-tuning entire diffusion model to align the subject with unique token.
Building on these foundational works, recent research has focused on enhancing the efficiency of this personalization process, primarily through two approaches. The first approach involves decreasing the number of training steps, mostly by utilizing an additional large pre-trained model. popular method is to use pre-trained image//ulti-monn encoder to generate personalized text embeddings

---

or image prompts from a user-specific image 12 10]. Other recent works 8 I1 pre-optimized LoRA parameters or millions of fine-tuned expert models pre-initialize for efficient fine-tuning or enable zero-shot generation with in-context learning. While these models demonstrate significant reductions in the number of training steps, the requirement for additional large pre-trained models limits their application to on-device settings. Moreover, models with zero-shot personalization capacities [1 [0] cannot be one-size-fits-ae solution for addressing different types of user-subject prompts. These models often struggle with flexibility in editing subjects or maintaining subject fidelity, and in these cases, additional fine-tuning with specific subjects needed to further enhance their personalization capacity
propose utilizing
On the other hand, another stream of work adapts parameter-efficieet fine-tuning (PEFT) approaches. These methods demonstrate significant reductions in the number of training parameters by limiting updates to a small subset of model weights in cross-attention layers 7 or further reducing the updating parameters by applying singular vector decomposition to weight matrices [14]. However, these methods are still limited in environments with extremely low computational resources, as they require backpropagation over large diffusion models and do not reduce memory usage from the model weights. Therefore, it is crucial to explore new approaches for personalizing T2I diffusion models in resource-limited settings, as we propose with our novel method, Hollowed Net. Notably, our method can be integrated with previously discussed techniques to further improve efficiency based on specific resource constraints.
2.2 Fine-Tuning with Side Networks
The idea of of side-tuning has been introduced by Zhang et al. [5], proposing the training of lightweight "side" network instead of directly fine-tuning a pre-trained network for adaptation. In terms of efficiency, Cai et al. [16] has demonstrated an additional lightweight residual module reduce memory overhead associated with the activations of the original network. Similarly, AuxAdapt 17 has shown that a small auxiliary network can be fine-tuned to adjust the main network's decisions, enabling efficient test-time adaptation for video semantic segmentation tasks.
In the context of generative models, LST [8] has demonstrated the effectiveness of side networks for different NLP tasks with LLMs by introducing small side network that takes intermediate activations of the main network as input via shortcut connections. However, directly applying LST to diffusion U-Nets poses challenges due varying spatial dimensions, channel sizes, and skip-connections across blocks, unlike the consistent dimensions in transformer layers of LLMs. Furthermore, the structural pruning and specific weight initialization required to construct side-tuning networks complicate LST's adaptability for personalized tasks across a range of subjects and domains.
2.3 Layer Pruning of Large Generative Models
Several concurrent works demonstrate that layer-pruning methods be applied to generative models, particularly for NLP tasks. Gromov et al. [19] suggest that for fine-tuning LLM models, up to 40% of deep layers can be removed, while still achieving comparable results. The authors propose that the optimal block of layers to prune can be selected based on similarity across layers. Similarly, Kim et al. 20 also propose a depth-pruning approach by evaluating block-level importance.
These approaches differ from ours due to the distinct characteristics of LLMS versus diffusion U-Nets. The aforementioned approaches involve the complete removal of deep layers for both fine-tuning and inference, considering that those layers store less critical knowledge. However, our study finds that the deep layers of diffusion U-Nets may be less involved with personalization but still contain crucial high-level image features for generating to severe performance degradation, even with additional pre-training (2I], as shown in Appendix [ This highlights the importance of our two-stage fine-tuning strategy, which excludes layers during fine-tuning reduce memory overhead while preserving the knowledge from these excluded layers throughout both training and inference stages.
images. Thus, their removal can lead
3 Preliminaries
In this section, we describe some preliminaries on T21 diffusion models. First, we discuss the basics of Stable Diffusson (SD) model and how they can be used for fine-tuning. The SD model is large

---

down blocks.0
0.0035
down blocks.0
0.0032
down_blocks.
0.0031
down_blocks.
0.0030
down blocks.2
Nown_blockss_
down_blocks.3
0.0016
3lock_bocks.3
0.0017
mid_ block
0.0011
mid mid_block
0.0012
blocks.0
0.0022
up_ blocks.0
0.0020
up_blocks.1
0.0035
up_ up_bkocks.1
0.0032
up_
blocks.2
0.0048
up_blocks.2
0.0045
blocks.3
0.0048
blocks.3
0.045
0.000 0.001 0.002
0.003
0.004
0.005
0.000 0.001 0.002
0.003
0.004
0.005
DreamBooth
(h) CustomConcept
Figure 2: Analysis of the LoRA weight change before and after personalization, per block of U-Net.
foundational T2I model, pre-trained on large amount of text-image pairs (P,x), where we have image r and associated text prompt P. The SD contains the following components: (a) Autoencoder consisting of the encoder-decoder pair (E, D), (b) Text Encoder as CLIP Er(.). and (c) Conditional Diffusion Model as U-Net co(.). The encoder E(-) processes an image into a latent space
E(x),
and the decoder is used reconstruct the input image from latent such that x = (22). The diffusion process of SD is conducted in the latent space. For randomly sampled noise ~ NNO, I) at time step I, the standard scheduler produces a noisy latent code 2, = dØz + dre, where Or and a, are coefficients controlling the noise schedule. The conditional diffusion model is trained using the following de-noising objective:
min Ep..cell - co(2,, Cp Er
After the training is carried out, the conditioned model c0(-) is used to predict the noise by using the conditional embedding (P) and time step input. To personalize diffusion models for subject-driven generation introduced by [5], the same loss is used except that the data sampled from user-specific subjects such dog, person, backpack, and etc. For the prompt, special identifier S* is used and described as "a S* person", " "a S* backpack", etc. For regularization, [5] introduces an additional class-specific prior preservation loss term, written
min E,,,,el/<pr- - ca co(2 ET (P
(2)
where fpr the ground truth noise for the data generated using the frozen pre-trained diffusion model with prompts Ppr described more generic as "a person", "a backpack", and etc.
The diffusion U-Net can be fully fine-tuned, but it is also possible to fine-tune only subset of parameters with LoRA [13] for better efficiency. In LoRA, network weight residuals AW are fine-tuned instead of the full weights W. For the fine-tunne tine-tuning of AW, it is further decomposed into low-rank matrices A and B such that AW = AB. Since A and B are low-rank matrices, the total number parameters optimize in AW is significantly smaller than W.
Methodology
In this section, we describe the details of our novel memory-efficient personalization technique, Hollowed Net, and its fine-tuning strategy. We begin by identifying less significant layers for personalization from diffusion U-Nets. Based on these observations, explain how to construct Hollowed Net from pre-trained U-Net. Next, we present our fine-tuning and inference processes for memory-efficient personalization of T2I diffusion models.
4.1 Analysis of the LoRA Weight Changes per Block of U-Net
To achieve the goal of reducing the required memory for fine-tuning a diffusion model, we first identify less significant layers in the diffusion U-Net for personalization. Similar to Li et al. (2) Kumari al. [ and Shah al. (6. we analyze the LoRA weight changes AW in the fine-tuned model for each block:
5

---

" A dog
82
83
0.
86
A [V] dog
X Personalized LoRA
Inference Path LoRA
Inference Path w/ LoRA
Figure 3: The inference process with personalized LoRA parameters transferred from Hollowed Net to the original U-Net. The input image is from the DrcamBooth dataset [5]
AW
(3)
where w and w' respectively represent the weights before and after personalization, and ni is the total number of weights in specific block. This represents the average weight change per element i. Figure [] shows the analysis of the weight changes AW before and after personalization for each block of U-Net: (a) for all subjects from the DreamBooth dataset and (b) for all subjects from the CustomConcept1001 dataset by fine-tuning Stable Diffusion v2. diffusion model [2] for 1000 steps with learning rate of le-4. The x-axis shows the changes in LoRA weights before and after personalization, while the y-axis of each plot represents the specific U-Net blocks. For each dataset, average the weight changes across subjects and provide error bars to indicate the statistical variance within each dataset.
From the figures, we observe that the average weight changes tend to be close to zero around the cen- tral blocks and become increasing for the layers farther from the mid block. This demonstrates that the blocks around the center less involved in the personalization compared to those at the begin- ning and end of the U-Net (c.g., down down_blocks.l We leverage this characteristic for designing Hollowed Net.
blocks.O. O, down blocks.1, up up_ blocks.2, 2, and up blocks.3).
4.2 Hollowed Net
Based on the aforementioned observations, we propose fine-tuning a layer-pruned U-Net, which we refer to as Hollowed Net, instead of directly fine-tuning the entire diffusion model. The core concept of Hollowed Net involves removing deep layers that are not vital for personalization from pre-trained diffusion U-Net. This strategy decreases the need to store the entire model in GPU memory, thereby reducing the memory cost associated with the model's weights.
However, unlike transformer layers in large language models, where input and output maintain the same data structure, the alterations in spatial and channel dimensions in U-Net architectures complicate the removal of its deep layers in the middle. To address this, we utilize the symmetrical "U-shape" architecture the diffusion U-Net, where each down-block layer's output is concatenated with corresponding up-block layer's input via skip-connection. This design permits us to select any up-block layer skip-connected to down-block layer and hollow out the middle layers between the pair, ensuring that the processed information from the remaining down-blocks to the remaining up-blocks without the need for additional projection layers to adjust for dimensional differences. The missing input for the upper layer, due to the removal of the middle layers, is replaced with the pre-computed output from the full diffusion U-Net, which is illustrated in the next section.
still be transferred
4.3 LoRA Personalization with Hollowed Net
To optimize GPU memory utilization, we propose two-stage fine-tuning strategy: (1) pre-computing intermediate activations the original diffusion U-Net and (2) fine-tuning the Hollowed Net using the pre-computed activations, shown in the upper and bottom half of Fig. respectively. Initially,

---

Table 1: The quantitative comparisons of fine-tunie Lne-tuning methods with three evaluation metrics. The number of parameters are the ones held in GPU memory during fine-tuning stage. The results are obtained by averaging over four runs with different seeds (standard deviation added in small-sized text).
Method
of Parameters
Training Memory
DreamBooth
CustomConsope100
Base
LoRA
Peak
Comp. Inf.
DINO
CLIP-1
CLIP-T
DINO
CLIP-1
CLIP-T
Full FT
866M
16,62GB
+376%
0.663
0.802
0.302
0.605
0.773
0.302
#0.03a
#0.007
20,002
20.000
#0.006
20.002
LoRA FT (x=128)
866M
27M
5.23GB
+50%
0.658
0.806
0.299
0.603
0.773
0.302
#0,001
#0.005
$0.002
$0,008
#0.005
$0,002
LoRA FT (F])
866M
207K
4.84GB
+39%
0.516
0,738
0.314
0.522
0.737
0.305
10.011
Hollowed Net (Ours)
527M
24M
3888B
+11%
0.660
0.805
0.300
0.603
0.773
0.302
Table 2: Human evaluation results
Table Computational loads (FLOPs)
Method
Subject Fidelity
Text Fidelity
Method
Pre-computing
Fine-tuning
Inference
Hollowed Net
31.2%
18.1%
Hollowed Net
0.238T
2.004T
0,920T
Tie
49.3%
69.4%
LoRA
2.148T
0.716T
LoRA FT
19.5%
12.5%
conduct a forward pass with pre-trained diffusion model for the specified number of pre-computing steps. During each step, given input images and sampled noise, we calculate and store intermediate activations in the data storage, which serve as inputs for the upper-block layer of the Hollowed Net.
We also store the sampled noises, time steps, and the IDs when there are multiple user images.
Once the data from the pre-trained model is pre-computed, we fine-tune the Hollowed Net by loading data from data storage, thereby avoiding the need to keep the original model in GPU memory. To further improve efficiency, we apply LoRA fine-tuning for the Hollowed Net instead of updating entire parameters. The reduced number of parameters of the Hollowed Net decreases the required GPU memory, satisfying the device's 's low memory I/O threshold and computational load during backpropagation.
Additionally, our inference process ensures that both the original diffusion model and Hollowed Net are not simultaneously maintained on GPU. Unlike side-tuning networks [75] [8] that differ in architecture and parameters from their original models, Hollowed Net maintains the same architectures and parameters as the original diffusion U-Net, except for the removed middle layers. Thus, the LoRA parameters fine-tuned on Hollowed Net can be seamlessly transferred to the corresponding layers in the original U-Net. As depicted in Fig. 3 there are two inference paths, respectively corresponds to each stage of fine-tumee fine-tuning. The first path (green line) represents the process of computing intermediate activations without using LoRA, aligning with the pre-computing stage. The second path (red line) involves using personalized LoRA parameters, which matches the application of these parameters for generating personalized images during the fine-tuning stage. By sequentially executing these paths, we enable personalized generation using the transferred LoRÁ parameters without requiring additional memory beyond the small set of LoRA parameters.
Experiments
5.1 Experimental Settings
We conduct experiments following the protocol proposed in DreamBooth (5]. We use total of 131 subjects for experiments, utilizing both the DreamBooth 5 and CustomConcept1101 [7] datasets. The DreamBooth dataset includes 30 image sets from 15 different classes, each containing 4-6 images of given subject. The subjects are divided into living subjects and objects, and 25 different prompts are assigned based on this division. Meanwhile, the CustomConcept dataset includes 101 image sets, each containing 3-15 images of a given subject. The subjects consist of 15 different large categories, with 20 unique prompts assigned to each category. For evaluation, four images with different fixed random seeds are generated per subject per prompt for both datasets.
7

---

INPUT
A [V] tespot
bronze [V] reepor
A toy
[M toy the snow
A [ dog
A dog weering red nat
A salver [V] tespot
A gold M teapot
[V] boy with the Effel Tower the background
IV] tay with mountain in the badiground
A.V)] dog wearing pink glassos
AM dog in police cutti
APUTIMAGE
INPUT IMAGE
INPUTIMAGE
painting plushia
piushie
houseplan
bam
[ Japanese painting houseplamt
[V] painting
[V] barn
Canyon
[V] winiage typewiter
Qualitative generation results of Hollowed Net with different subjects and prompts. The upper half the examples from the DreamBooth dataset (5), and the lower half the examples from the CustomConcept110) dataset 77
Figure 4:
We adopt the three evaluation metries from 5 DINO and CLIP-I for subject fidelity and CLIP-T for prompt fidelity. DINO and CLIP-I are the average pairwise cosine similarities between feature embeddings of the real and generated images, using DINO ViT-S/16 and CLIP ViT-B/32, respectively. As DINO is more sensitive to differences between subjects of the same class due to its training on instance discrimination, the DINO score considered the preferred metric for measuring subject fidelity. The CLIP-T score is the average cosine similarity between text prompt embeddings and image CLIP embeddings. We use the Stable Diffusion v2.1 diffusion model (2). Following DreamBooth [5]. we use prior preservation loss with ~ 1000 pre-generated class samples. LoRA [3] is applied for the cross and self-attention layers and fine-tuned for ~000 steps. We use AdamW optimizer with the learning rate of le-5 for full-finee full-finetunig and le-4 for the others. Assuming resource-constrane environment, we use batch size of and do not update the pre-trained text encoder, while text embeddings are pre-computed before fine-tuning.
5.2 Results
In this section, we present the results of our proposed Hollowed Net evaluate its effectiveness in terms of both memory efficiency and personalization performance. We conduct experiments with Hollowed Net, applying hollowed fraction of 39.2%. Architectural details are provided in
Appendix B Ablation studies on different fractions of hollowed layers can be found in Sec. 5.3 In the main results, the rank of Hollowed Net is fixed to 128. Experimental results on different ranks are presented in Appendix C

---

6.0
0.800
0.850
0.340
LoRA (--228)
5.0
0.700
LoRA (1-28)
0.800
LARA (e-128)
0.320
Hollowed Net
0.600
0.300
3.0
Hollowed Net
Hollowed Net
LoRA (1-128)
2.0
Hollowed Net
0.500
0.750
0.280
0.260
0.0
0.6
0.6
0.8
(a) Peak Usage (GB)
(b) DINO
(c) CLIP-I
(d) CLIP-T
Figure 5: Analysis of different fractions of hollowed layers. For all figures, the x-axis represents the fractions of layers removed from the pre-trained diffusion U-Net. The y-axis corresponds the metric used for each figure.
Quantitative Evaluation The quantitative results are displayed in Table For comparison base- lines, we implement full fine-tuning (Full FT) and LoRA fine-tuning (LoRAFT) methods with rank 128 and rank [13]. We find that while Full FT results in slightly higher performance than other methods, particularly in terms of DINO, the differences between Full FT and Hollowed Net are not significant as is within the range of standard deviations of Full FT results across different seeds. Moreover, Full FT requires more than 16GB of GPU memory which is nearly 4.7 times the memory cost of performing an inference with diffusion U-Net. Clearly, this is not feasible solution for on-device learning, where computation resources, especially memory I/O, are extremely limited.
Our Hollowed Net demonstrates its superior memory efficiency based on significant reduction in model size, requiring only 3.88GB of GPU memory usage for fine-tuning. This is only an 11% increase compared to inference. Its personalization performance is comparable or marginally better than that of LoRA fine-tuning using the same rank (r = 128). while LoRA requires a 50% increase in GPU memory compared inference. Using the lowest rank of LoRA (r 1) does not compete with Hollowed Net either, as its memory efficiency is limited by the need to run backpropagation on the entire U-Net, even though the number of fine-ture tine-tunin parameters significantly small. Additionally, the use of low number of fine-tuning parameters significantly degrades personalization capacity.
For human evaluation, we conduct user studies with 40 participants, each completing set of 25 comparative tasks. In each task, participants are presented with a reference image, a prompt, and two generated images (A and B). They answer of generated images, A and B, is created using Hollowed Net and LoRA FT, and the labels (A or B) are randomly assigned for each task. Table 2 displays the results of these user studies. These findings confirm that users generally perceive the images generated by Hollowed Net and LoRA FT to be similar in both subject fidelity and text fidelity, consistent with the main results presented in Table
questions: subject fidelity and text fidelity. Each pair
Additionally, we include the analysis of computational loads for Hollowed Net and LoRA FT in Table 3 Each number corresponds to one step of each stage: one forward pass for pre-computing and inference and one forward+backwant pass for fine-tuning, For the fine-tuning of Hollowed Net, 1000 steps required, totaling 2.004 1000 2004 TFLOPs. For pre-computing, we find 200 pre-computed samples are sufficient to achieve high-fideling results (see Appendix [ for a detailed analysis), requiring 0.238 x 200 47.6 TFLOPs of additional computation. Therefore, the total computation required for training with Hollowed Net is 2004 + 47.6 2051.6 TFLOPs, which is lower than 2.148 1000- 2148 TFLOPs needed for LoRA FT. On the other hand, for running an inference pass, Hollowet Net requires approximately 0.204 TFLOPs more than LoRA FT, as it needs to repeat part of the early down-blocks to reproduce the path used in training.
Qualitative Evaluation In Fig. we present qualitative generation results of Hollowed Net for various subjects and prompts. The upper half shows examples from the DreamBooth dataset, and the lower half displays examples from the CustomConcept1 dataset. These results demonstrate that Hollowed Net effectively captures the visual details of the target subjects, while maintaining high text-image alignment for different types of applications including property modification, recontextu- alization, accessorization, and artistic rendition. Its ability enabling high-fidelity personalization with memory costs as low as those of inference makes it an efficient solution for a range of on-device appli- cations with constrained computational resources. Additional qualitative examples with SDXL. 23 are included in Appendix E illustrating the scalability of our approach in a larger model.

---

A [V] stuffed animal the jungle.
A [M] toy with wheat
field the
background
A [V] sneaker with mountain in background.
Original Image
Small
The fractions of hollowed layers
Large
Figure 6: Qualitative results with different fractions of hollowed layers given three subjects from the DreamBooth dataset [5].
5.3 Ablation Study on Fractions of Hollowed Layers
Based on the symmetrical "U-shape" architecture of the diffusion U-Net, design different Hollowed Net architectures by selecting a different up-block layer skip-connected to a down-block layer and hollowing out the middle layers between the pair. Figure presents experimental results across different fractions of hollowed layers, ranging from around 10% to 85% of layers removed. In we observe the peak GPU memory usage decreasing linearly with layer removal, fewer model weights need to be stored on the GPU during backpropagation. Analyzing the DINO and CLIP-I scores in Fig. (b) and (c), we find that the model's capacity preserve subject fidelity remains comparable to or slightly better than LoRA until around 39.2% of layers are removed, where memory cost reduces nearly to the level of inference. Beyond this threshold, however, subject fidelity significantly diminishes, as fewer layers essential for personalization are included in the Hollowed Net. This effect of hollowed layer fractions is also visible in the qualitative results in Fig. 6 Meanwhile, the CLIP-T T score does not exhibit a general trend, except in cases of very high hollowed fractions, where the model not capable of personalization, and thus generates images solely based on given prompt. However, note that the increase in CLIP-T remains marginal, as Hollowed Nets with low hollowed fractions also maintain a high capacity for text-image alignment.
Fig. 5.
Conclusion
In conclusion, our paper introduces novel approach for on-device personalization through memory- efficient fine-tuning with Hollowed Net. Hollowed Net effectively leverages the architecture of the diffusion U-Net, enabling fine-tuning with significantly reduced memory costs by minimizing the model's 's size during fine-tuning without requiring any additional processes such as structural pruning or pre-training on large-scale datasets. However, we observe that, due the use of non-personalized prompts with the original network, the model's performance can be sensitive to the granularity of class token definitions. For example, the DreamBooth dataset contains "poop emoji" images, for which the class token is very coarsely defined as "toy". In this case, non pons-eennalleee intermediate activations generated with prompts using "toy" struggle effectively correlate and generate "poop emoji" image. Therefore, a careful choice of fine-grained class tokens is necessary for the effective application of Hollowed Net.
Additionally, it is worth noting that our methodology is orthogonal to existing different PEFT methods 24 25 and quantization methods [26] (7). Thus, our approach offers substantial potential for further memory reduction, which is crucial for training under constrained computational resources. Furthermore, while our primary focus in this paper has been on image generation tasks, our method is not limited to diffusion models and can be seamlessly extended to various NLP tasks with LLMs, which we leave for future work. We anticipate that Hollowed Net will be applied wide range of tasks requiring constrained computational resources, serving as an efficient solution for various on-device applications.
10

---

References
[1] Jonathan Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6885 6851, 2020.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[3] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0160 2022.
[4] Lymin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[5] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Ktir Aberman. Dream- booth: Fine tuning ext-to-image diffusion models for bobject-driveed generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12500-22510 2023, licensed under CC BY 4.0.
[6] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311. 2023.
[7] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[8] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingho Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreamhooth: Hyperneworrs for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06990 2023.
[9] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or, Encoder- based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 42(4):1-13 13, 2023.
[10] Yuxiang Wei, Yabo Zhang, Zhilong Ji. Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15943-15953. 2023.
[11] Wenhu Chen, Hexiang Hu, Yandong Li. Nataniel Ruiz, Xuhui Jia, Ming Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems, 36, 2024.
[12] Dongxu Li. Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Syxtems, 36, 2024.
[13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation large language models, arXiv preprint arXiv:2106 09685, 2021.
[14] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[15] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a haseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings. Part III 16, pages 698-714. Springer, 2020.
[16] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce activations, not trainable parameters for efficient on-device learning, arXiv preprint ar arXiv:2007.1 10077.16222, 1622, 2020.
[17] Yizhe Zhang. Shubhankar Borse. Hong Cai, and Fatih Porikli. Auxadapt: Stable and efficient test-time adaptation for temporally consistent video semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2238, 2022.
[18] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side- aide-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35:12991- 13005, 2022.
11

---

[19] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024.
[20] Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-K3 Kyu Song. Shortened Ilaaa: simple depth pruning for large language models. arXiv preprint (iXiv:2402.02834.. 2024.
[21] Bo- Bo-Kye Kim, Hyoung-Kye Song, Thibault Castells, and Shinkook Choi. Bk-sdm: lightweight, fast, and cheap version of stable diffusion. arXiv preprint arXiv:2305. 157988, 2023.
[22] Yijun Li. Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. arXiv preprint arXiv:2012.02780. 2020.
[23] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint Xi::23077.09 2023.
[24] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[25] Shih Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint urXiv:2402.09353. 2024.
[26] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1972-1981, 2023.
[27] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 17535 17545, 2023.
12

---

A Experiments with Layer-Pruned Diffusion Models
As shown in recent work 21] layer pruning involves the complete removal of selected layers, which necessitates extensive pre-training on large datasets to recover lost information and restore model functionality.. However, diffusion models often suffer from substantial performance degradation post-pruning, as the lost information may not be fully recoverable through pre-training.
Table 4 presents experiments with BK-SDM 21 models, layer-pruned SD models, using rank- 128 LoRA. Compared to the results in Table these models achieve memory usage comparable to Hollowed Net but show significant performance degradation. Despite extensive pre-training, their performance remains compromised.
In contrast, Hollowed Net does not completely remove deep layers and requires additional pre-training. Instead, we temporarily exclude selected layers during fine-tuning while preserving essential information through pre-computation stage. Notably, despite this added stage, the overall computational load for training Hollowed Net can remain more efficient than LoRA fine-tuning, as discussed in Sec. 5.2 and Appendix D
Table 4: Quantitative results of BK-SDM on the DreamBooth dataset [5]
Method
# of Parameters
Training Memory
DINO
CLIP-I
CLIP-T
BK-SDM-Bass
595.7M
3,546GB 3.133GB
0.629 + 0.012
0.788 1:2 0.007
0.300 - 0.001
BK BK-SDM- SDM-Small
496.2M
0.602 = 0.013
0.774 + 0.008
0.298 = 0.001
Original U-Net (0% Removed)
RA
RA
54
Hollowed Net (39.2% Removed)
1.0
3. RA
9.3
1
32
a
6.3 RA
doem Mocke
cown bocto.1
doen booka 2
doem bloclt.3
1el Mook
up Modia
Modea
vp Medie3
Figure Architectural details of Hollowed Net. represents ResBlock. RA represents a set of ResBlock and CrossAttentioonllock. D and U represent Downsample and Upsample Convs, respectively.
B Architectural Details of Hollowed Net
In this section, present the architectural details of Hollowed Net. We leverage the skip connections inherent in the U-Net architecture to determine which layers to be removed during fine-tuning (hollowed). For our main results, choose the third block of the down blocks.2 (block 3-3), the entire down blocks.3 down_blocks.3 (blocks 4-1 and 4-2), the entire mid block (blocks 5-1 and 5-2), and the entire up_ blocks.0 (blocks 6-1, 6-2, 6-3, and 6-4) to be hollowed, which corresponds around 39, 99.%% of the U-Net's parameters, as described in Fig.
Similarly, Hollowed Net with different fractions of hollowed layers can be achieved as follows:
11.5% removed: blocks 5-1 and 5-2 are hollowed.
-00.%% removed: blocks 4-2, 5-1, 5-2, and 6-1 are hollowed.
13

---

30.1% % removed: blocks 4-1, 111,4,,,-1,,, 5-2, 6-1, and 6-2 are hollowed.
56.6% removed: blocks 3-2, 4-1, 4-2, 5-1, 6-1, 6-2, 6-3, 6-4, and 7-1 are hollowed.
73.3% removed: blocks 3-1,3-2, 4-1, 4-,, 5-2, 6-2, 6-3, 6-4, 7-1, and 7-2 hollowed.
84.3% removed: blocks 3-2,3-1, and 7-4 are hollowed.
6-2, 6-3, 6-4, 7-1, 7-2, 7-3,
Table Quantitative results of LoRA FT' and Hollowed Net with different ranks
Method
# of Parameters
Training Memory
DINO
CLIP-I
CLIP-T
LoRA 1
866.7M
4.847GB
0.564 0.014
0.766 1 0.006
0.311
0.001
LoRA r=16
869.2
4,883GB
0,618 + 0.008
0.788 + 0.005
0.305 + 0.001
Hollow r=4
527.7M
3.526GB
0.566 + 0.009
0.763 = 0.003
0.311 + 0.001
Hollow 1=16
529.9M
3.558GB
0.626 0.009
0.789 = 0.005
0.305
0.001
C Experiments with Different Ranks
In Table 5 present the results using LoRA and Hollowed Net with different ranks (4 and 16) using the DreamBooth dataset. While the default rank in the diffusers library is often used, we have found that often oversimplifies personalization details fails effectively handle range of challenging subjects and prompts. Increasing the rank from 4 to 16 improves subject fidelity. However, to achieve personalization quality comparable full fine-tuning across all subjects and prompts, we find that the rank of 128 is necessary.
0.29
0.50
0.28
200 400 600 000 1000
200 400 600
800 1000
200 400 900 800 1000
200 400 000 800 1000
(a) Data Storage (GB)
(b) DINO
(c) CLIP4
(d) CLIPT
Figure 8: Analysis of different numbers of pre-computed samples. For all figures, the x-axis represents the number of pre-computed samples. The y-axis corresponds the metric used for each figure.
Table 6: Resource usage analysis for different hollowed fractions and numbers of pre-computed
samples
Hollowed Fraction
A of Procomputed Samples
Peak GPU Usage
Date Storage
Pre-computing FLOPs
Fine- tuning FLOP,
Total FLOPs
11.5%
4 4.49GB
0.08GB
44T
208TT
2125T
11.5%
4.49GB 4. 4.49GB
0,19GB
109T
208IT
2190T
11.5%
1000
0.38G3
218T'
208TT
2299T
29.25%
3 3,800H
0.26G3
2004T
20521
39.2%
1000
3 3.88GB
1,3GBB
238T
2004T
2242T
56.6%
200
3. J,16GB
0.26GB
56T
1776T
1832T
56.6%
1000
3. 3.16BB
1.3GGB
279T
1776T
2055T
D Further Analysis on Computational Costs
In Fig. provide ablation studies the impact of varying the number of samples both quantitative and qualitative results. The findings indicate that using only 200 pre-computed samples results in minimal performance degradation compared using 1000 pre-computed samples.
Additionally, we present a detailed analysis computational loads and space consumption in Table 6 for different numbers of precomputed samples and different fractions of hollowed layers, which
14

---

will enable users to choose the optimal configurations of Hollowed Net according to their specific resource constraints.
down blockblocks.
0.0036
down, down__block blocks.0
0.0035
Nown_blocks.]
0.0027
down_blocks.3
0.0027
down blocks.2
0.0024
blocks.2
mid_block
0.0025
mid block
0.0125
blooooks.00
0.0030
blockkk..0
0.0029
blocks.1
0.0046
blocks.1
0.0046
up_ blockkkk22
0.0046
up_ blocks.2
0,0046
0.000 0.001 0.002
0.003
0.004
0.005
0.000 0.001
0.002
0.003
0.004
0.005
(a) DreamBooth
(b) CustomConcept1001
Figure 9: Analysis the LoRA weight change before and after personalization, per block of U-Net using SDXL 23
on purple rug in the forest
M weering yellow shirt
[V] a purple wizard outfit
wearing pink glasses
INPUT IMAGE
M =
M with colorful flower bouquet
[V] the snow
[V] the beach
[M] on top of wooden floor
HOLLOWED NET
M = Vase
Figure 10: The qualitative examples of Hollowed Net and LoRA FT with the samples from the DreamBooth dataset 5 using SDXL 23
15

---

E Experiments with SDXL
To demonstrate the scalability of Hollowed Net, we present additional analysis and qualitative examples using SDXL [23] Figure shows that similar patterns of weight changes are observable with SDXL, as displayed in Fig. 2 In Fig. T0. we present qualitative examples of Hollowed Net and LoRA FT with the samples from the DreamBooth dataset using SDXL. Hollowed Net is applied by removing the entire mid mid_block layers (410M parameters) of SDXL. The results show that Hollowed Net achieves high-fidelity personalization results comparable to LoRA FT.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Refer Section 5 Experiments which includes the materials to support the
answer.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Refer Section Conclusion which describes the limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Refer Section and which describe the details of our architecture and experimental settings.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

reproducible, they Answer: [No]
Justification: We used a public dataset and described its experimental settings (Refer the Section 5). The code would be available after internal review process has been completed (not available at this submission time).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
public/guidess httss//ooduumssssooooooo..
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb httppss.ccpublli/guudeeee wttpeee/s/oooSSeeesseeiiiiii
ssioiiolioiiioioccicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Refer Section Experiments which describes the details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Refer the table in Section 5 Experiments where the statistical variations of the results are included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Refer the Section Experiments for the details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pp..paooo...oooooo/////ea
Answer: [Yes]
Justification: Conform with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: (based on the guideline described below) Our work does not consider to improve the quality of generative models itself but to optimize the network to fine-tune the model in a faster and more efficient way.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not release any model dataset but considers an efficient fine-tune fine-tuning method.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our work the model and dataset with the license and terms of use explicitly mentioned in the release package.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "5qCiRHBogF", "reviewer": "Reviewer_cT56", "summary": "The paper introduces Hollowed Net, a novel approach for on-device personalization with T2I LDMs. The model gives better memory efficiency by modifying the U-Net to remove less-essential deep layers. This helps with the on-device memory constraints. It reduces memory usage during fine-tuning while maintaining personalization performance. The paper provides quantitative and qualitative analyses to demonstrate its effectiveness and discusses potential applications and scalability.", "questions": "Is there a comparison with other memory-efficient approaches besides LoRA?", "limitations": "The authors have discussed the limitations of their work, along with potential negative societal impacts.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper focuses on reducing memory usage by modifying the U-Net architecture to remove non-essential deep layers.\nThe paper addresses memory efficiency and demonstrates reduced GPU memory while maintaining personalization performance.\nThe paper provides quantitative and qualitative results to show its effectiveness.\nThe paper discusses potential applications and scalability.", "weaknesses": "- In table 1, a 1.~GB memory usage save, while beneficial, is still marginal compared to LoRA, especially if the overall computational efficiency does not significantly surpass those of existing methods.\n- As mentioned by the author, methods like SVDiff optimizes singular values of the weight matrix, leading to an even smaller model checkpoint. Is there a comparison with other memory-efficient approaches like this?\n- The idea of removing less-important unet layers seems to be a one-time optimization, because it relies on identifying and justifying certain layers as less important compared to others. This might not be scalable/generalizable across different models and tasks, as the importance of different layers could be different, depending on the application or dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Please note our top-level comment with additional experimental results. Below we address specific questions.\n\n>1. 1.$\\sim$GB memory usage save, while beneficial, is still marginal\n\nIn this paper, we are assuming an extremely resource-constrained environment where even running an inference (costing approximately 3.49GB in our experiments) can exhaust most of the available memory. This assumption is reasonable given that the total memory capacity of recently released mobile phones (e.g., iPhone) ranges from 6 to 8GB, which must be shared with the operating system and other background apps. In such cases, a 1.35GB memory usage reduction can be substantial, directly affecting the feasibility of performing fine-tuning. \n\nThe strength of Hollowed Net lies in its ability to enable model fine-tuning with memory usage similar to or even less than that required for inference, depending on the proportion of hollowed layers. Furthermore, in proportional terms, applying Hollowed Net results in a 35\\% memory reduction compared to LoRA, which is a significant improvement. In addition, more memory reduction can be achieved with more hollowness depending on the usecases and tasks. We believe this substantial reduction is crucial in environments with limited resources and makes our approach highly beneficial.\n\n>2. Comparison with other memory-efficient approaches\n\nUnfortunately, SVDiff does not have an official code release. We attempted to use a third-party reproduced code from GitHub, but it was not implemented in a memory-efficient manner, requiring over 14GB of peak GPU memory for processing a single image.\n\nTheoretically speaking, the mechanism of SVDiff is not significantly different from LoRA, in terms of memory usage.  As described in the paper, SVDiff uses 2200 times fewer parameters compared to vanilla DreamBooth, which suggests that SVDiff updates around 391K parameters. This is nearly double the number of parameters updated by rank-1 LoRA, presented in Table 1 of our paper. Consequently, we estimate that the minimal memory usage achievable by SVDiff would be at least 1GB more than Hollowed Net. In fact, all fine-tuning methods based on reducing updating parameters like SVDiff have a fundamental limitation: they can reduce the number of updated parameters but cannot reduce the actual model size, which takes a significant portion of memory usage when performing backpropagation over entire backbone network. In this context, rank-1 LoRA demonstrates the minimal baseline of memory usage that these methods can achieve.\n\nAnother memory-efficient approach to consider is model compression, which reduces the backbone model's size through pre-training with a large dataset. We present results using BK-SDM, a recent method accepted at ECCV '24, in the table below. We fine-tuned the compressed Stable Diffusion models of BK-SDM using rank-128 LoRA on the DreamBooth dataset. Our experimental results show that while these models achieve memory usage comparable to Hollowed Net, they suffer from significant performance degradation. These models' capacity is limited to the data used during model compression, and for subjects and prompts outside this data, they can even generate seriously malformed images.\n\nIn contrast, Hollowed Net achieves both low memory usage, comparable to compressed models, and high-quality results, similar to full fine-tuning of diffusion models. This is achieved through our novel approach, which effectively removes layers during training while preserving the knowledge from those hollowed layers throughout both training and inference stages.\n\n\n| Method        | # of Parameters | Model Size | Training Memory | DINO          | CLIP-I        | CLIP-T         |\n|---------------|-----------------|------------|-----------------|---------------|---------------|----------------|\n| BK-SDM-Base   | 595.7M          | 2.272GB    | 3.546GB         | 0.629 ± 0.012 | 0.788 ± 0.007 | 0.3 ± 0.001    |\n| BK-SDM-SMALL  | 496.2M          | 1.893GB    | 3.133GB         | 0.602 ± 0.013 | 0.774 ± 0.008 | 0.298 ± 0.001  |\n\n\n>3. Discussion on scalability and generalizability\n\nTo demonstrate the scalability and generalizability of our method, we include additional results and analysis in top-level comments. Please also refer to the answer for the first question of the last reviewer: Jp6t.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d1a9HZZoq1", "reviewer": "Reviewer_fGHD", "summary": "This work proposes Hollowed Net, a parameter-pruning method for training subject-driven synthesis models under limited computing resources.", "questions": "In general, the idea of pruning less important parameters is a promising topic on finetuning large models. However, I suggest further polishing w.r.t. the issues above to make this work better.", "limitations": "Limitations have been discussed in the paper. The authors have claimed in the checklist that there is no societal impacts of this work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "According to the experimental results, applying Hollowed Net will reduce the memory costs during training stages, while keeping similar performance.", "weaknesses": "1. In on-device cases, the computing time is always another key issue. However, it seems that Hollowed Net needs extra forward and buffering stages (first stages in the paper) during both training and inference periods. I wonder how much computational overload and extra time will be caused by these stages.\n2. Since this work is mainly focusing on space consumption, it is also necessary to evaluate the memory usage of the buffering stages. Even if these stages may take RAM instead of VRAM, it is also important for on-device computation since RAM is usually limited as well. Besides, does the *Peak Usage* reported in Tab. 1 include the buffering stage?\n3. It is necessary to provide the results with LoRA rank between 1 and 128, for both plain LoRA and Hollowed Net. To the best of my knowledge, using rank 1 may be inadequate, while using rank 128 may be excessive for subject-driven synthesis, since the official examples of the *diffusers* library suggest rank 4 for DreamBooth [A].\n4. The hollowed part in Hollowed Net cuts the gradients from backpropagating from up blocks to down blocks of the UNet. Providing more discussion on the effects of these gradients will make the design of Hollowed Net more reasonable. Besides, another possible design is to remain the hollowed part (without LoRA either) in forwarding, while stopping the gradients to go through this part. I suppose such design may get rid of the extra buffering stages in both training and inference periods.\n5. From a perspective of higher level, the idea of Hollowed Net may be applicable to many other tasks of finetuning relatively large models. The authors are encouraged to do exploration on such direction in the future.\n\n[A] https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Please note our top-level comment with additional experimental results. Below we address specific questions.\n\n| Methods       | LoRA            |             | Hollowed Net   |             |\n|---------------|------------------|-------------|----------------|-------------|\n|               | FLOPs            | GPU Memory  | FLOPs          | GPU Memory  |\n| Collection    | -                | -           | 0.238T         | 2.138GB     |\n| Fine-tuning   | 2.148T           | 5.226GB     | 2.004T         | 3.875GB     |\n| Inference     | 0.716T           | 3.597GB     | 0.920T         | 3.588GB     |\n\n> 1. Further discussion on computational overload\n\nIn the table above, we provide a detailed analysis of computational load (FLOPs) and memory usage (peak GPU memory) for each stage of Hollowed Net and LoRA. Each number corresponds to one step of each stage: one forward pass for Collection and Inference, and one forward+backward pass for Fine-tuning. \n\nFor fine-tuning, $\\sim$1000 steps are required, totaling $2.004 \\times 1000 = 2004$ TFLOPs of computation. For Collection, 200 buffered samples are sufficient for achieving high-fidelity results, which requires $0.238 \\times 200 = 47.6$ TFLOPs of additional computation. Thus, the total computation required for training with Hollowed Net is $2004 + 47.6 = 2051.6$ TFLOPs, which is lower than the $2.148 \\times 1000 = 2148$ TFLOPs needed for fine-tuning LoRA.\n\nFor inference, Hollowet Net requires around 28\\% more computation than LoRA, as it needs to repeat a part of early downblocks to reproduce the path used in training. Considering that recent works show that on-device inference of Stable Diffusion models can be done within 7-15 seconds, those additional computational cost for inference may not be a significant issue, considering the advantage of significantly lowering the memory cost of fine-tuning to a feasible level.\n\nHowever, if inference latency is a concern, a variant of the inference method for Hollowed Net can be considered. This variant involves a single inference path with LoRA, where the cross-attention layers hollowed during training receive a non-personalized prompt (e.g., \"A dog\") as input. As shown in the table below, this variant (Single Path) requires the same amount of computational FLOPs as LoRA but achieves a level of personalization equivalent to that of the default inference method (Dual Path). One potential explanation for the effectiveness of this efficient method is that the high-level features in the middle layers are focused on processing abstract class-level characteristics of an image rather than capturing personalization details at the pixel level. Therefore, even if the middle layers are fed features of a personalized dog (in Single Path) instead of a non-personalized dog (in Dual Path) from the early layers, the extraction of high-level features for \"A dog\" is not affected.\n\n|                         | FLOPs | DINO           | CLIP-I         | CLIP-T         |\n|-------------------------|-------|----------------|----------------|----------------|\n| Hollowed Inference (Dual Path)   | 0.920T | 0.660 ± 0.011  | 0.805 ± 0.006  | 0.300 ± 0.001  |\n| Hollowed Inference (Single Path)  | 0.716T | 0.660 ± 0.011  | 0.802 ± 0.006  | 0.301 ± 0.001  |\n\n> 2. Further discussion on space consumption\n\nThe Peak Usage reported in Table 1 of our paper represents the maximum GPU memory (VRAM) consumption during the fine-tuning stage. As shown in the table above, the VRAM usage during the Collection stage is significantly lower because only part of the upper blocks needs to be retained. Each set of pre-computed outputs requires approximately 1.35 MB. Therefore, $\\(1.35 \\times 200 = 270\\)$ MB of additional storage is needed for keeping all data required later during fine-tuning. Whether all this data is held in RAM simultaneously rather than SSD will depend on the specific implementation.\n\n> 3. Results with LoRA rank between 1 and 128\n\nIn the table below, we present the results using LoRA with different ranks (4 and 16). While the default rank of 4 in the diffusers library is often used, we have found that it may oversimplify personalization details or fail to effectively handle a range of challenging subjects and prompts. Increasing the rank from 4 to 16 improves subject fidelity. However, to achieve personalization quality comparable to full fine-tuning across all subjects and prompts in the DreamBooth dataset, we find that a rank of 128 is necessary.\n\n| Method        | # of Parameters | Model Size | Training Memory | DINO          | CLIP-I        | CLIP-T        |\n|---------------|-----------------|------------|-----------------|---------------|---------------|---------------|\n| LoRA r=4      | 866.7M          | 3.306GB     | 4.847GB         | 0.564 ± 0.014 | 0.766 ± 0.006 | 0.311 ± 0.001 |\n| LoRA r=16     | 869.2M          | 3.316GB     | 4.883GB         | 0.618 ± 0.008 | 0.788 ± 0.005 | 0.305 ± 0.001 |\n| Hollow r=4    | 527.7M          | 2.013GB     | 3.526GB         | 0.566 ± 0.009 | 0.763 ± 0.003 | 0.311 ± 0.001 |\n| Hollow r=16   | 529.9M          | 2.021GB     | 3.558GB         | 0.626 ± 0.009 | 0.789 ± 0.005 | 0.305 ± 0.001 |\n\n> 4. Discussion on the design of Hollowed Net\n\nThis is indeed a very interesting suggestion. Modifying the gradient flow by stopping gradients from passing through the hollowed part could replicate backpropagation in Hollowed Net while maintaining the hollowed layers. However, the primary advantage of Hollowed Net is that it avoids holding these hollowed layers in GPU memory during fine-tuning, which saves approximately 1.29GB of additional memory. This reduction in memory usage is a significant benefit, making it challenging to eliminate the extra buffering stages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z3Ku8sqhnl", "reviewer": "Reviewer_SH8P", "summary": "This paper proposes a novel framework named “Hollowed Net” for memory-efficient LoRA fine-tuning on the personalization task, showing its potential for on-device personalization tasks with limited resources. With the hollowed UNet structure, the training memory requirement is largely reduced while the performance maintains at the same level. In the meantime, the memory cost during inference is unchanged since the hollowed net is a sub-network of the original UNet and can be directly transferred back. Besides, the proposed method is flexible and scalable to different model architectures with controllable hollowed layers.", "questions": "1. How is Hollowed net compared with a LoRA model that does not finetune all the attention layers?\n2. In figure 1 (top), the output of 5th block is conditioned on \"a dog\", while in figure 1 (bottom), the condition is \"a [v] dog\". In this way, the input from data buffer and conditional input/skip connection to block 6 is not consistent. Will this be a problem in training? e.g., requires a carefully selected class token as mentioned in the conclusion\n3. How is the analysis in Section 4.1 conducted? Is it only an example on one subject in dreambooth dataset? If different personalization tasks shared similar observations, can you elaborate?\n4. In figure 3, how is the red arrow from block2 to block3 used?", "limitations": "The authors have added justifications for limitations and societal impact of their work, both of which are reasonable. It would be better to include what potential social impact would be for improving memory efficiency for on-device personalization.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Improved Efficiency:** As shown in Table 1, the hollowed net reduced training time memory cost from 5.23GB to 3.88GB while maintaining competitive quantitative performance.\n\n**Well-motivated:** The observation that middle layers of the UNet contribute less during fine-tuning in the personalization task directly inspires removing them to save memory, and the paper proposes a feasible way to do it.", "weaknesses": "**Experimental evaluation to be improved:** the paper uses DINO and CLIP scores for quantitative evaluation, however these metrics are sometimes considered less reliable because they could be inconsistent with human preference (e.g., [1][2]). Similar works have been using user studies with detailed configurations to demonstrate quantitative performance (e.g., [1][2][3][4]), which is missing in this paper.\n\nThe method took a step forward towards better efficiency, but **more analysis on model size and latency will be needed** to claim “on-device personalization”, since the current experiment settings still seem to be large for real on-device use cases, e.g., 1000 steps with 3.88 GB. For example: How much memory is required for the data buffer? Would the observation/motivation hold for other smaller/faster models like quantized/flow models?\n\n\n\n[1] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang,\nJarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. “StyleDrop: Text-to-Image Generation\nin Any Style”. In: 37th Conference on Neural Information Processing Systems (NeurIPS).\nNeural Information Processing Systems Foundation. 2023.\n\n[2] Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. \"RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control.\" arXiv preprint arXiv:2405.17401 (2024).\n\n[3] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510. 2023.\n\n[4] Nataniel, Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. \"Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6527-6536. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Please note our top-level comment with additional experimental results. Below we address specific questions.\n\n> 1. Experimental evaluation with user studies\n\nBased on you suggestion, we conducted user studies with 40 participants, each completing a set of 25 comparative tasks. For each task, participants were presented with one reference image, one prompt, and two generated images (A and B). They answered two questions: subject fidelity and text fidelity. Each pair of generated images, A and B, was created using Hollowed Net and LoRA FT, and the labels (A or B) were randomly assigned for each task. The table below displays the results of the user studies. These results confirm that users generally find the images generated by Hollowed Net and LoRA FT to be similar in terms of both subject fidelity and text fidelity, consistent with the main results presented in Table 1 of the paper.\n\n|                     | Hollowed Net | Tie  | LoRA FT |\n|---------------------|--------------|------|---------|\n| **Subject Fidelity** | 31.2%        | 49.3%| 19.5%   |\n| **Text Fidelity**    | 18.1%        | 69.4%| 12.5%   |\n\n> 2. More analysis on model size and latency \n\nWe evaluated our HollowedNet based on the Stable Diffusion v2.1 with 866M. We think that our fine-tuning approach of 1000 steps and 3.88GB given this backbone is an on-device-feasible solution with the following reasons: \n\n2.1) [Memory] Recent mobile phones can support 8GB or 12GB memory (iPhone15 or GalaxyS23 Ultra), and thus the training memory 3.88GB can fit into this space. \n\n2.2) [Latency] Though we couldn't measure the actual latency on a mobile device, we expect the latency based on the previous report. For example, the Table 1 in [1] shows around 7s-15s for an inference on a GalaxyS23 device. Compared to an inference, we observed that our fine-tuning needs 2.2x more computations than an inference (6.013 TFlops vs 2.717 TFlops). Based on this, for a single training step, roughly we can say it takes around 22 sec (2.2times x 10sec/inference). So, for 1000 steps, it would take 6 hours, and this can be done during charging time at night. Furthermore, combining with a HyperNetwork which provides initial LoRA parameters, we may also expect 40 times reduction in training steps (as reported in the Hyper-dreambooth paper), and this leads to less than 10min solution. \n\n[1] Choi, Jiwoong, et al. \"Squeezing large-scale diffusion models for mobile.\" arXiv preprint arXiv:2307.01193 (2023).\n\n> 3. Memory for data buffer\n\nFor collection, 200 buffered samples are sufficient for achieving high-fidelity results\nEach set of pre-computed outputs requires approximately 1.35 MB. Therefore, \\(1.35 \\times 200 = 270\\) MB of additional storage is needed for keeping all data required later during fine-tuning. Whether all this data is held in RAM simultaneously rather than SSD will depend on the specific implementation. \n\n> 4. Working with quantized/flow models\n\nWe believe that there is no clear reason why Hollowed Net cannot be applied to quantized/flow models, but some modifications may be necessary depending on the technical differences in the model architecture.\n\n> 5. LoRA model that does not finetune all the attention layers\n\nThe table below shows the results of selectively applying LoRAs to the layers, excluding those removed in the Hollowed Net. We find that selectively applying LoRAs leads to a slight decrease in performance. This may be due to the cross-attention layers in the middle receiving a personalization prompt (e.g., \"a [V] dog\") but not learning how to process the unique token \"[V]\". More importantly, in terms of memory efficiency, using fewer LoRAs results in only a minimal decrease in training memory. As shown in the table, the main advantage of the Hollowed Net is its ability to avoid holding the hollowed layers in GPU memory during fine-tuning, resulting in a 35\\% reduction in GPU memory usage compared to LoRA.\n\n> 6. Discussion on conditional input/skip connection\n\nHow to handle inconsistency and leverage it to produce better personalization results is what the model learns during fine-tuning. The deep layers in the middle are more involved with high-level abstract features (e.g., the concept of a dog) rather than the pixel-level personalization details of a specific instance (e.g., a [V] dog). Thus, at this level, as long as the categories are well-defined, the deep features of a specific instance can be replaced with the deep features of its corresponding class (e.g., a dog) through proper fine-tuning. In fact, both the results in Table 1 of the paper and the user study presented above show that Hollowed Net can perform slightly better than LoRA fine-tuning in some cases. We assume this is because using class-level inputs for the middle layers can result in positive regularization for effective personalization to a certain extent. This suggests that the resolution of defining class tokens can lead to better or worse results compared to LoRA.\n\n| Method              | # of Parameters | Model Size | Training Memory | DINO            | CLIP-I          | CLIP-T          |\n|---------------------|-----------------|------------|-----------------|-----------------|-----------------|-----------------|\n| LoRA r=128          | 892.5M          | 3.404GB    | 5.226GB         | 0.658 ± 0.001   | 0.806 ± 0.005   | 0.299 ± 0.002   |\n| LoRA r=128 (No middle) | 889.9M        | 3.395GB    | 5.199GB         | 0.654 ± 0.011   | 0.801 ± 0.004   | 0.301 ± 0.002   |\n| Hollowed Net        | 550.9M          | 2.102GB    | 3.875GB         | 0.660 ± 0.011   | 0.805 ± 0.006   | 0.300 ± 0.001   |\n\n> 7. Expanded explanation of analysis on LoRA weight changes\n\nPlease refer to the answer for the first question of the last reviewer: Jp6t.\n\n> 8. In figure 3, how is the red arrow from block2 to block3 used?\n\nThank you for your keen observation and a thorough review. The red arrow from block2 to block3 should be removed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hn7SVCVJS", "reviewer": "Reviewer_Jp6t", "summary": "The authors propose an efficient personalization method using LoRA for on-device text-to-image diffusion models. The method relies on removing non-essential layers in the U-net, reducing GPU usage during fine-tuning.", "questions": "Maybe the evaluation on 30 images is a bit too small, line 241. Can the authors evaluate on a larger dataset to compare the performance?", "limitations": "The output quality might be affected by the reduction of the layers. Removing the middle-layers means that you are excluding long range image editing (e.g. the background editing might be affected). A further investigation on a larger dataset might be required.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written.\n- Discarding a few layers in the U-net copy architecture returns similar results in generation.", "weaknesses": "- The whole paper is based on the assumption of Fig.2 that during after the fine-tuning, the external layers of the U-net are more affected than the internal one. I said assumption because I cannot find anywhere in the paper on which data they perform that statistics neither further explanations on it.\n\n- Based on this, the authors just optimised the external layers using LoRA without considering the internal one. The gain in efficiency shown in Table 1 is relatively small respect to the standard LoRA over the whole model, mainly because they are excluding the less computationally costly layers from the fine-tuning.\n\n- The idea is similar to ControlNet, https://arxiv.org/abs/2302.05543, combined with LoRA, involving the exclusion of a few layers in the cloned copy. I do not see too much novelty in it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Please note our top-level comment with additional experimental results. Below we address specific questions.\n\n> 1. Clarification on assumption and experimental details\n\nThe experiment depicted in Figure 2 of the manuscript, as detailed in Section 4.1, was conducted to identify less significant layers within the U-Net by analyzing the changes in LoRA weights before and after personalization. Specifically, we measured the extent of weight changes across different blocks of the U-Net, where smaller weight changes in certain layers after personalization indicate that these layers are less important for the personalization process. This assumption is inspired by previous work, notably Li et al. [22], Kumari et al. [7], and Shah et al. [6], which demonstrated similar analyses of weight changes.\n\nFigure 2 of the manuscript illustrates the results obtained when personalizing the \"monster toy\" subject from the DreamBooth dataset, starting from the pretrained weights of the Stable Diffusion V2.1 model. We injected LoRA into various layers of the U-Net's attention and ResNet blocks, and then fine-tuned the model for 1000 steps with a learning rate of 1e-4, using a LoRA rank of 128. We observed that the overall trends in weight changes remained consistent across different learning rates, steps, LoRA ranks, LoRA injection locations, and personalizing subjects.\n\nAdditionally, to substantiate our findings, we conducted further experiments, the results of which are described in Figure 1 of the attached document. We analyzed more diverse subjects (30 subjects from the DreamBooth dataset and 101 subjects from the CustomConcept101 dataset) and across different models (Stable Diffusion V2.1 and SDXL). The x-axis of each plot represents the specific U-Net blocks, while the y-axis shows the changes in LoRA weights before and after personalization. For each dataset, we averaged the weight changes across subjects and provided error bars to indicate the statistical variance within each dataset.\n\nSimilar to the original experiment in Figure 2 of the manuscript, these extended experiments revealed that weight changes around the central blocks tend to be minimal, confirming that these layers are less involved in personalization. This suggests that our assumption holds across various personalization tasks and models. We plan to include these additional findings in the supplementary materials.\n\n\n> 2. Discussion on the efficiency of Hollowed Net\n\nThe training memory used for training LoRA is 5.226GB, and more than 65\\% (3.404GB) comes from holding the entire model on GPU, and our aim is to remove the part of the models during training, in order to reduce the memory consumption during training without undermining the performance. \n\nIn particular, we assume an extremely resource-constrained environment where even running an inference (costing approximately 3.49GB in our experiments) can exhaust most of the available memory. This assumption is reasonable given that the total memory capacity of recently released mobile phones (e.g., iPhone) ranges from 6 to 8GB, which must be shared with the operating system and other background apps. In such cases, a 1.35GB memory usage reduction can be substantial, directly affecting the feasibility of performing fine-tuning. \n\n\n> 3. Comparison with ControlNet\n\nThe main differences between ControlNet and Hollowed Net arise from their distinct goals. First, ControlNet focuses on adding spatial conditioning controls to diffusion models. It leverages a cloned copy to use the pre-trained knowledge of diffusion models as initialization. However, ControlNet does not address memory efficiency concerns, as maintaining both the original and cloned networks simultaneously for training and inference requires substantial GPU memory.\n\nIn contrast, Hollowed Net is designed to achieve personalized LoRA parameters in a memory-efficient manner. We are the first to introduce a two-step fine-tuning strategy that enables efficient personalization of diffusion models with memory demands as low as inference. Compared to ControlNet, Hollowed Net uses a cloned copy to ensure that the personalized LoRA parameters can be transferred to the original network. Unlike ControlNet, which employs a cloned copy to complement the original network after training, Hollowed Net utilizes the cloned copy as a proxy for training LoRA parameters, without intending for the cloned copy to be used concurrently with the original network.\n\n\n> 4. Experiments with large datasets\n\nWe agree with you for the necessity of testing with larger datasets. Following your suggestions, we include the analysis with CustomConcept101 datasets, which you can find the analysis in the top-level comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewer,

We sincerely appreciate the reviewers for dedicating their time and effort to review our work. We have conducted multiple sets of experiments and studies to address all feedback and questions from the reviewers. Please refer to the attached document for a one-page analysis.

In order to show the scalability and generalizability of Hollowed Net approach, we include the experimental results with larger dataset, called CustomConcept101, which includes 101 different subjects with 15 different large categories and 20 unique prompts for each category. We conduct evaluation following the same protocols described in our paper for the DreamBooth dataset. As presented in the table of the top-level comment, the experimental results demonstrate that Hollowed Net consistently achieves the comparable results with full/LoRA fine-tuning of the diffusion models. Qualitative results can be found in the pdf, and the results show that our method works very for different instances with various challenging prompts of different applications including background, artistic rendition, accessorization, property modification, and etc.

|               | # of Parameters | Model Size | Training Memory | DINO          | CLIP-I        | CLIP-T        |
|---------------|-----------------|------------|-----------------|---------------|---------------|---------------|
| Full FT       | 865.9M          | 3.303GB     | 16.623GB        | 0.605 ± 0.005 | 0.773 ± 0.006 | 0.302 ± 0.002 |
| LoRA FT       | 892.5M          | 3.404GB     | 5.226GB         | 0.603 ± 0.008 | 0.773 ± 0.005 | 0.302 ± 0.002 |
| Hollowed Net  | 550.9M          | 2.102GB     | 3.875GB         | 0.603 ± 0.007 | 0.773 ± 0.005 | 0.302 ± 0.002 |

We also use the SDXL models and apply the Hollowed Net by removing the entire layers in the "mid_block", containing 410M parameters. We include the example qualitative results in the pdf. The results show that Hollowed Net achieves the high-fidelity personalization results compared to LoRA.

Additional experiments are included in the rebuttal comments for each corresponding reviewer.

Meta-review: The paper proposes a new approach for hardware-constrained personalization. Three out of four reviewers ranked the paper positively, recognizing the novelty of the proposed approach. They highlighted that the reduction in memory is significant, and importantly, does not come with a drop in performance. There was a fruitful discussion between reviewers and authors, in which, according to AC's understanding, most of comments were addressed. One reviewer provided a strongly negative rating. Compared to a typical review, the AC found their review somewhat short. The authors tried to address their concerns and answer their questions, by providing further discussion and experiments. The reviewer, however, remained unconvinced and kept the negative score. The AC read their comments and authors' responses, and believes that the authors addressed their concerns to a reasonable extent. Hence, the decision is to recommend acceptance. Congrats!
Decision: Accept (poster)